MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification
This research proposes a deep learning method for classifying student behavior in classrooms that follow the professional learning community teaching approach. We collected data on five student activities: hand-raising, interacting, sitting, turning around, and writing. We used the sum of absolute d...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/14/1/230 |
| _version_ | 1797359116229804032 |
|---|---|
| author | Somsawut Nindam Seung-Hoon Na Hyo Jong Lee |
| author_facet | Somsawut Nindam Seung-Hoon Na Hyo Jong Lee |
| author_sort | Somsawut Nindam |
| collection | DOAJ |
| description | This research proposes a deep learning method for classifying student behavior in classrooms that follow the professional learning community teaching approach. We collected data on five student activities: hand-raising, interacting, sitting, turning around, and writing. We used the sum of absolute differences (SAD) in the LUV color space to detect scene changes. The K-means algorithm was then applied to select keyframes using the computed SAD. Next, we extracted features using multiple pretrained deep learning models from the convolutional neural network family. The pretrained models considered were InceptionV3, ResNet50V2, VGG16, and EfficientNetB7. We leveraged feature fusion, incorporating optical flow features and data augmentation techniques, to increase the necessary spatial features of selected keyframes. Finally, we classified the students’ behavior using a deep sequence model based on the bidirectional long short-term memory network with an attention mechanism (BiLSTM-AT). The proposed method with the BiLSTM-AT model can recognize behaviors from our dataset with high accuracy, precision, recall, and F1-scores of 0.97, 0.97, and 0.97, respectively. The overall accuracy was 96.67%. This high efficiency demonstrates the potential of the proposed method for classifying student behavior in classrooms. |
| first_indexed | 2024-03-08T15:11:23Z |
| format | Article |
| id | doaj.art-132ff405e8534685bad02003b76dffc4 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-08T15:11:23Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-132ff405e8534685bad02003b76dffc42024-01-10T14:51:25ZengMDPI AGApplied Sciences2076-34172023-12-0114123010.3390/app14010230MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior ClassificationSomsawut Nindam0Seung-Hoon Na1Hyo Jong Lee2Division of Computer Science and Engineering, CAIIT, Jeonbuk National University, Jeonju 54896, Republic of KoreaDivision of Computer Science and Engineering, CAIIT, Jeonbuk National University, Jeonju 54896, Republic of KoreaDivision of Computer Science and Engineering, CAIIT, Jeonbuk National University, Jeonju 54896, Republic of KoreaThis research proposes a deep learning method for classifying student behavior in classrooms that follow the professional learning community teaching approach. We collected data on five student activities: hand-raising, interacting, sitting, turning around, and writing. We used the sum of absolute differences (SAD) in the LUV color space to detect scene changes. The K-means algorithm was then applied to select keyframes using the computed SAD. Next, we extracted features using multiple pretrained deep learning models from the convolutional neural network family. The pretrained models considered were InceptionV3, ResNet50V2, VGG16, and EfficientNetB7. We leveraged feature fusion, incorporating optical flow features and data augmentation techniques, to increase the necessary spatial features of selected keyframes. Finally, we classified the students’ behavior using a deep sequence model based on the bidirectional long short-term memory network with an attention mechanism (BiLSTM-AT). The proposed method with the BiLSTM-AT model can recognize behaviors from our dataset with high accuracy, precision, recall, and F1-scores of 0.97, 0.97, and 0.97, respectively. The overall accuracy was 96.67%. This high efficiency demonstrates the potential of the proposed method for classifying student behavior in classrooms.https://www.mdpi.com/2076-3417/14/1/230lesson studyprofessional learning communitystudent behaviors classificationvideo classificationkeyframe selectionmulti-feature fusion |
| spellingShingle | Somsawut Nindam Seung-Hoon Na Hyo Jong Lee MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification Applied Sciences lesson study professional learning community student behaviors classification video classification keyframe selection multi-feature fusion |
| title | MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification |
| title_full | MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification |
| title_fullStr | MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification |
| title_full_unstemmed | MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification |
| title_short | MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification |
| title_sort | multifusednet a multi feature fused network of pretrained vision models via keyframes for student behavior classification |
| topic | lesson study professional learning community student behaviors classification video classification keyframe selection multi-feature fusion |
| url | https://www.mdpi.com/2076-3417/14/1/230 |
| work_keys_str_mv | AT somsawutnindam multifusednetamultifeaturefusednetworkofpretrainedvisionmodelsviakeyframesforstudentbehaviorclassification AT seunghoonna multifusednetamultifeaturefusednetworkofpretrainedvisionmodelsviakeyframesforstudentbehaviorclassification AT hyojonglee multifusednetamultifeaturefusednetworkofpretrainedvisionmodelsviakeyframesforstudentbehaviorclassification |