Human emotion recognition from videos using spatio-temporal and audio features
In this paper, we present human emotion recognition systems based on audio and spatio-temporal visual features. The proposed system has been tested on audio visual emotion data set with different subjects for both genders. The mel-frequency cepstral coefficient (MFCC) and prosodic features are first...
| Main Authors: | , , |
|---|---|
| Format: | Conference or Workshop Item |
| Published: |
2013
|
| Subjects: |
| _version_ | 1796859575792566272 |
|---|---|
| author | Rashid, M. Abu-Bakar, S. A. R. Mokji, M. |
| author_facet | Rashid, M. Abu-Bakar, S. A. R. Mokji, M. |
| author_sort | Rashid, M. |
| collection | ePrints |
| description | In this paper, we present human emotion recognition systems based on audio and spatio-temporal visual features. The proposed system has been tested on audio visual emotion data set with different subjects for both genders. The mel-frequency cepstral coefficient (MFCC) and prosodic features are first identified and then extracted from emotional speech. For facial expressions spatio-temporal features are extracted from visual streams. Principal component analysis (PCA) is applied for dimensionality reduction of the visual features and capturing 97 % of variances. Codebook is constructed for both audio and visual features using Euclidean space. Then occurrences of the histograms are employed as input to the state-of-the-art SVM classifier to realize the judgment of each classifier. Moreover, the judgments from each classifier are combined using Bayes sum rule (BSR) as a final decision step. The proposed system is tested on public data set to recognize the human emotions. Experimental results and simulations proved that using visual features only yields on average 74.15 % accuracy, while using audio features only gives recognition average accuracy of 67.39 %. Whereas by combining both audio and visual features, the overall system accuracy has been significantly improved up to 80.27 %. |
| first_indexed | 2024-03-05T19:29:12Z |
| format | Conference or Workshop Item |
| id | utm.eprints-51104 |
| institution | Universiti Teknologi Malaysia - ePrints |
| last_indexed | 2024-03-05T19:29:12Z |
| publishDate | 2013 |
| record_format | dspace |
| spelling | utm.eprints-511042017-08-20T04:08:01Z http://eprints.utm.my/51104/ Human emotion recognition from videos using spatio-temporal and audio features Rashid, M. Abu-Bakar, S. A. R. Mokji, M. TK Electrical engineering. Electronics Nuclear engineering In this paper, we present human emotion recognition systems based on audio and spatio-temporal visual features. The proposed system has been tested on audio visual emotion data set with different subjects for both genders. The mel-frequency cepstral coefficient (MFCC) and prosodic features are first identified and then extracted from emotional speech. For facial expressions spatio-temporal features are extracted from visual streams. Principal component analysis (PCA) is applied for dimensionality reduction of the visual features and capturing 97 % of variances. Codebook is constructed for both audio and visual features using Euclidean space. Then occurrences of the histograms are employed as input to the state-of-the-art SVM classifier to realize the judgment of each classifier. Moreover, the judgments from each classifier are combined using Bayes sum rule (BSR) as a final decision step. The proposed system is tested on public data set to recognize the human emotions. Experimental results and simulations proved that using visual features only yields on average 74.15 % accuracy, while using audio features only gives recognition average accuracy of 67.39 %. Whereas by combining both audio and visual features, the overall system accuracy has been significantly improved up to 80.27 %. 2013 Conference or Workshop Item PeerReviewed Rashid, M. and Abu-Bakar, S. A. R. and Mokji, M. (2013) Human emotion recognition from videos using spatio-temporal and audio features. In: Visual Computer. http://dx.doi.org/ 10.1007/s00371-012-0768-y |
| spellingShingle | TK Electrical engineering. Electronics Nuclear engineering Rashid, M. Abu-Bakar, S. A. R. Mokji, M. Human emotion recognition from videos using spatio-temporal and audio features |
| title | Human emotion recognition from videos using spatio-temporal and audio features |
| title_full | Human emotion recognition from videos using spatio-temporal and audio features |
| title_fullStr | Human emotion recognition from videos using spatio-temporal and audio features |
| title_full_unstemmed | Human emotion recognition from videos using spatio-temporal and audio features |
| title_short | Human emotion recognition from videos using spatio-temporal and audio features |
| title_sort | human emotion recognition from videos using spatio temporal and audio features |
| topic | TK Electrical engineering. Electronics Nuclear engineering |
| work_keys_str_mv | AT rashidm humanemotionrecognitionfromvideosusingspatiotemporalandaudiofeatures AT abubakarsar humanemotionrecognitionfromvideosusingspatiotemporalandaudiofeatures AT mokjim humanemotionrecognitionfromvideosusingspatiotemporalandaudiofeatures |