DRER: Deep Learning–Based Driver’s Real Emotion Recognizer
In intelligent vehicles, it is essential to monitor the driver’s condition; however, recognizing the driver’s emotional state is one of the most challenging and important tasks. Most previous studies focused on facial expression recognition to monitor the driver’s emotional state. However, while dri...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/6/2166 |
| _version_ | 1797540637150543872 |
|---|---|
| author | Geesung Oh Junghwan Ryu Euiseok Jeong Ji Hyun Yang Sungwook Hwang Sangho Lee Sejoon Lim |
| author_facet | Geesung Oh Junghwan Ryu Euiseok Jeong Ji Hyun Yang Sungwook Hwang Sangho Lee Sejoon Lim |
| author_sort | Geesung Oh |
| collection | DOAJ |
| description | In intelligent vehicles, it is essential to monitor the driver’s condition; however, recognizing the driver’s emotional state is one of the most challenging and important tasks. Most previous studies focused on facial expression recognition to monitor the driver’s emotional state. However, while driving, many factors are preventing the drivers from revealing the emotions on their faces. To address this problem, we propose a deep learning-based driver’s real emotion recognizer (DRER), which is a deep learning-based algorithm to recognize the drivers’ real emotions that cannot be completely identified based on their facial expressions. The proposed algorithm comprises of two models: (i) facial expression recognition model, which refers to the state-of-the-art convolutional neural network structure; and (ii) sensor fusion emotion recognition model, which fuses the recognized state of facial expressions with electrodermal activity, a bio-physiological signal representing electrical characteristics of the skin, in recognizing even the driver’s real emotional state. Hence, we categorized the driver’s emotion and conducted human-in-the-loop experiments to acquire the data. Experimental results show that the proposed fusing approach achieves 114% increase in accuracy compared to using only the facial expressions and 146% increase in accuracy compare to using only the electrodermal activity. In conclusion, our proposed method achieves 86.8% recognition accuracy in recognizing the driver’s induced emotion while driving situation. |
| first_indexed | 2024-03-10T13:03:59Z |
| format | Article |
| id | doaj.art-95800ceaab8c404b87cb02018e9897d7 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T13:03:59Z |
| publishDate | 2021-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-95800ceaab8c404b87cb02018e9897d72023-11-21T11:15:51ZengMDPI AGSensors1424-82202021-03-01216216610.3390/s21062166DRER: Deep Learning–Based Driver’s Real Emotion RecognizerGeesung Oh0Junghwan Ryu1Euiseok Jeong2Ji Hyun Yang3Sungwook Hwang4Sangho Lee5Sejoon Lim6Graduate School of Automotive Engineering, Kookmin University, 77, Jeongneung-ro, Seongbuk-gu, Seoul 02707, KoreaGraduate School of Automotive Engineering, Kookmin University, 77, Jeongneung-ro, Seongbuk-gu, Seoul 02707, KoreaGraduate School of Automotive Engineering, Kookmin University, 77, Jeongneung-ro, Seongbuk-gu, Seoul 02707, KoreaDepartment of Automobile and IT Convergence, Kookmin University, 77, Jeongneung-ro, Seongbuk-gu, Seoul 02707, KoreaChassis System Control Research Lab, Hyundai Motor Group, Hwaseong 18280, KoreaChassis System Control Research Lab, Hyundai Motor Group, Hwaseong 18280, KoreaDepartment of Automobile and IT Convergence, Kookmin University, 77, Jeongneung-ro, Seongbuk-gu, Seoul 02707, KoreaIn intelligent vehicles, it is essential to monitor the driver’s condition; however, recognizing the driver’s emotional state is one of the most challenging and important tasks. Most previous studies focused on facial expression recognition to monitor the driver’s emotional state. However, while driving, many factors are preventing the drivers from revealing the emotions on their faces. To address this problem, we propose a deep learning-based driver’s real emotion recognizer (DRER), which is a deep learning-based algorithm to recognize the drivers’ real emotions that cannot be completely identified based on their facial expressions. The proposed algorithm comprises of two models: (i) facial expression recognition model, which refers to the state-of-the-art convolutional neural network structure; and (ii) sensor fusion emotion recognition model, which fuses the recognized state of facial expressions with electrodermal activity, a bio-physiological signal representing electrical characteristics of the skin, in recognizing even the driver’s real emotional state. Hence, we categorized the driver’s emotion and conducted human-in-the-loop experiments to acquire the data. Experimental results show that the proposed fusing approach achieves 114% increase in accuracy compared to using only the facial expressions and 146% increase in accuracy compare to using only the electrodermal activity. In conclusion, our proposed method achieves 86.8% recognition accuracy in recognizing the driver’s induced emotion while driving situation.https://www.mdpi.com/1424-8220/21/6/2166human–machine interfaceemotion recognitionreal emotiondriver’s emotional statedeep learningsensor fusion |
| spellingShingle | Geesung Oh Junghwan Ryu Euiseok Jeong Ji Hyun Yang Sungwook Hwang Sangho Lee Sejoon Lim DRER: Deep Learning–Based Driver’s Real Emotion Recognizer Sensors human–machine interface emotion recognition real emotion driver’s emotional state deep learning sensor fusion |
| title | DRER: Deep Learning–Based Driver’s Real Emotion Recognizer |
| title_full | DRER: Deep Learning–Based Driver’s Real Emotion Recognizer |
| title_fullStr | DRER: Deep Learning–Based Driver’s Real Emotion Recognizer |
| title_full_unstemmed | DRER: Deep Learning–Based Driver’s Real Emotion Recognizer |
| title_short | DRER: Deep Learning–Based Driver’s Real Emotion Recognizer |
| title_sort | drer deep learning based driver s real emotion recognizer |
| topic | human–machine interface emotion recognition real emotion driver’s emotional state deep learning sensor fusion |
| url | https://www.mdpi.com/1424-8220/21/6/2166 |
| work_keys_str_mv | AT geesungoh drerdeeplearningbaseddriversrealemotionrecognizer AT junghwanryu drerdeeplearningbaseddriversrealemotionrecognizer AT euiseokjeong drerdeeplearningbaseddriversrealemotionrecognizer AT jihyunyang drerdeeplearningbaseddriversrealemotionrecognizer AT sungwookhwang drerdeeplearningbaseddriversrealemotionrecognizer AT sangholee drerdeeplearningbaseddriversrealemotionrecognizer AT sejoonlim drerdeeplearningbaseddriversrealemotionrecognizer |