Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion
Existing vision-based fatigue detection methods commonly utilize RGB cameras to extract facial and physiological features for monitoring driver fatigue. These features often include single indicators such as eyelid movement, yawning frequency, and heart rate. However, the accuracy of RGB cameras can...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-01-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/2/455 |
| _version_ | 1797339398687162368 |
|---|---|
| author | Lingjian Kong Kai Xie Kaixuan Niu Jianbiao He Wei Zhang |
| author_facet | Lingjian Kong Kai Xie Kaixuan Niu Jianbiao He Wei Zhang |
| author_sort | Lingjian Kong |
| collection | DOAJ |
| description | Existing vision-based fatigue detection methods commonly utilize RGB cameras to extract facial and physiological features for monitoring driver fatigue. These features often include single indicators such as eyelid movement, yawning frequency, and heart rate. However, the accuracy of RGB cameras can be affected by factors like varying lighting conditions and motion. To address these challenges, we propose a non-invasive method for multi-modal fusion fatigue detection called RPPMT-CNN-BiLSTM. This method incorporates a feature extraction enhancement module based on the improved Pan–Tompkins algorithm and 1D-MTCNN. This enhances the accuracy of heart rate signal extraction and eyelid features. Furthermore, we use one-dimensional neural networks to construct two models based on heart rate and PERCLOS values, forming a fatigue detection model. To enhance the robustness and accuracy of fatigue detection, the trained model data results are input into the BiLSTM network. This generates a time-fitting relationship between the data extracted from the CNN, allowing for effective dynamic modeling and achieving multi-modal fusion fatigue detection. Numerous experiments validate the effectiveness of the proposed method, achieving an accuracy of 98.2% on the self-made MDAD (Multi-Modal Driver Alertness Dataset). This underscores the feasibility of the algorithm. In comparison with traditional methods, our approach demonstrates higher accuracy and positively contributes to maintaining traffic safety, thereby advancing the field of smart transportation. |
| first_indexed | 2024-03-08T09:47:19Z |
| format | Article |
| id | doaj.art-36dae800f24d4c91985c434666ecf8f5 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-08T09:47:19Z |
| publishDate | 2024-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-36dae800f24d4c91985c434666ecf8f52024-01-29T14:14:54ZengMDPI AGSensors1424-82202024-01-0124245510.3390/s24020455Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal FusionLingjian Kong0Kai Xie1Kaixuan Niu2Jianbiao He3Wei Zhang4School of Electronic Information and Electrical Engineering, Yangtze University, Jingzhou 434023, ChinaSchool of Electronic Information and Electrical Engineering, Yangtze University, Jingzhou 434023, ChinaSchool of Electronic Information and Electrical Engineering, Yangtze University, Jingzhou 434023, ChinaSchool of Computer Science, Central South University, Changsha 410083, ChinaSchool of Electronic Information, Central South University, Changsha 410083, ChinaExisting vision-based fatigue detection methods commonly utilize RGB cameras to extract facial and physiological features for monitoring driver fatigue. These features often include single indicators such as eyelid movement, yawning frequency, and heart rate. However, the accuracy of RGB cameras can be affected by factors like varying lighting conditions and motion. To address these challenges, we propose a non-invasive method for multi-modal fusion fatigue detection called RPPMT-CNN-BiLSTM. This method incorporates a feature extraction enhancement module based on the improved Pan–Tompkins algorithm and 1D-MTCNN. This enhances the accuracy of heart rate signal extraction and eyelid features. Furthermore, we use one-dimensional neural networks to construct two models based on heart rate and PERCLOS values, forming a fatigue detection model. To enhance the robustness and accuracy of fatigue detection, the trained model data results are input into the BiLSTM network. This generates a time-fitting relationship between the data extracted from the CNN, allowing for effective dynamic modeling and achieving multi-modal fusion fatigue detection. Numerous experiments validate the effectiveness of the proposed method, achieving an accuracy of 98.2% on the self-made MDAD (Multi-Modal Driver Alertness Dataset). This underscores the feasibility of the algorithm. In comparison with traditional methods, our approach demonstrates higher accuracy and positively contributes to maintaining traffic safety, thereby advancing the field of smart transportation.https://www.mdpi.com/1424-8220/24/2/455intelligent trafficfatigue detectionmulti-modal feature fusionheart ratebidirectional LSTM |
| spellingShingle | Lingjian Kong Kai Xie Kaixuan Niu Jianbiao He Wei Zhang Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion Sensors intelligent traffic fatigue detection multi-modal feature fusion heart rate bidirectional LSTM |
| title | Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion |
| title_full | Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion |
| title_fullStr | Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion |
| title_full_unstemmed | Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion |
| title_short | Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion |
| title_sort | remote photoplethysmography and motion tracking convolutional neural network with bidirectional long short term memory non invasive fatigue detection method based on multi modal fusion |
| topic | intelligent traffic fatigue detection multi-modal feature fusion heart rate bidirectional LSTM |
| url | https://www.mdpi.com/1424-8220/24/2/455 |
| work_keys_str_mv | AT lingjiankong remotephotoplethysmographyandmotiontrackingconvolutionalneuralnetworkwithbidirectionallongshorttermmemorynoninvasivefatiguedetectionmethodbasedonmultimodalfusion AT kaixie remotephotoplethysmographyandmotiontrackingconvolutionalneuralnetworkwithbidirectionallongshorttermmemorynoninvasivefatiguedetectionmethodbasedonmultimodalfusion AT kaixuanniu remotephotoplethysmographyandmotiontrackingconvolutionalneuralnetworkwithbidirectionallongshorttermmemorynoninvasivefatiguedetectionmethodbasedonmultimodalfusion AT jianbiaohe remotephotoplethysmographyandmotiontrackingconvolutionalneuralnetworkwithbidirectionallongshorttermmemorynoninvasivefatiguedetectionmethodbasedonmultimodalfusion AT weizhang remotephotoplethysmographyandmotiontrackingconvolutionalneuralnetworkwithbidirectionallongshorttermmemorynoninvasivefatiguedetectionmethodbasedonmultimodalfusion |