Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism
Abstract Driver activity engagement while driving plays a vital role that leads to negative outcomes of driving safety. To reduce traffic accidents and ensure driving safety, real‐time driver activity recognition architecture is proposed in this study. Specifically, a total of eight kinds of common...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2021-02-01
|
| Series: | IET Intelligent Transport Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1049/itr2.12025 |
| _version_ | 1798035562391666688 |
|---|---|
| author | Chaopeng Pan Haotian Cao Weiwei Zhang Xiaolin Song Mingjun Li |
| author_facet | Chaopeng Pan Haotian Cao Weiwei Zhang Xiaolin Song Mingjun Li |
| author_sort | Chaopeng Pan |
| collection | DOAJ |
| description | Abstract Driver activity engagement while driving plays a vital role that leads to negative outcomes of driving safety. To reduce traffic accidents and ensure driving safety, real‐time driver activity recognition architecture is proposed in this study. Specifically, a total of eight kinds of common driving‐related activities are identified, which include the normal driving, left or right checking, texting, answering the phone, using media, drinking, and picking up objects. Raw experiment videos are collected via onboard monocular cameras, which are used for the upper body skeleton information extraction of the driver. Then, the graph convolutional networks (GCN) are constructed for spatial structure feature reasoning in a single frame, which is consecutively followed by long short‐term memory (LSTM) networks for temporal motion feature learning within the sequence. Moreover, the attention mechanism is further utilised to emphasise the keyframes to select discriminative sequential information. Finally, a large‐scale driver activity dataset, consisting of both naturalistic driving data and simulative driving data, is collected for model training and evaluations. Experimental results show that the general recall ratio of those eight driving‐related activities reaches up to 88.8% and the real‐time recognition efficiency can reach up to 24 fps, which would satisfy the real‐time requirements of engineering applications. |
| first_indexed | 2024-04-11T20:59:55Z |
| format | Article |
| id | doaj.art-239fd3ba711f44d0828c9707f3405c1b |
| institution | Directory Open Access Journal |
| issn | 1751-956X 1751-9578 |
| language | English |
| last_indexed | 2024-04-11T20:59:55Z |
| publishDate | 2021-02-01 |
| publisher | Wiley |
| record_format | Article |
| series | IET Intelligent Transport Systems |
| spelling | doaj.art-239fd3ba711f44d0828c9707f3405c1b2022-12-22T04:03:33ZengWileyIET Intelligent Transport Systems1751-956X1751-95782021-02-0115229730710.1049/itr2.12025Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanismChaopeng Pan0Haotian Cao1Weiwei Zhang2Xiaolin Song3Mingjun Li4State Key Laboratory of Advanced Design and Manufacturing for Vehicle Body Hunan University Changsha ChinaState Key Laboratory of Advanced Design and Manufacturing for Vehicle Body Hunan University Changsha ChinaCollege of Automotive Engineering Shanghai University of Engineering Science Shanghai ChinaState Key Laboratory of Advanced Design and Manufacturing for Vehicle Body Hunan University Changsha ChinaState Key Laboratory of Advanced Design and Manufacturing for Vehicle Body Hunan University Changsha ChinaAbstract Driver activity engagement while driving plays a vital role that leads to negative outcomes of driving safety. To reduce traffic accidents and ensure driving safety, real‐time driver activity recognition architecture is proposed in this study. Specifically, a total of eight kinds of common driving‐related activities are identified, which include the normal driving, left or right checking, texting, answering the phone, using media, drinking, and picking up objects. Raw experiment videos are collected via onboard monocular cameras, which are used for the upper body skeleton information extraction of the driver. Then, the graph convolutional networks (GCN) are constructed for spatial structure feature reasoning in a single frame, which is consecutively followed by long short‐term memory (LSTM) networks for temporal motion feature learning within the sequence. Moreover, the attention mechanism is further utilised to emphasise the keyframes to select discriminative sequential information. Finally, a large‐scale driver activity dataset, consisting of both naturalistic driving data and simulative driving data, is collected for model training and evaluations. Experimental results show that the general recall ratio of those eight driving‐related activities reaches up to 88.8% and the real‐time recognition efficiency can reach up to 24 fps, which would satisfy the real‐time requirements of engineering applications.https://doi.org/10.1049/itr2.12025Optical, image and video signal processingImage recognitionComputer vision and image processing techniquesVideo signal processingTraffic engineering computing |
| spellingShingle | Chaopeng Pan Haotian Cao Weiwei Zhang Xiaolin Song Mingjun Li Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism IET Intelligent Transport Systems Optical, image and video signal processing Image recognition Computer vision and image processing techniques Video signal processing Traffic engineering computing |
| title | Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism |
| title_full | Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism |
| title_fullStr | Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism |
| title_full_unstemmed | Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism |
| title_short | Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism |
| title_sort | driver activity recognition using spatial temporal graph convolutional lstm networks with attention mechanism |
| topic | Optical, image and video signal processing Image recognition Computer vision and image processing techniques Video signal processing Traffic engineering computing |
| url | https://doi.org/10.1049/itr2.12025 |
| work_keys_str_mv | AT chaopengpan driveractivityrecognitionusingspatialtemporalgraphconvolutionallstmnetworkswithattentionmechanism AT haotiancao driveractivityrecognitionusingspatialtemporalgraphconvolutionallstmnetworkswithattentionmechanism AT weiweizhang driveractivityrecognitionusingspatialtemporalgraphconvolutionallstmnetworkswithattentionmechanism AT xiaolinsong driveractivityrecognitionusingspatialtemporalgraphconvolutionallstmnetworkswithattentionmechanism AT mingjunli driveractivityrecognitionusingspatialtemporalgraphconvolutionallstmnetworkswithattentionmechanism |