Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory
Recognition of lying is a more complex cognitive process than truth-telling because of the presence of involuntary cognitive cues that are useful to lie recognition. Researchers have proposed different approaches in the literature to solve the problem of lie recognition from either handcrafted and/o...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-03-01
|
| Series: | Brain Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3425/13/4/555 |
| _version_ | 1797606215237238784 |
|---|---|
| author | Sunusi Bala Abdullahi Zakariyya Abdullahi Bature Lubna A. Gabralla Haruna Chiroma |
| author_facet | Sunusi Bala Abdullahi Zakariyya Abdullahi Bature Lubna A. Gabralla Haruna Chiroma |
| author_sort | Sunusi Bala Abdullahi |
| collection | DOAJ |
| description | Recognition of lying is a more complex cognitive process than truth-telling because of the presence of involuntary cognitive cues that are useful to lie recognition. Researchers have proposed different approaches in the literature to solve the problem of lie recognition from either handcrafted and/or automatic lie features during court trials and police interrogations. Unfortunately, due to the cognitive complexity and the lack of involuntary cues related to lying features, the performances of these approaches suffer and their generalization ability is limited. To improve performance, this study proposed state transition patterns based on hands, body motions, and eye blinking features from real-life court trial videos. Each video frame is represented according to a computed threshold value among neighboring pixels to extract spatial–temporal state transition patterns (STSTP) of the hand and face poses as involuntary cues using fully connected convolution neural network layers optimized with the weights of ResNet-152 learning. In addition, this study computed an eye aspect ratio model to obtain eye blinking features. These features were fused together as a single multi-modal STSTP feature model. The model was built using the enhanced calculated weight of bidirectional long short-term memory. The proposed approach was evaluated by comparing its performance with current state-of-the-art methods. It was found that the proposed approach improves the performance of detecting lies. |
| first_indexed | 2024-03-11T05:12:01Z |
| format | Article |
| id | doaj.art-75c189a5e7e54bc994494e0c27dfd720 |
| institution | Directory Open Access Journal |
| issn | 2076-3425 |
| language | English |
| last_indexed | 2024-03-11T05:12:01Z |
| publishDate | 2023-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Brain Sciences |
| spelling | doaj.art-75c189a5e7e54bc994494e0c27dfd7202023-11-17T18:31:52ZengMDPI AGBrain Sciences2076-34252023-03-0113455510.3390/brainsci13040555Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term MemorySunusi Bala Abdullahi0Zakariyya Abdullahi Bature1Lubna A. Gabralla2Haruna Chiroma3Department of Computer Engineering, Faculty of Engineering, King Mongkut’s University of Technology Thonburi, 126 Pracha-Uthit Road, Bang Mod, Thrung Khru, Bangkok 10140, ThailandDepartment of Electrical and Information Engineering, Faculty of Engineering, King Mongkut’s University of Technology Thonburi, 126 Pracha-Uthit Road, Bang Mod, Thrung Khru, Bangkok 10140, ThailandDepartment of Computer Science and Information Technology, Applied College, Princess Nourah Bint Abdulrahman University, Riyadh 11671, Saudi ArabiaCollege of Computer Science and Engineering, University of Hafr Al Batin, Hafar al-Batin 31991, Saudi ArabiaRecognition of lying is a more complex cognitive process than truth-telling because of the presence of involuntary cognitive cues that are useful to lie recognition. Researchers have proposed different approaches in the literature to solve the problem of lie recognition from either handcrafted and/or automatic lie features during court trials and police interrogations. Unfortunately, due to the cognitive complexity and the lack of involuntary cues related to lying features, the performances of these approaches suffer and their generalization ability is limited. To improve performance, this study proposed state transition patterns based on hands, body motions, and eye blinking features from real-life court trial videos. Each video frame is represented according to a computed threshold value among neighboring pixels to extract spatial–temporal state transition patterns (STSTP) of the hand and face poses as involuntary cues using fully connected convolution neural network layers optimized with the weights of ResNet-152 learning. In addition, this study computed an eye aspect ratio model to obtain eye blinking features. These features were fused together as a single multi-modal STSTP feature model. The model was built using the enhanced calculated weight of bidirectional long short-term memory. The proposed approach was evaluated by comparing its performance with current state-of-the-art methods. It was found that the proposed approach improves the performance of detecting lies.https://www.mdpi.com/2076-3425/13/4/555artificial intelligencebidirectional long short-term memoryconvolutional neural networkcomputational intelligenceeye aspect ratiohand gestures |
| spellingShingle | Sunusi Bala Abdullahi Zakariyya Abdullahi Bature Lubna A. Gabralla Haruna Chiroma Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory Brain Sciences artificial intelligence bidirectional long short-term memory convolutional neural network computational intelligence eye aspect ratio hand gestures |
| title | Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory |
| title_full | Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory |
| title_fullStr | Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory |
| title_full_unstemmed | Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory |
| title_short | Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory |
| title_sort | lie recognition with multi modal spatial temporal state transition patterns based on hybrid convolutional neural network bidirectional long short term memory |
| topic | artificial intelligence bidirectional long short-term memory convolutional neural network computational intelligence eye aspect ratio hand gestures |
| url | https://www.mdpi.com/2076-3425/13/4/555 |
| work_keys_str_mv | AT sunusibalaabdullahi lierecognitionwithmultimodalspatialtemporalstatetransitionpatternsbasedonhybridconvolutionalneuralnetworkbidirectionallongshorttermmemory AT zakariyyaabdullahibature lierecognitionwithmultimodalspatialtemporalstatetransitionpatternsbasedonhybridconvolutionalneuralnetworkbidirectionallongshorttermmemory AT lubnaagabralla lierecognitionwithmultimodalspatialtemporalstatetransitionpatternsbasedonhybridconvolutionalneuralnetworkbidirectionallongshorttermmemory AT harunachiroma lierecognitionwithmultimodalspatialtemporalstatetransitionpatternsbasedonhybridconvolutionalneuralnetworkbidirectionallongshorttermmemory |