Multimodal deception detection in videos
The ability to identify if a person is lying is immensely powerful, to the extent that some might call it a superpower. There have been many approaches to the problem of deception detection, which include psychological, physiological and even machine learning methods. Deception detection has been su...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Final Year Project (FYP) |
| Language: | Spanish / Castilian |
| Published: |
Nanyang Technological University
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/167743 |
| _version_ | 1824456844997820416 |
|---|---|
| author | Syazwan Bin Jainal |
| author2 | Alex Chichung Kot |
| author_facet | Alex Chichung Kot Syazwan Bin Jainal |
| author_sort | Syazwan Bin Jainal |
| collection | NTU |
| description | The ability to identify if a person is lying is immensely powerful, to the extent that some might call it a superpower. There have been many approaches to the problem of deception detection, which include psychological, physiological and even machine learning methods. Deception detection has been successful in high-stakes situations, like courtrooms, where subjects are put under a stressful situation and experiments have yielded an accuracy of over 90%. However, in low-stakes situations, the results are often not much better than a random guess with correct results obtained only around 60% of the time. Moreover, existing datasets on deception detection cannot be generalised into the Singapore context, due to the difference in ethnicities of the subjects. This project builds a deception detection dataset and develops a multimodal deception detection detector based on the Convolutional Neural Network (CNN) deep learning architecture, and explores several loss functions to train the CNN model. It takes advantage of the multimodal nature of video data and applies different fusion methods on the visual and audio features. Experimental results have shown that fusion methods such as Multi-Layer Perceptron Mixer and employing loss functions such as Focal Loss can yield a 6% relative increase in accuracy over the simple concatenation fusion method and cross-entropy loss function. |
| first_indexed | 2025-02-19T04:00:34Z |
| format | Final Year Project (FYP) |
| id | ntu-10356/167743 |
| institution | Nanyang Technological University |
| language | Spanish / Castilian |
| last_indexed | 2025-02-19T04:00:34Z |
| publishDate | 2023 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1677432023-07-07T15:41:10Z Multimodal deception detection in videos Syazwan Bin Jainal Alex Chichung Kot School of Electrical and Electronic Engineering Defense Science Organisation Rapid-Rich Object Search (ROSE) Lab EACKOT@ntu.edu.sg Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence The ability to identify if a person is lying is immensely powerful, to the extent that some might call it a superpower. There have been many approaches to the problem of deception detection, which include psychological, physiological and even machine learning methods. Deception detection has been successful in high-stakes situations, like courtrooms, where subjects are put under a stressful situation and experiments have yielded an accuracy of over 90%. However, in low-stakes situations, the results are often not much better than a random guess with correct results obtained only around 60% of the time. Moreover, existing datasets on deception detection cannot be generalised into the Singapore context, due to the difference in ethnicities of the subjects. This project builds a deception detection dataset and develops a multimodal deception detection detector based on the Convolutional Neural Network (CNN) deep learning architecture, and explores several loss functions to train the CNN model. It takes advantage of the multimodal nature of video data and applies different fusion methods on the visual and audio features. Experimental results have shown that fusion methods such as Multi-Layer Perceptron Mixer and employing loss functions such as Focal Loss can yield a 6% relative increase in accuracy over the simple concatenation fusion method and cross-entropy loss function. Bachelor of Engineering (Electrical and Electronic Engineering) 2023-06-04T11:04:38Z 2023-06-04T11:04:38Z 2023 Final Year Project (FYP) Syazwan Bin Jainal (2023). Multimodal deception detection in videos. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/167743 https://hdl.handle.net/10356/167743 es B3001-221 application/pdf Nanyang Technological University |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence Syazwan Bin Jainal Multimodal deception detection in videos |
| title | Multimodal deception detection in videos |
| title_full | Multimodal deception detection in videos |
| title_fullStr | Multimodal deception detection in videos |
| title_full_unstemmed | Multimodal deception detection in videos |
| title_short | Multimodal deception detection in videos |
| title_sort | multimodal deception detection in videos |
| topic | Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence |
| url | https://hdl.handle.net/10356/167743 |
| work_keys_str_mv | AT syazwanbinjainal multimodaldeceptiondetectioninvideos |