Nodding detection system based on head motion and voice rhythm
Human beings communicate using more than verbal communication. To communicate smoothly, humans tend to use nonverbal information such as nodding during embodied interactions. Nodding is a form of affirmative gesture that is commonly used by humans worldwide while communicating with each other. The a...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Japan Society of Mechanical Engineers
2021-01-01
|
| Series: | Journal of Advanced Mechanical Design, Systems, and Manufacturing |
| Subjects: | |
| Online Access: | https://www.jstage.jst.go.jp/article/jamdsm/15/1/15_2021jamdsm0005/_pdf/-char/en |
| _version_ | 1811225733061672960 |
|---|---|
| author | Shunsuke OTA Seiko TAKI Mitsuru JINDAI Toshiyuki YASUDA |
| author_facet | Shunsuke OTA Seiko TAKI Mitsuru JINDAI Toshiyuki YASUDA |
| author_sort | Shunsuke OTA |
| collection | DOAJ |
| description | Human beings communicate using more than verbal communication. To communicate smoothly, humans tend to use nonverbal information such as nodding during embodied interactions. Nodding is a form of affirmative gesture that is commonly used by humans worldwide while communicating with each other. The accurate detection of nodding is expected to support communication, analyze a conversation scene, and facilitate embodied interaction. In contrast, nodding is known to be related not only to the motion of the head but also to the rhythm of the voice. By using both head motion and voice rhythms, nodding can be estimated more accurately than methods that use only head motion or voice rhythms. Therefore, in this study, we develop a nodding detection system based on head motion and voice rhythm. In this system, the head motion of the listener is measured using face tracking. Then, the nodding motion is estimated using a neural network for head motion. Furthermore, the neural networks estimate the timings at which the listener is nodding by using the voice of the speaker. Subsequently, the nodding is estimated using logical OR and logical AND based on outputs of the head movement and speech rhythm neural networks. In addition, a neural network that integrates head motion and voice rhythm is developed. Furthermore, the effectiveness of the developed methods is demonstrated through evaluation experiments. |
| first_indexed | 2024-04-12T09:13:31Z |
| format | Article |
| id | doaj.art-2cc8bdc6907c49ed988c4646a709dade |
| institution | Directory Open Access Journal |
| issn | 1881-3054 |
| language | English |
| last_indexed | 2024-04-12T09:13:31Z |
| publishDate | 2021-01-01 |
| publisher | The Japan Society of Mechanical Engineers |
| record_format | Article |
| series | Journal of Advanced Mechanical Design, Systems, and Manufacturing |
| spelling | doaj.art-2cc8bdc6907c49ed988c4646a709dade2022-12-22T03:38:55ZengThe Japan Society of Mechanical EngineersJournal of Advanced Mechanical Design, Systems, and Manufacturing1881-30542021-01-01151JAMDSM0005JAMDSM000510.1299/jamdsm.2021jamdsm0005jamdsmNodding detection system based on head motion and voice rhythmShunsuke OTA0Seiko TAKI1Mitsuru JINDAI2Toshiyuki YASUDA3Faculty of Computer Science and System Engineering, Okayama Prefectural UniversityFaculty of Social Systems Science, Chiba Institute of TechnologyFaculty of Engineering, University of ToyamaFaculty of Engineering, University of ToyamaHuman beings communicate using more than verbal communication. To communicate smoothly, humans tend to use nonverbal information such as nodding during embodied interactions. Nodding is a form of affirmative gesture that is commonly used by humans worldwide while communicating with each other. The accurate detection of nodding is expected to support communication, analyze a conversation scene, and facilitate embodied interaction. In contrast, nodding is known to be related not only to the motion of the head but also to the rhythm of the voice. By using both head motion and voice rhythms, nodding can be estimated more accurately than methods that use only head motion or voice rhythms. Therefore, in this study, we develop a nodding detection system based on head motion and voice rhythm. In this system, the head motion of the listener is measured using face tracking. Then, the nodding motion is estimated using a neural network for head motion. Furthermore, the neural networks estimate the timings at which the listener is nodding by using the voice of the speaker. Subsequently, the nodding is estimated using logical OR and logical AND based on outputs of the head movement and speech rhythm neural networks. In addition, a neural network that integrates head motion and voice rhythm is developed. Furthermore, the effectiveness of the developed methods is demonstrated through evaluation experiments.https://www.jstage.jst.go.jp/article/jamdsm/15/1/15_2021jamdsm0005/_pdf/-char/enhuman interfacehuman interactionhuman-machine communicationnodding detectionembodied interactionneural network |
| spellingShingle | Shunsuke OTA Seiko TAKI Mitsuru JINDAI Toshiyuki YASUDA Nodding detection system based on head motion and voice rhythm Journal of Advanced Mechanical Design, Systems, and Manufacturing human interface human interaction human-machine communication nodding detection embodied interaction neural network |
| title | Nodding detection system based on head motion and voice rhythm |
| title_full | Nodding detection system based on head motion and voice rhythm |
| title_fullStr | Nodding detection system based on head motion and voice rhythm |
| title_full_unstemmed | Nodding detection system based on head motion and voice rhythm |
| title_short | Nodding detection system based on head motion and voice rhythm |
| title_sort | nodding detection system based on head motion and voice rhythm |
| topic | human interface human interaction human-machine communication nodding detection embodied interaction neural network |
| url | https://www.jstage.jst.go.jp/article/jamdsm/15/1/15_2021jamdsm0005/_pdf/-char/en |
| work_keys_str_mv | AT shunsukeota noddingdetectionsystembasedonheadmotionandvoicerhythm AT seikotaki noddingdetectionsystembasedonheadmotionandvoicerhythm AT mitsurujindai noddingdetectionsystembasedonheadmotionandvoicerhythm AT toshiyukiyasuda noddingdetectionsystembasedonheadmotionandvoicerhythm |