Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks
In this work, we present a method for personalizing human-robot interaction by using emotive facial expressions to generate affective robot movements. Movement is an important medium for robots to communicate affective states, but the expertise and time required to craft new robot movements promotes...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
ACM
2023
|
| Online Access: | https://hdl.handle.net/1721.1/152915 |
| _version_ | 1811085419054366720 |
|---|---|
| author | Suguitan, Michael DePalma, Nicholas Hoffman, Guy Hodgins, Jessica |
| author2 | Massachusetts Institute of Technology. Media Laboratory |
| author_facet | Massachusetts Institute of Technology. Media Laboratory Suguitan, Michael DePalma, Nicholas Hoffman, Guy Hodgins, Jessica |
| author_sort | Suguitan, Michael |
| collection | MIT |
| description | In this work, we present a method for personalizing human-robot interaction by using emotive facial expressions to generate affective robot movements. Movement is an important medium for robots to communicate affective states, but the expertise and time required to craft new robot movements promotes a reliance on fixed preprogrammed behaviors. Enabling robots to respond to multimodal user input with newly generated movements could stave off staleness of interaction and convey a deeper degree of affective understanding than current retrieval-based methods. We use autoencoder neural networks to compress robot movement data and facial expression images into a shared latent embedding space. Then, we use a reconstruction loss to generate movements from these embeddings and triplet loss to align the embeddings by emotion classes rather than data modality. To subjectively evaluate our method, we conducted a user survey and found that generated happy and sad movements could be matched to their source face images. However, angry movements were most often mismatched to sad images. This multimodal data-driven generative method can expand an interactive agent's behavior library and could be adopted for other multimodal affective applications. |
| first_indexed | 2024-09-23T13:09:17Z |
| format | Article |
| id | mit-1721.1/152915 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:09:17Z |
| publishDate | 2023 |
| publisher | ACM |
| record_format | dspace |
| spelling | mit-1721.1/1529152024-01-11T20:43:12Z Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks Suguitan, Michael DePalma, Nicholas Hoffman, Guy Hodgins, Jessica Massachusetts Institute of Technology. Media Laboratory In this work, we present a method for personalizing human-robot interaction by using emotive facial expressions to generate affective robot movements. Movement is an important medium for robots to communicate affective states, but the expertise and time required to craft new robot movements promotes a reliance on fixed preprogrammed behaviors. Enabling robots to respond to multimodal user input with newly generated movements could stave off staleness of interaction and convey a deeper degree of affective understanding than current retrieval-based methods. We use autoencoder neural networks to compress robot movement data and facial expression images into a shared latent embedding space. Then, we use a reconstruction loss to generate movements from these embeddings and triplet loss to align the embeddings by emotion classes rather than data modality. To subjectively evaluate our method, we conducted a user survey and found that generated happy and sad movements could be matched to their source face images. However, angry movements were most often mismatched to sad images. This multimodal data-driven generative method can expand an interactive agent's behavior library and could be adopted for other multimodal affective applications. 2023-11-06T19:03:10Z 2023-11-06T19:03:10Z 2023-11-01T07:58:12Z Article http://purl.org/eprint/type/JournalArticle 2573-9522 https://hdl.handle.net/1721.1/152915 Suguitan, Michael, DePalma, Nicholas, Hoffman, Guy and Hodgins, Jessica. "Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks." ACM Transactions on Human-Robot Interaction. PUBLISHER_CC en https://doi.org/10.1145/3623386 ACM Transactions on Human-Robot Interaction Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. The author(s) application/pdf ACM Association for Computing Machinery |
| spellingShingle | Suguitan, Michael DePalma, Nicholas Hoffman, Guy Hodgins, Jessica Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks |
| title | Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks |
| title_full | Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks |
| title_fullStr | Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks |
| title_full_unstemmed | Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks |
| title_short | Face2Gesture: Translating Facial Expressions Into Robot Movements Through Shared Latent Space Neural Networks |
| title_sort | face2gesture translating facial expressions into robot movements through shared latent space neural networks |
| url | https://hdl.handle.net/1721.1/152915 |
| work_keys_str_mv | AT suguitanmichael face2gesturetranslatingfacialexpressionsintorobotmovementsthroughsharedlatentspaceneuralnetworks AT depalmanicholas face2gesturetranslatingfacialexpressionsintorobotmovementsthroughsharedlatentspaceneuralnetworks AT hoffmanguy face2gesturetranslatingfacialexpressionsintorobotmovementsthroughsharedlatentspaceneuralnetworks AT hodginsjessica face2gesturetranslatingfacialexpressionsintorobotmovementsthroughsharedlatentspaceneuralnetworks |