Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception
The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robo...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2024-01-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1324831/full |
| _version_ | 1827369202073206784 |
|---|---|
| author | Lu Wang |
| author_facet | Lu Wang |
| author_sort | Lu Wang |
| collection | DOAJ |
| description | The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector. |
| first_indexed | 2024-03-08T09:41:49Z |
| format | Article |
| id | doaj.art-78c7401b567a4c62874ee80dd031bae9 |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-03-08T09:41:49Z |
| publishDate | 2024-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-78c7401b567a4c62874ee80dd031bae92024-01-30T04:15:24ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182024-01-011710.3389/fnbot.2023.13248311324831Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perceptionLu WangThe field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector.https://www.frontiersin.org/articles/10.3389/fnbot.2023.1324831/fullemotionmusicmultimodal robotGRUGoogLeNet |
| spellingShingle | Lu Wang Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception Frontiers in Neurorobotics emotion music multimodal robot GRU GoogLeNet |
| title | Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception |
| title_full | Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception |
| title_fullStr | Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception |
| title_full_unstemmed | Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception |
| title_short | Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception |
| title_sort | multimodal robotic music performance art based on gru googlenet model fusing audiovisual perception |
| topic | emotion music multimodal robot GRU GoogLeNet |
| url | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1324831/full |
| work_keys_str_mv | AT luwang multimodalroboticmusicperformanceartbasedongrugooglenetmodelfusingaudiovisualperception |