A Multimodal User Interface for an Assistive Robotic Shopping Cart
This paper presents the research and development of the prototype of the assistive mobile information robot (AMIR). The main features of the presented prototype are voice and gesture-based interfaces with Russian speech and sign language recognition and synthesis techniques and a high degree of robo...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-12-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/9/12/2093 |
| _version_ | 1797545358667022336 |
|---|---|
| author | Dmitry Ryumin Ildar Kagirov Alexandr Axyonov Nikita Pavlyuk Anton Saveliev Irina Kipyatkova Milos Zelezny Iosif Mporas Alexey Karpov |
| author_facet | Dmitry Ryumin Ildar Kagirov Alexandr Axyonov Nikita Pavlyuk Anton Saveliev Irina Kipyatkova Milos Zelezny Iosif Mporas Alexey Karpov |
| author_sort | Dmitry Ryumin |
| collection | DOAJ |
| description | This paper presents the research and development of the prototype of the assistive mobile information robot (AMIR). The main features of the presented prototype are voice and gesture-based interfaces with Russian speech and sign language recognition and synthesis techniques and a high degree of robot autonomy. AMIR prototype’s aim is to be used as a robotic cart for shopping in grocery stores and/or supermarkets. Among the main topics covered in this paper are the presentation of the interface (three modalities), the single-handed gesture recognition system (based on a collected database of Russian sign language elements), as well as the technical description of the robotic platform (architecture, navigation algorithm). The use of multimodal interfaces, namely the speech and gesture modalities, make human-robot interaction natural and intuitive, as well as sign language recognition allows hearing-impaired people to use this robotic cart. AMIR prototype has promising perspectives for real usage in supermarkets, both due to its assistive capabilities and its multimodal user interface. |
| first_indexed | 2024-03-10T14:14:18Z |
| format | Article |
| id | doaj.art-1821ab2a9bc543d4b0b648f5bd11a1d6 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T14:14:18Z |
| publishDate | 2020-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-1821ab2a9bc543d4b0b648f5bd11a1d62023-11-20T23:52:44ZengMDPI AGElectronics2079-92922020-12-01912209310.3390/electronics9122093A Multimodal User Interface for an Assistive Robotic Shopping CartDmitry Ryumin0Ildar Kagirov1Alexandr Axyonov2Nikita Pavlyuk3Anton Saveliev4Irina Kipyatkova5Milos Zelezny6Iosif Mporas7Alexey Karpov8St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaDepartment of Cybernetics, Faculty of Applied Sciences, University of West Bohemia, 301 00 Pilsen, Czech RepublicSchool of Engineering and Computer Science, University of Hertfordshire, Hatfield, Herts AL10 9AB, UKSt. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaThis paper presents the research and development of the prototype of the assistive mobile information robot (AMIR). The main features of the presented prototype are voice and gesture-based interfaces with Russian speech and sign language recognition and synthesis techniques and a high degree of robot autonomy. AMIR prototype’s aim is to be used as a robotic cart for shopping in grocery stores and/or supermarkets. Among the main topics covered in this paper are the presentation of the interface (three modalities), the single-handed gesture recognition system (based on a collected database of Russian sign language elements), as well as the technical description of the robotic platform (architecture, navigation algorithm). The use of multimodal interfaces, namely the speech and gesture modalities, make human-robot interaction natural and intuitive, as well as sign language recognition allows hearing-impaired people to use this robotic cart. AMIR prototype has promising perspectives for real usage in supermarkets, both due to its assistive capabilities and its multimodal user interface.https://www.mdpi.com/2079-9292/9/12/2093assistive roboticsservice roboticsmultimodal user interfacesign language processinggesture interfacespeech recognition |
| spellingShingle | Dmitry Ryumin Ildar Kagirov Alexandr Axyonov Nikita Pavlyuk Anton Saveliev Irina Kipyatkova Milos Zelezny Iosif Mporas Alexey Karpov A Multimodal User Interface for an Assistive Robotic Shopping Cart Electronics assistive robotics service robotics multimodal user interface sign language processing gesture interface speech recognition |
| title | A Multimodal User Interface for an Assistive Robotic Shopping Cart |
| title_full | A Multimodal User Interface for an Assistive Robotic Shopping Cart |
| title_fullStr | A Multimodal User Interface for an Assistive Robotic Shopping Cart |
| title_full_unstemmed | A Multimodal User Interface for an Assistive Robotic Shopping Cart |
| title_short | A Multimodal User Interface for an Assistive Robotic Shopping Cart |
| title_sort | multimodal user interface for an assistive robotic shopping cart |
| topic | assistive robotics service robotics multimodal user interface sign language processing gesture interface speech recognition |
| url | https://www.mdpi.com/2079-9292/9/12/2093 |
| work_keys_str_mv | AT dmitryryumin amultimodaluserinterfaceforanassistiveroboticshoppingcart AT ildarkagirov amultimodaluserinterfaceforanassistiveroboticshoppingcart AT alexandraxyonov amultimodaluserinterfaceforanassistiveroboticshoppingcart AT nikitapavlyuk amultimodaluserinterfaceforanassistiveroboticshoppingcart AT antonsaveliev amultimodaluserinterfaceforanassistiveroboticshoppingcart AT irinakipyatkova amultimodaluserinterfaceforanassistiveroboticshoppingcart AT miloszelezny amultimodaluserinterfaceforanassistiveroboticshoppingcart AT iosifmporas amultimodaluserinterfaceforanassistiveroboticshoppingcart AT alexeykarpov amultimodaluserinterfaceforanassistiveroboticshoppingcart AT dmitryryumin multimodaluserinterfaceforanassistiveroboticshoppingcart AT ildarkagirov multimodaluserinterfaceforanassistiveroboticshoppingcart AT alexandraxyonov multimodaluserinterfaceforanassistiveroboticshoppingcart AT nikitapavlyuk multimodaluserinterfaceforanassistiveroboticshoppingcart AT antonsaveliev multimodaluserinterfaceforanassistiveroboticshoppingcart AT irinakipyatkova multimodaluserinterfaceforanassistiveroboticshoppingcart AT miloszelezny multimodaluserinterfaceforanassistiveroboticshoppingcart AT iosifmporas multimodaluserinterfaceforanassistiveroboticshoppingcart AT alexeykarpov multimodaluserinterfaceforanassistiveroboticshoppingcart |