MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment
We present an Augmented Reality (AR) visualization and interaction tool for users to control Internet of Things (IoT) devices with hand gestures. Today, smart IoT devices are becoming increasingly ubiquitous with diverse forms and functions, yet most user controls over them are still limited to mobi...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2020
|
| Online Access: | https://hdl.handle.net/1721.1/127844 |
| _version_ | 1826211843449815040 |
|---|---|
| author | Sun, Yongbin Armengol Urpi, Alexandre Kantareddy, Sai Nithin R. Siegel, Joshua E Sarma, Sanjay E |
| author2 | Massachusetts Institute of Technology. Department of Mechanical Engineering |
| author_facet | Massachusetts Institute of Technology. Department of Mechanical Engineering Sun, Yongbin Armengol Urpi, Alexandre Kantareddy, Sai Nithin R. Siegel, Joshua E Sarma, Sanjay E |
| author_sort | Sun, Yongbin |
| collection | MIT |
| description | We present an Augmented Reality (AR) visualization and interaction tool for users to control Internet of Things (IoT) devices with hand gestures. Today, smart IoT devices are becoming increasingly ubiquitous with diverse forms and functions, yet most user controls over them are still limited to mobile devices and web interfaces. Recently, AR has been developed rapidly, and provided immersive solutions to enhance user experience of applications in many fields. Its capability to create immersive interactions allows AR to improve the way smart devices are controlled via more direct visual feedback. In this paper, we create a functional prototype of one such system, enabling seamless interactions with sound and lighting systems through the use of augmented hand-controlled interaction panels. To interpret users' intentions, we implement a standard 2D convolution neural network (CNN) for real-time hand gesture recognition and deploy it within our system. Our prototype is also equipped with a simple but effective object detector which can identify target devices within a proper range by analyzing geometric features. We evaluate the performance of our system qualitatively and quantitatively and demonstrate it on two smart devices. |
| first_indexed | 2024-09-23T15:12:20Z |
| format | Article |
| id | mit-1721.1/127844 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T15:12:20Z |
| publishDate | 2020 |
| publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| record_format | dspace |
| spelling | mit-1721.1/1278442022-10-02T01:20:44Z MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment MagicHand: Interact with IoT Devices in Augmented Reality Environment Sun, Yongbin Armengol Urpi, Alexandre Kantareddy, Sai Nithin R. Siegel, Joshua E Sarma, Sanjay E Massachusetts Institute of Technology. Department of Mechanical Engineering Massachusetts Institute of Technology. Office of Digital Learning We present an Augmented Reality (AR) visualization and interaction tool for users to control Internet of Things (IoT) devices with hand gestures. Today, smart IoT devices are becoming increasingly ubiquitous with diverse forms and functions, yet most user controls over them are still limited to mobile devices and web interfaces. Recently, AR has been developed rapidly, and provided immersive solutions to enhance user experience of applications in many fields. Its capability to create immersive interactions allows AR to improve the way smart devices are controlled via more direct visual feedback. In this paper, we create a functional prototype of one such system, enabling seamless interactions with sound and lighting systems through the use of augmented hand-controlled interaction panels. To interpret users' intentions, we implement a standard 2D convolution neural network (CNN) for real-time hand gesture recognition and deploy it within our system. Our prototype is also equipped with a simple but effective object detector which can identify target devices within a proper range by analyzing geometric features. We evaluate the performance of our system qualitatively and quantitatively and demonstrate it on two smart devices. 2020-10-08T20:13:15Z 2020-10-08T20:13:15Z 2019-08 2019-03 2020-09-21T17:14:13Z Article http://purl.org/eprint/type/ConferencePaper 9781728113777 2642-5254 https://hdl.handle.net/1721.1/127844 Sun, Yongbin et al. "MagicHand: Interact with IoT Devices in Augmented Reality Environment." 26th IEEE Conference on Virtual Reality and 3D User Interfaces, March 2019, Osaka, Japan, Institute of Electrical and Electronics Engineers, August 2019. © 2019 IEEE en http://dx.doi.org/10.1109/vr.2019.8798053 26th IEEE Conference on Virtual Reality and 3D User Interfaces Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute of Electrical and Electronics Engineers (IEEE) Prof. Sarma via Elizabeth Soergel |
| spellingShingle | Sun, Yongbin Armengol Urpi, Alexandre Kantareddy, Sai Nithin R. Siegel, Joshua E Sarma, Sanjay E MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment |
| title | MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment |
| title_full | MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment |
| title_fullStr | MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment |
| title_full_unstemmed | MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment |
| title_short | MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment |
| title_sort | magichand a deep learning approach towards manipulating iot devices in augmented reality environment |
| url | https://hdl.handle.net/1721.1/127844 |
| work_keys_str_mv | AT sunyongbin magichandadeeplearningapproachtowardsmanipulatingiotdevicesinaugmentedrealityenvironment AT armengolurpialexandre magichandadeeplearningapproachtowardsmanipulatingiotdevicesinaugmentedrealityenvironment AT kantareddysainithinr magichandadeeplearningapproachtowardsmanipulatingiotdevicesinaugmentedrealityenvironment AT siegeljoshuae magichandadeeplearningapproachtowardsmanipulatingiotdevicesinaugmentedrealityenvironment AT sarmasanjaye magichandadeeplearningapproachtowardsmanipulatingiotdevicesinaugmentedrealityenvironment AT sunyongbin magichandinteractwithiotdevicesinaugmentedrealityenvironment AT armengolurpialexandre magichandinteractwithiotdevicesinaugmentedrealityenvironment AT kantareddysainithinr magichandinteractwithiotdevicesinaugmentedrealityenvironment AT siegeljoshuae magichandinteractwithiotdevicesinaugmentedrealityenvironment AT sarmasanjaye magichandinteractwithiotdevicesinaugmentedrealityenvironment |