ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos
Annotated videos are commonly produced for documenting assembly and maintenance processes in the manufacturing industry. However, according to a semi-structured interview we conducted with industrial experts, the current process of creating annotated assembly videos, in which the annotator annotates...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9925238/ |
| _version_ | 1797989396401618944 |
|---|---|
| author | Truong an Pham Tim Moesgen Sanni Siltanen Joanna Bergstrom Yu Xiao |
| author_facet | Truong an Pham Tim Moesgen Sanni Siltanen Joanna Bergstrom Yu Xiao |
| author_sort | Truong an Pham |
| collection | DOAJ |
| description | Annotated videos are commonly produced for documenting assembly and maintenance processes in the manufacturing industry. However, according to a semi-structured interview we conducted with industrial experts, the current process of creating annotated assembly videos, in which the annotator annotates the video capturing the expert’s demonstration of assembly and maintenance process, is cumbersome and time-consuming. The key challenges include three key problems in annotation: (1) unnecessary extra communications between field workers and annotators, (2) lack of suitable camera gear, and (3) wasting time in the manual removal of non-informative portions of captured videos. Because annotation always follows video capture, the problem 1 remains out of scope for state-of-the-art video annotation tools. And making the assumption of a perfect captured video, which covers no occlusion and contains only relevant assembly or maintenance information, causes problem 2 and 3. As a result, we have developed ARiana, a wearable augmented reality-based in-situ video annotation tool that guides field experts to create annotations efficiently while conducting the assembly or maintenance tasks. ARiana has three key features that include context-awareness enabled by hand-object interaction, multimodal interaction for annotation on the fly, and real-time audiovisual guidance enabled by edge offloading. We have implemented ARiana on Android-based smart glasses, equipped with first-person camera and microphone. In a usability test based on attempting to assemble a toy model and to annotate the recorded video simultaneously, ARiana demonstrated higher efficiency and effectiveness compared to one of the state-of-the-art video annotation tools, in which the assembling process is followed by the annotation process. In particular, ARiana helps users finish annotation tasks four times faster, and increase the annotation accuracy by 23%. |
| first_indexed | 2024-04-11T08:18:35Z |
| format | Article |
| id | doaj.art-cfecd955b878491e9557bd0926e2b7db |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-11T08:18:35Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-cfecd955b878491e9557bd0926e2b7db2022-12-22T04:35:03ZengIEEEIEEE Access2169-35362022-01-011011170411172410.1109/ACCESS.2022.32160159925238ARiana: Augmented Reality Based In-Situ Annotation of Assembly VideosTruong an Pham0https://orcid.org/0000-0002-6861-1495Tim Moesgen1Sanni Siltanen2https://orcid.org/0000-0002-6057-9849Joanna Bergstrom3https://orcid.org/0000-0001-6764-5661Yu Xiao4https://orcid.org/0000-0002-4517-3779Department of Communications and Networking, School of Electrical Engineering, Aalto University, Espoo, FinlandDepartment of Communications and Networking, School of Electrical Engineering, Aalto University, Espoo, FinlandDimecc Ltd., Tampere, FinlandDepartment of Computer Science, University of Copenhagen, Copenhagen, DenmarkDepartment of Communications and Networking, School of Electrical Engineering, Aalto University, Espoo, FinlandAnnotated videos are commonly produced for documenting assembly and maintenance processes in the manufacturing industry. However, according to a semi-structured interview we conducted with industrial experts, the current process of creating annotated assembly videos, in which the annotator annotates the video capturing the expert’s demonstration of assembly and maintenance process, is cumbersome and time-consuming. The key challenges include three key problems in annotation: (1) unnecessary extra communications between field workers and annotators, (2) lack of suitable camera gear, and (3) wasting time in the manual removal of non-informative portions of captured videos. Because annotation always follows video capture, the problem 1 remains out of scope for state-of-the-art video annotation tools. And making the assumption of a perfect captured video, which covers no occlusion and contains only relevant assembly or maintenance information, causes problem 2 and 3. As a result, we have developed ARiana, a wearable augmented reality-based in-situ video annotation tool that guides field experts to create annotations efficiently while conducting the assembly or maintenance tasks. ARiana has three key features that include context-awareness enabled by hand-object interaction, multimodal interaction for annotation on the fly, and real-time audiovisual guidance enabled by edge offloading. We have implemented ARiana on Android-based smart glasses, equipped with first-person camera and microphone. In a usability test based on attempting to assemble a toy model and to annotate the recorded video simultaneously, ARiana demonstrated higher efficiency and effectiveness compared to one of the state-of-the-art video annotation tools, in which the assembling process is followed by the annotation process. In particular, ARiana helps users finish annotation tasks four times faster, and increase the annotation accuracy by 23%.https://ieeexplore.ieee.org/document/9925238/Augmented realityfirst-person videosmultimodal interactionprocess documentationvideo annotationworkflow extraction |
| spellingShingle | Truong an Pham Tim Moesgen Sanni Siltanen Joanna Bergstrom Yu Xiao ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos IEEE Access Augmented reality first-person videos multimodal interaction process documentation video annotation workflow extraction |
| title | ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos |
| title_full | ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos |
| title_fullStr | ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos |
| title_full_unstemmed | ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos |
| title_short | ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos |
| title_sort | ariana augmented reality based in situ annotation of assembly videos |
| topic | Augmented reality first-person videos multimodal interaction process documentation video annotation workflow extraction |
| url | https://ieeexplore.ieee.org/document/9925238/ |
| work_keys_str_mv | AT truonganpham arianaaugmentedrealitybasedinsituannotationofassemblyvideos AT timmoesgen arianaaugmentedrealitybasedinsituannotationofassemblyvideos AT sannisiltanen arianaaugmentedrealitybasedinsituannotationofassemblyvideos AT joannabergstrom arianaaugmentedrealitybasedinsituannotationofassemblyvideos AT yuxiao arianaaugmentedrealitybasedinsituannotationofassemblyvideos |