RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition
Generally, the action recognition task requires a vast amount of labeled data, which represents a time-consuming human annotation effort. To mitigate the dependency on labeled data, this study proposes Semi-Supervised and Iterative Reinforcement Learning (RL-SSI), which adapts a supervised approach...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/11/9/1471 |
| _version_ | 1797505116185559040 |
|---|---|
| author | Lucas Lisboa dos Santos Ingrid Winkler Erick Giovani Sperandio Nascimento |
| author_facet | Lucas Lisboa dos Santos Ingrid Winkler Erick Giovani Sperandio Nascimento |
| author_sort | Lucas Lisboa dos Santos |
| collection | DOAJ |
| description | Generally, the action recognition task requires a vast amount of labeled data, which represents a time-consuming human annotation effort. To mitigate the dependency on labeled data, this study proposes Semi-Supervised and Iterative Reinforcement Learning (RL-SSI), which adapts a supervised approach that uses 100% labeled data to a semi-supervised and iterative approach using reinforcement learning for human action recognition in videos. The JIGSAWS and Breakfast datasets were used to evaluate the RL-SSI model, because they are commonly used in the action segmentation task. The same applies to the performance metrics used in this work-F-Score (F1) and Edit Score-which are commonly applied for such tasks. In JIGSAWS tests, we observed that the RL-SSI outperformed previously developed state-of-the-art techniques in all quantitative measures, while using only 65% of the labeled data. When analysing the Breakfast tests, we compared the effectiveness of RL-SSI with the results of the self-supervised technique called SSTDA. We have found that RL-SSI outperformed SSTDA with an accuracy of 66.44% versus 65.8%, but RL-SSI was surpassed by the F1@10 segmentation measure, which presented an accuracy of 67.33% versus 69.3% for SSTDA. Despite this, our experiment only used 55.8% of the labeled data, while SSTDA used 65%. We conclude that our approach outperformed equivalent supervised learning methods and is comparable to SSTDA, when evaluated on multiple datasets of human action recognition, proving to be an important innovative method to successfully building solutions to reduce the amount of fully labeled data, leveraging the work of human specialists in the task of data labeling of videos, and their respectives frames, for human action recognition, thus reducing the required resources to accomplish it. |
| first_indexed | 2024-03-10T04:14:00Z |
| format | Article |
| id | doaj.art-59c298010ddc4f6d93fa6ac0dddc8a7e |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T04:14:00Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-59c298010ddc4f6d93fa6ac0dddc8a7e2023-11-23T08:04:05ZengMDPI AGElectronics2079-92922022-05-01119147110.3390/electronics11091471RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action RecognitionLucas Lisboa dos Santos0Ingrid Winkler1Erick Giovani Sperandio Nascimento2Department of Stricto Sensu Studies, SENAI CIMATEC University Center, Salvador 41650-010, BrazilDepartment of Management and Industrial Technology, SENAI CIMATEC University Center, Salvador 41650-010, BrazilDepartment of Stricto Sensu Studies, SENAI CIMATEC University Center, Salvador 41650-010, BrazilGenerally, the action recognition task requires a vast amount of labeled data, which represents a time-consuming human annotation effort. To mitigate the dependency on labeled data, this study proposes Semi-Supervised and Iterative Reinforcement Learning (RL-SSI), which adapts a supervised approach that uses 100% labeled data to a semi-supervised and iterative approach using reinforcement learning for human action recognition in videos. The JIGSAWS and Breakfast datasets were used to evaluate the RL-SSI model, because they are commonly used in the action segmentation task. The same applies to the performance metrics used in this work-F-Score (F1) and Edit Score-which are commonly applied for such tasks. In JIGSAWS tests, we observed that the RL-SSI outperformed previously developed state-of-the-art techniques in all quantitative measures, while using only 65% of the labeled data. When analysing the Breakfast tests, we compared the effectiveness of RL-SSI with the results of the self-supervised technique called SSTDA. We have found that RL-SSI outperformed SSTDA with an accuracy of 66.44% versus 65.8%, but RL-SSI was surpassed by the F1@10 segmentation measure, which presented an accuracy of 67.33% versus 69.3% for SSTDA. Despite this, our experiment only used 55.8% of the labeled data, while SSTDA used 65%. We conclude that our approach outperformed equivalent supervised learning methods and is comparable to SSTDA, when evaluated on multiple datasets of human action recognition, proving to be an important innovative method to successfully building solutions to reduce the amount of fully labeled data, leveraging the work of human specialists in the task of data labeling of videos, and their respectives frames, for human action recognition, thus reducing the required resources to accomplish it.https://www.mdpi.com/2079-9292/11/9/1471deep learninghuman action recognitiondata annotationsemi-supervised learningaction segmentation |
| spellingShingle | Lucas Lisboa dos Santos Ingrid Winkler Erick Giovani Sperandio Nascimento RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition Electronics deep learning human action recognition data annotation semi-supervised learning action segmentation |
| title | RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition |
| title_full | RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition |
| title_fullStr | RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition |
| title_full_unstemmed | RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition |
| title_short | RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition |
| title_sort | rl ssi model adapting a supervised learning approach to a semi supervised approach for human action recognition |
| topic | deep learning human action recognition data annotation semi-supervised learning action segmentation |
| url | https://www.mdpi.com/2079-9292/11/9/1471 |
| work_keys_str_mv | AT lucaslisboadossantos rlssimodeladaptingasupervisedlearningapproachtoasemisupervisedapproachforhumanactionrecognition AT ingridwinkler rlssimodeladaptingasupervisedlearningapproachtoasemisupervisedapproachforhumanactionrecognition AT erickgiovanisperandionascimento rlssimodeladaptingasupervisedlearningapproachtoasemisupervisedapproachforhumanactionrecognition |