A union of deep learning and swarm-based optimization for 3D human action recognition
Abstract Human Action Recognition (HAR) is a popular area of research in computer vision due to its wide range of applications such as surveillance, health care, and gaming, etc. Action recognition based on 3D skeleton data allows simplistic, cost-efficient models to be formed making it a widely use...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-03-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-022-09293-8 |
| _version_ | 1818645164727468032 |
|---|---|
| author | Hritam Basak Rohit Kundu Pawan Kumar Singh Muhammad Fazal Ijaz Marcin Woźniak Ram Sarkar |
| author_facet | Hritam Basak Rohit Kundu Pawan Kumar Singh Muhammad Fazal Ijaz Marcin Woźniak Ram Sarkar |
| author_sort | Hritam Basak |
| collection | DOAJ |
| description | Abstract Human Action Recognition (HAR) is a popular area of research in computer vision due to its wide range of applications such as surveillance, health care, and gaming, etc. Action recognition based on 3D skeleton data allows simplistic, cost-efficient models to be formed making it a widely used method. In this work, we propose DSwarm-Net, a framework that employs deep learning and swarm intelligence-based metaheuristic for HAR that uses 3D skeleton data for action classification. We extract four different types of features from the skeletal data namely: Distance, Distance Velocity, Angle, and Angle Velocity, which capture complementary information from the skeleton joints for encoding them into images. Encoding the skeleton data features into images is an alternative to the traditional video-processing approach and it helps in making the classification task less complex. The Distance and Distance Velocity encoded images have been stacked depth-wise and fed into a Convolutional Neural Network model which is a modified version of Inception-ResNet. Similarly, the Angle and Angle Velocity encoded images have been stacked depth-wise and fed into the same network. After training these models, deep features have been extracted from the pre-final layer of the networks, and the obtained feature representation is optimized by a nature-inspired metaheuristic, called Ant Lion Optimizer, to eliminate the non-informative or misleading features and to reduce the dimensionality of the feature set. DSwarm-Net has been evaluated on three publicly available HAR datasets, namely UTD-MHAD, HDM05, and NTU RGB+D 60 achieving competitive results, thus confirming the superiority of the proposed model compared to state-of-the-art models. |
| first_indexed | 2024-12-17T00:26:24Z |
| format | Article |
| id | doaj.art-550b14d575cd40b3b944a34d616cee91 |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-12-17T00:26:24Z |
| publishDate | 2022-03-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-550b14d575cd40b3b944a34d616cee912022-12-21T22:10:28ZengNature PortfolioScientific Reports2045-23222022-03-0112111710.1038/s41598-022-09293-8A union of deep learning and swarm-based optimization for 3D human action recognitionHritam Basak0Rohit Kundu1Pawan Kumar Singh2Muhammad Fazal Ijaz3Marcin Woźniak4Ram Sarkar5Department of Electrical Engineering, Jadavpur UniversityDepartment of Electrical Engineering, Jadavpur UniversityDepartment of Information Technology, Jadavpur UniversityDepartment of Intelligent Mechatronics Engineering, Sejong UniversityFaculty of Applied Mathematics, Silesian University of TechnologyDepartment of Computer Science & Engineering, Jadavpur UniversityAbstract Human Action Recognition (HAR) is a popular area of research in computer vision due to its wide range of applications such as surveillance, health care, and gaming, etc. Action recognition based on 3D skeleton data allows simplistic, cost-efficient models to be formed making it a widely used method. In this work, we propose DSwarm-Net, a framework that employs deep learning and swarm intelligence-based metaheuristic for HAR that uses 3D skeleton data for action classification. We extract four different types of features from the skeletal data namely: Distance, Distance Velocity, Angle, and Angle Velocity, which capture complementary information from the skeleton joints for encoding them into images. Encoding the skeleton data features into images is an alternative to the traditional video-processing approach and it helps in making the classification task less complex. The Distance and Distance Velocity encoded images have been stacked depth-wise and fed into a Convolutional Neural Network model which is a modified version of Inception-ResNet. Similarly, the Angle and Angle Velocity encoded images have been stacked depth-wise and fed into the same network. After training these models, deep features have been extracted from the pre-final layer of the networks, and the obtained feature representation is optimized by a nature-inspired metaheuristic, called Ant Lion Optimizer, to eliminate the non-informative or misleading features and to reduce the dimensionality of the feature set. DSwarm-Net has been evaluated on three publicly available HAR datasets, namely UTD-MHAD, HDM05, and NTU RGB+D 60 achieving competitive results, thus confirming the superiority of the proposed model compared to state-of-the-art models.https://doi.org/10.1038/s41598-022-09293-8 |
| spellingShingle | Hritam Basak Rohit Kundu Pawan Kumar Singh Muhammad Fazal Ijaz Marcin Woźniak Ram Sarkar A union of deep learning and swarm-based optimization for 3D human action recognition Scientific Reports |
| title | A union of deep learning and swarm-based optimization for 3D human action recognition |
| title_full | A union of deep learning and swarm-based optimization for 3D human action recognition |
| title_fullStr | A union of deep learning and swarm-based optimization for 3D human action recognition |
| title_full_unstemmed | A union of deep learning and swarm-based optimization for 3D human action recognition |
| title_short | A union of deep learning and swarm-based optimization for 3D human action recognition |
| title_sort | union of deep learning and swarm based optimization for 3d human action recognition |
| url | https://doi.org/10.1038/s41598-022-09293-8 |
| work_keys_str_mv | AT hritambasak aunionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT rohitkundu aunionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT pawankumarsingh aunionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT muhammadfazalijaz aunionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT marcinwozniak aunionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT ramsarkar aunionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT hritambasak unionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT rohitkundu unionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT pawankumarsingh unionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT muhammadfazalijaz unionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT marcinwozniak unionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition AT ramsarkar unionofdeeplearningandswarmbasedoptimizationfor3dhumanactionrecognition |