Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition
Skeleton-based action recognition has attracted extensive attention recently in the computer vision community. Previous studies, especially GCN-based methods, have presented remarkable improvements for this task. However, in existing GCN-based methods, global average pooling is applied to the extrac...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9260250/ |
| _version_ | 1819172781641695232 |
|---|---|
| author | Zheyuan Xu Yingfu Wang Jiaqin Jiang Jian Yao Liang Li |
| author_facet | Zheyuan Xu Yingfu Wang Jiaqin Jiang Jian Yao Liang Li |
| author_sort | Zheyuan Xu |
| collection | DOAJ |
| description | Skeleton-based action recognition has attracted extensive attention recently in the computer vision community. Previous studies, especially GCN-based methods, have presented remarkable improvements for this task. However, in existing GCN-based methods, global average pooling is applied to the extracted features before the classifier. This may hurt the recognition performance since it neglects the fact that not all features are equally important in the temporal dimension. To tackle this issue, in this article, we propose a feature selection network (FSN) with actor-critic reinforcement learning. Given the extracted feature sequence, FSN learns to adaptively select the most representative features and discard the ambiguous features for action recognition. In addition, conventional graph convolution is a local operation, it cannot fully capture the non-local joint dependencies that could be vital to recognize the action. Thus, we also propose a generalized graph generation module to capture latent dependencies and further propose a generalized graph convolution network (GGCN). The GGCN and FSN are combined in a three-stream recognition framework, in which different types of information from skeleton data are further fused to improve the recognition accuracy. Extensive experiments demonstrate that the proposed FSN is a flexible and effective module that can cooperate with any existing GCN-based framework to enhance the recognition accuracy, the proposed GGCN can extract richer skeleton features for skeleton-based action recognition, and our method achieves superior performance over several public datasets, e.g. 95.7 top-1 accuracy on NTU-RGB+D, 86.7 top-1 accuracy on NTU-RGB+D 120, etc. |
| first_indexed | 2024-12-22T20:12:38Z |
| format | Article |
| id | doaj.art-6af2fffe5f3e49aba9bfddd677825753 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-22T20:12:38Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-6af2fffe5f3e49aba9bfddd6778257532022-12-21T18:14:01ZengIEEEIEEE Access2169-35362020-01-01821303821305110.1109/ACCESS.2020.30382359260250Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action RecognitionZheyuan Xu0https://orcid.org/0000-0001-9925-7559Yingfu Wang1https://orcid.org/0000-0001-9949-7579Jiaqin Jiang2Jian Yao3https://orcid.org/0000-0002-9134-5084Liang Li4School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, ChinaSchool of Remote Sensing and Information Engineering, Wuhan University, Wuhan, ChinaSchool of Remote Sensing and Information Engineering, Wuhan University, Wuhan, ChinaSchool of Remote Sensing and Information Engineering, Wuhan University, Wuhan, ChinaShenzhen Polytechnic, Shenzhen, ChinaSkeleton-based action recognition has attracted extensive attention recently in the computer vision community. Previous studies, especially GCN-based methods, have presented remarkable improvements for this task. However, in existing GCN-based methods, global average pooling is applied to the extracted features before the classifier. This may hurt the recognition performance since it neglects the fact that not all features are equally important in the temporal dimension. To tackle this issue, in this article, we propose a feature selection network (FSN) with actor-critic reinforcement learning. Given the extracted feature sequence, FSN learns to adaptively select the most representative features and discard the ambiguous features for action recognition. In addition, conventional graph convolution is a local operation, it cannot fully capture the non-local joint dependencies that could be vital to recognize the action. Thus, we also propose a generalized graph generation module to capture latent dependencies and further propose a generalized graph convolution network (GGCN). The GGCN and FSN are combined in a three-stream recognition framework, in which different types of information from skeleton data are further fused to improve the recognition accuracy. Extensive experiments demonstrate that the proposed FSN is a flexible and effective module that can cooperate with any existing GCN-based framework to enhance the recognition accuracy, the proposed GGCN can extract richer skeleton features for skeleton-based action recognition, and our method achieves superior performance over several public datasets, e.g. 95.7 top-1 accuracy on NTU-RGB+D, 86.7 top-1 accuracy on NTU-RGB+D 120, etc.https://ieeexplore.ieee.org/document/9260250/Action recognitionskeletonfeature selectionreinforcement learninggraph convolutional network |
| spellingShingle | Zheyuan Xu Yingfu Wang Jiaqin Jiang Jian Yao Liang Li Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition IEEE Access Action recognition skeleton feature selection reinforcement learning graph convolutional network |
| title | Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition |
| title_full | Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition |
| title_fullStr | Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition |
| title_full_unstemmed | Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition |
| title_short | Adaptive Feature Selection With Reinforcement Learning for Skeleton-Based Action Recognition |
| title_sort | adaptive feature selection with reinforcement learning for skeleton based action recognition |
| topic | Action recognition skeleton feature selection reinforcement learning graph convolutional network |
| url | https://ieeexplore.ieee.org/document/9260250/ |
| work_keys_str_mv | AT zheyuanxu adaptivefeatureselectionwithreinforcementlearningforskeletonbasedactionrecognition AT yingfuwang adaptivefeatureselectionwithreinforcementlearningforskeletonbasedactionrecognition AT jiaqinjiang adaptivefeatureselectionwithreinforcementlearningforskeletonbasedactionrecognition AT jianyao adaptivefeatureselectionwithreinforcementlearningforskeletonbasedactionrecognition AT liangli adaptivefeatureselectionwithreinforcementlearningforskeletonbasedactionrecognition |