Skeleton-based human action recognition with global context-aware attention LSTM networks

Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, long short-term memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints...

Full description

Bibliographic Details
Main Authors: Liu, Jun, Wang, Gang, Duan, Ling-Yu, Abdiyeva, Kamila, Kot, Alex Chichung
Other Authors: School of Electrical and Electronic Engineering
Format: Journal Article
Language:English
Published: 2020
Subjects:
Online Access:https://hdl.handle.net/10356/142304
_version_ 1826127059914588160
author Liu, Jun
Wang, Gang
Duan, Ling-Yu
Abdiyeva, Kamila
Kot, Alex Chichung
author2 School of Electrical and Electronic Engineering
author_facet School of Electrical and Electronic Engineering
Liu, Jun
Wang, Gang
Duan, Ling-Yu
Abdiyeva, Kamila
Kot, Alex Chichung
author_sort Liu, Jun
collection NTU
description Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, long short-term memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, global context-aware attention LSTM, for skeleton-based action recognition, which is capable of selectively focusing on the informative joints in each frame by using a global context memory cell. To further improve the attention capability, we also introduce a recurrent attention mechanism, with which the attention performance of our network can be enhanced progressively. Besides, a two-stream framework, which leverages coarse-grained attention and fine-grained attention, is also introduced. The proposed method achieves state-of-the-art performance on five challenging datasets for skeleton-based action recognition.
first_indexed 2024-10-01T07:02:35Z
format Journal Article
id ntu-10356/142304
institution Nanyang Technological University
language English
last_indexed 2024-10-01T07:02:35Z
publishDate 2020
record_format dspace
spelling ntu-10356/1423042020-06-18T08:46:12Z Skeleton-based human action recognition with global context-aware attention LSTM networks Liu, Jun Wang, Gang Duan, Ling-Yu Abdiyeva, Kamila Kot, Alex Chichung School of Electrical and Electronic Engineering Engineering::Computer science and engineering Action Recognition Long Short-term Memory Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, long short-term memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, global context-aware attention LSTM, for skeleton-based action recognition, which is capable of selectively focusing on the informative joints in each frame by using a global context memory cell. To further improve the attention capability, we also introduce a recurrent attention mechanism, with which the attention performance of our network can be enhanced progressively. Besides, a two-stream framework, which leverages coarse-grained attention and fine-grained attention, is also introduced. The proposed method achieves state-of-the-art performance on five challenging datasets for skeleton-based action recognition. NRF (Natl Research Foundation, S’pore) 2020-06-18T08:46:12Z 2020-06-18T08:46:12Z 2017 Journal Article Liu, J., Wang, G., Duan, L.-Y., Abdiyeva, K., & Kot, A. C. (2018). Skeleton-based human action recognition with global context-aware attention LSTM networks. IEEE Transactions on Image Processing, 27(4), 1586-1599. doi:10.1109/TIP.2017.2785279 1057-7149 https://hdl.handle.net/10356/142304 10.1109/TIP.2017.2785279 29324413 2-s2.0-85039779403 4 27 1586 1599 en IEEE Transactions on Image Processing © 2017 IEEE. All rights reserved.
spellingShingle Engineering::Computer science and engineering
Action Recognition
Long Short-term Memory
Liu, Jun
Wang, Gang
Duan, Ling-Yu
Abdiyeva, Kamila
Kot, Alex Chichung
Skeleton-based human action recognition with global context-aware attention LSTM networks
title Skeleton-based human action recognition with global context-aware attention LSTM networks
title_full Skeleton-based human action recognition with global context-aware attention LSTM networks
title_fullStr Skeleton-based human action recognition with global context-aware attention LSTM networks
title_full_unstemmed Skeleton-based human action recognition with global context-aware attention LSTM networks
title_short Skeleton-based human action recognition with global context-aware attention LSTM networks
title_sort skeleton based human action recognition with global context aware attention lstm networks
topic Engineering::Computer science and engineering
Action Recognition
Long Short-term Memory
url https://hdl.handle.net/10356/142304
work_keys_str_mv AT liujun skeletonbasedhumanactionrecognitionwithglobalcontextawareattentionlstmnetworks
AT wanggang skeletonbasedhumanactionrecognitionwithglobalcontextawareattentionlstmnetworks
AT duanlingyu skeletonbasedhumanactionrecognitionwithglobalcontextawareattentionlstmnetworks
AT abdiyevakamila skeletonbasedhumanactionrecognitionwithglobalcontextawareattentionlstmnetworks
AT kotalexchichung skeletonbasedhumanactionrecognitionwithglobalcontextawareattentionlstmnetworks