Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction Recognition
Two-person interaction recognition has become an area of growing interest in human action recognition. The graph convolutional network (GCN) using human skeleton data has been shown to be highly effective for action recognition. Most GCN-based methods focus on recognizing an individual person&#x...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9507447/ |
| _version_ | 1819131721370566656 |
|---|---|
| author | Yoshiki Ito Kenichi Morita Quan Kong Tomoaki Yoshinaga |
| author_facet | Yoshiki Ito Kenichi Morita Quan Kong Tomoaki Yoshinaga |
| author_sort | Yoshiki Ito |
| collection | DOAJ |
| description | Two-person interaction recognition has become an area of growing interest in human action recognition. The graph convolutional network (GCN) using human skeleton data has been shown to be highly effective for action recognition. Most GCN-based methods focus on recognizing an individual person’s actions on the basis of an intra-body graph. However, many of these methods do not represent the relation between two bodies, making it difficult to accurately recognize human interaction. In this work, we propose multi-stream adaptive GCN using inter- and intra-body graphs (MAGCN-IIG) as a new method of human interaction recognition. To ensure highly accurate human interaction recognition, our method cooperatively utilizes two types of graphs: an inter-body graph and an intra-body graph. The inter-body graph, which is newly introduced in this paper, connects the inter-body joints between two people as well as intra-body connections. The adaptive GCN using the inter-body graph captures the relation of joints between two people, even different types of joints located far away from each other. Further, by implementing a multi-stream architecture, our method simultaneously captures both inter-body and intra-body relations in each of two units that represent the position and motion of people. Experiments on interaction recognition using two large-scale human action datasets, NTU RGB+D and NTU RGB+D 120, showed that our method recognized human interactions more accurately than state-of-the-art methods. |
| first_indexed | 2024-12-22T09:20:00Z |
| format | Article |
| id | doaj.art-c1354e545b0d4c0080bbba1dd2fc4627 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-22T09:20:00Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-c1354e545b0d4c0080bbba1dd2fc46272022-12-21T18:31:14ZengIEEEIEEE Access2169-35362021-01-01911067011068210.1109/ACCESS.2021.31026719507447Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction RecognitionYoshiki Ito0https://orcid.org/0000-0002-8813-9458Kenichi Morita1https://orcid.org/0000-0001-6550-9156Quan Kong2https://orcid.org/0000-0002-4511-4031Tomoaki Yoshinaga3https://orcid.org/0000-0003-1975-6273Lumada Data Science Lab., Hitachi, Ltd., Kokubunji-shi, Tokyo, JapanLumada Data Science Lab., Hitachi, Ltd., Kokubunji-shi, Tokyo, JapanLumada Data Science Lab., Hitachi, Ltd., Kokubunji-shi, Tokyo, JapanLumada Data Science Lab., Hitachi, Ltd., Kokubunji-shi, Tokyo, JapanTwo-person interaction recognition has become an area of growing interest in human action recognition. The graph convolutional network (GCN) using human skeleton data has been shown to be highly effective for action recognition. Most GCN-based methods focus on recognizing an individual person’s actions on the basis of an intra-body graph. However, many of these methods do not represent the relation between two bodies, making it difficult to accurately recognize human interaction. In this work, we propose multi-stream adaptive GCN using inter- and intra-body graphs (MAGCN-IIG) as a new method of human interaction recognition. To ensure highly accurate human interaction recognition, our method cooperatively utilizes two types of graphs: an inter-body graph and an intra-body graph. The inter-body graph, which is newly introduced in this paper, connects the inter-body joints between two people as well as intra-body connections. The adaptive GCN using the inter-body graph captures the relation of joints between two people, even different types of joints located far away from each other. Further, by implementing a multi-stream architecture, our method simultaneously captures both inter-body and intra-body relations in each of two units that represent the position and motion of people. Experiments on interaction recognition using two large-scale human action datasets, NTU RGB+D and NTU RGB+D 120, showed that our method recognized human interactions more accurately than state-of-the-art methods.https://ieeexplore.ieee.org/document/9507447/Graph convolutional networkhuman interaction recognitionmulti-stream networkskeleton-based action recognition |
| spellingShingle | Yoshiki Ito Kenichi Morita Quan Kong Tomoaki Yoshinaga Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction Recognition IEEE Access Graph convolutional network human interaction recognition multi-stream network skeleton-based action recognition |
| title | Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction Recognition |
| title_full | Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction Recognition |
| title_fullStr | Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction Recognition |
| title_full_unstemmed | Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction Recognition |
| title_short | Multi-Stream Adaptive Graph Convolutional Network Using Inter- and Intra-Body Graphs for Two-Person Interaction Recognition |
| title_sort | multi stream adaptive graph convolutional network using inter and intra body graphs for two person interaction recognition |
| topic | Graph convolutional network human interaction recognition multi-stream network skeleton-based action recognition |
| url | https://ieeexplore.ieee.org/document/9507447/ |
| work_keys_str_mv | AT yoshikiito multistreamadaptivegraphconvolutionalnetworkusinginterandintrabodygraphsfortwopersoninteractionrecognition AT kenichimorita multistreamadaptivegraphconvolutionalnetworkusinginterandintrabodygraphsfortwopersoninteractionrecognition AT quankong multistreamadaptivegraphconvolutionalnetworkusinginterandintrabodygraphsfortwopersoninteractionrecognition AT tomoakiyoshinaga multistreamadaptivegraphconvolutionalnetworkusinginterandintrabodygraphsfortwopersoninteractionrecognition |