Skeleton-based relational reasoning for group activity analysis
Research on group activity recognition mostly leans on the standard two-stream approach (RGB and Optical Flow) as their input features. Few have explored explicit pose information, with none using it directly to reason about the persons interactions. In this paper, we leverage the skeleton informati...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2022
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/161422 |
| _version_ | 1811690336323371008 |
|---|---|
| author | Perez, Mauricio Liu, Jun Kot, Alex Chichung |
| author2 | School of Electrical and Electronic Engineering |
| author_facet | School of Electrical and Electronic Engineering Perez, Mauricio Liu, Jun Kot, Alex Chichung |
| author_sort | Perez, Mauricio |
| collection | NTU |
| description | Research on group activity recognition mostly leans on the standard two-stream approach (RGB and Optical Flow) as their input features. Few have explored explicit pose information, with none using it directly to reason about the persons interactions. In this paper, we leverage the skeleton information to learn the interactions between the individuals straight from it. With our proposed method GIRN, multiple relationship types are inferred from independent modules, that describe the relations between the body joints pair-by-pair. Additionally to the joints relations, we also experiment with the previously unexplored relationship between individuals and relevant objects (e.g. volleyball). The individuals distinct relations are then merged through an attention mechanism, that gives more importance to those individuals more relevant for distinguishing the group activity. We evaluate our method in the Volleyball dataset, obtaining competitive results to the state-of-the-art. Our experiments demonstrate the potential of skeleton-based approaches for modeling multi-person interactions. |
| first_indexed | 2024-10-01T06:02:23Z |
| format | Journal Article |
| id | ntu-10356/161422 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:02:23Z |
| publishDate | 2022 |
| record_format | dspace |
| spelling | ntu-10356/1614222022-08-31T07:01:12Z Skeleton-based relational reasoning for group activity analysis Perez, Mauricio Liu, Jun Kot, Alex Chichung School of Electrical and Electronic Engineering Engineering::Electrical and electronic engineering Engineering::Computer science and engineering Group Activity Recognition Skeleton Information Research on group activity recognition mostly leans on the standard two-stream approach (RGB and Optical Flow) as their input features. Few have explored explicit pose information, with none using it directly to reason about the persons interactions. In this paper, we leverage the skeleton information to learn the interactions between the individuals straight from it. With our proposed method GIRN, multiple relationship types are inferred from independent modules, that describe the relations between the body joints pair-by-pair. Additionally to the joints relations, we also experiment with the previously unexplored relationship between individuals and relevant objects (e.g. volleyball). The individuals distinct relations are then merged through an attention mechanism, that gives more importance to those individuals more relevant for distinguishing the group activity. We evaluate our method in the Volleyball dataset, obtaining competitive results to the state-of-the-art. Our experiments demonstrate the potential of skeleton-based approaches for modeling multi-person interactions. Nanyang Technological University This research was supported by a grant from NTU College of Engineering (M4081746.D90). 2022-08-31T07:01:11Z 2022-08-31T07:01:11Z 2022 Journal Article Perez, M., Liu, J. & Kot, A. C. (2022). Skeleton-based relational reasoning for group activity analysis. Pattern Recognition, 122, 108360-. https://dx.doi.org/10.1016/j.patcog.2021.108360 0031-3203 https://hdl.handle.net/10356/161422 10.1016/j.patcog.2021.108360 2-s2.0-85118648277 122 108360 en M4081746.D90 Pattern Recognition © 2021 Elsevier Ltd. All rights reserved. |
| spellingShingle | Engineering::Electrical and electronic engineering Engineering::Computer science and engineering Group Activity Recognition Skeleton Information Perez, Mauricio Liu, Jun Kot, Alex Chichung Skeleton-based relational reasoning for group activity analysis |
| title | Skeleton-based relational reasoning for group activity analysis |
| title_full | Skeleton-based relational reasoning for group activity analysis |
| title_fullStr | Skeleton-based relational reasoning for group activity analysis |
| title_full_unstemmed | Skeleton-based relational reasoning for group activity analysis |
| title_short | Skeleton-based relational reasoning for group activity analysis |
| title_sort | skeleton based relational reasoning for group activity analysis |
| topic | Engineering::Electrical and electronic engineering Engineering::Computer science and engineering Group Activity Recognition Skeleton Information |
| url | https://hdl.handle.net/10356/161422 |
| work_keys_str_mv | AT perezmauricio skeletonbasedrelationalreasoningforgroupactivityanalysis AT liujun skeletonbasedrelationalreasoningforgroupactivityanalysis AT kotalexchichung skeletonbasedrelationalreasoningforgroupactivityanalysis |