Local fusion networks with chained residual pooling for video action recognition
Action recognition is an important yet challenging problem. We here present a novel method, multistage local fusion networks with residual connections, to boost the performance of video action recognition. In realistic videos, an action instance may have a long time span and some frames may suffer f...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2020
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/143069 |
| _version_ | 1811693887048122368 |
|---|---|
| author | He, Feixiang Liu, Fayao Yao, Rui Lin, Guosheng |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering He, Feixiang Liu, Fayao Yao, Rui Lin, Guosheng |
| author_sort | He, Feixiang |
| collection | NTU |
| description | Action recognition is an important yet challenging problem. We here present a novel method, multistage local fusion networks with residual connections, to boost the performance of video action recognition. In realistic videos, an action instance may have a long time span and some frames may suffer from deteriorated object appearance due to motion blur or video defocus. Our method enhances the per-frame representation by capturing information from neighboring frames. We propose a local fusion block which considers neighboring frames to capture appearance and local motion information for generating per-frame representation. Our local fusion is performed in a multistage manner allowing feature fusion from varying neighborhood sizes in the temporal dimension. We employ residual connections in the fusion blocks to enable effective gradient propagation through the whole network allowing effective end-to-end training. We achieve competitive results on two challenging and public available datasets, namely HMDB51 and UCF101, which shows the effectiveness of the proposed method. |
| first_indexed | 2024-10-01T06:58:49Z |
| format | Journal Article |
| id | ntu-10356/143069 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:58:49Z |
| publishDate | 2020 |
| record_format | dspace |
| spelling | ntu-10356/1430692020-07-28T01:31:10Z Local fusion networks with chained residual pooling for video action recognition He, Feixiang Liu, Fayao Yao, Rui Lin, Guosheng School of Computer Science and Engineering Engineering::Computer science and engineering Action Recognition Residual Connection Action recognition is an important yet challenging problem. We here present a novel method, multistage local fusion networks with residual connections, to boost the performance of video action recognition. In realistic videos, an action instance may have a long time span and some frames may suffer from deteriorated object appearance due to motion blur or video defocus. Our method enhances the per-frame representation by capturing information from neighboring frames. We propose a local fusion block which considers neighboring frames to capture appearance and local motion information for generating per-frame representation. Our local fusion is performed in a multistage manner allowing feature fusion from varying neighborhood sizes in the temporal dimension. We employ residual connections in the fusion blocks to enable effective gradient propagation through the whole network allowing effective end-to-end training. We achieve competitive results on two challenging and public available datasets, namely HMDB51 and UCF101, which shows the effectiveness of the proposed method. Accepted version 2020-07-28T01:31:10Z 2020-07-28T01:31:10Z 2018 Journal Article He, F., Liu, F., Yao, R., & Lin, G. (2019). Local fusion networks with chained residual pooling for video action recognition. Image and Vision Computing, 81, 34-41. doi:10.1016/j.imavis.2018.12.002 0262-8856 https://hdl.handle.net/10356/143069 10.1016/j.imavis.2018.12.002 2-s2.0-85059864452 81 34 41 en Image and Vision Computing © 2019 Elsevier B.V. All rights reserved. This paper was published in Image and Vision Computing and is made available with permission of Elsevier B.V. application/pdf |
| spellingShingle | Engineering::Computer science and engineering Action Recognition Residual Connection He, Feixiang Liu, Fayao Yao, Rui Lin, Guosheng Local fusion networks with chained residual pooling for video action recognition |
| title | Local fusion networks with chained residual pooling for video action recognition |
| title_full | Local fusion networks with chained residual pooling for video action recognition |
| title_fullStr | Local fusion networks with chained residual pooling for video action recognition |
| title_full_unstemmed | Local fusion networks with chained residual pooling for video action recognition |
| title_short | Local fusion networks with chained residual pooling for video action recognition |
| title_sort | local fusion networks with chained residual pooling for video action recognition |
| topic | Engineering::Computer science and engineering Action Recognition Residual Connection |
| url | https://hdl.handle.net/10356/143069 |
| work_keys_str_mv | AT hefeixiang localfusionnetworkswithchainedresidualpoolingforvideoactionrecognition AT liufayao localfusionnetworkswithchainedresidualpoolingforvideoactionrecognition AT yaorui localfusionnetworkswithchainedresidualpoolingforvideoactionrecognition AT linguosheng localfusionnetworkswithchainedresidualpoolingforvideoactionrecognition |