TSM: Temporal Shift Module for Efficient Video Understanding
© 2019 IEEE. The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but a...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022
|
| Online Access: | https://hdl.handle.net/1721.1/143615 |
| _version_ | 1811097239486988288 |
|---|---|
| author | Lin, Ji Gan, Chuang Han, Song |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Lin, Ji Gan, Chuang Han, Song |
| author_sort | Lin, Ji |
| collection | MIT |
| description | © 2019 IEEE. The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: It ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: Https://github. com/mit-han-lab/temporal-shift-module. |
| first_indexed | 2024-09-23T16:56:30Z |
| format | Article |
| id | mit-1721.1/143615 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T16:56:30Z |
| publishDate | 2022 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1436152023-06-28T18:48:32Z TSM: Temporal Shift Module for Efficient Video Understanding Lin, Ji Gan, Chuang Han, Song Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science MIT-IBM Watson AI Lab © 2019 IEEE. The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: It ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: Https://github. com/mit-han-lab/temporal-shift-module. 2022-06-30T17:26:01Z 2022-06-30T17:26:01Z 2019 2022-06-30T17:03:24Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/143615 Lin, Ji, Gan, Chuang and Han, Song. 2019. "TSM: Temporal Shift Module for Efficient Video Understanding." Proceedings of the IEEE International Conference on Computer Vision, 2019-October. en 10.1109/ICCV.2019.00718 Proceedings of the IEEE International Conference on Computer Vision Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf IEEE Computer Vision Foundation |
| spellingShingle | Lin, Ji Gan, Chuang Han, Song TSM: Temporal Shift Module for Efficient Video Understanding |
| title | TSM: Temporal Shift Module for Efficient Video Understanding |
| title_full | TSM: Temporal Shift Module for Efficient Video Understanding |
| title_fullStr | TSM: Temporal Shift Module for Efficient Video Understanding |
| title_full_unstemmed | TSM: Temporal Shift Module for Efficient Video Understanding |
| title_short | TSM: Temporal Shift Module for Efficient Video Understanding |
| title_sort | tsm temporal shift module for efficient video understanding |
| url | https://hdl.handle.net/1721.1/143615 |
| work_keys_str_mv | AT linji tsmtemporalshiftmoduleforefficientvideounderstanding AT ganchuang tsmtemporalshiftmoduleforefficientvideounderstanding AT hansong tsmtemporalshiftmoduleforefficientvideounderstanding |