DisCoVQA: temporal distortion-content transformers for video quality assessment
Compared with spatial counterparts, temporal relationships between frames and their influences on video quality assessment (VQA) are still relatively under-studied in existing works. These relationships lead to two important types of effects for video quality. Firstly, some meaningless temporal vari...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/178454 |
| _version_ | 1826122611294208000 |
|---|---|
| author | Wu, Haoning Chen, Chaofeng Liao, Liang Hou, Jingwen Sun, Wenxiu Yan, Qiong Lin, Weisi |
| author2 | College of Computing and Data Science |
| author_facet | College of Computing and Data Science Wu, Haoning Chen, Chaofeng Liao, Liang Hou, Jingwen Sun, Wenxiu Yan, Qiong Lin, Weisi |
| author_sort | Wu, Haoning |
| collection | NTU |
| description | Compared with spatial counterparts, temporal relationships between frames and their influences on video quality assessment (VQA) are still relatively under-studied in existing works. These relationships lead to two important types of effects for video quality. Firstly, some meaningless temporal variations (such as shaking, flicker, and unsmooth scene transitions) cause temporal distortions that degrade quality of videos. Secondly, the human visual system often has different attention to frames with different contents, resulting in their different importance to the overall video quality. Based on prominent time-series modeling ability of transformers, we propose a novel and effective transformer-based VQA method to tackle these two issues. To better differentiate temporal variations and thus capture the temporal distortions, we design the Spatial-Temporal Distortion Extraction (STDE) module that extracts multi-level spatial-temporal features with a video swin transformer tiny (Swin-T) backbone and uses temporal difference layer to further capture these distortions. To tackle with temporal quality attention, we propose the encoder-decoder-like temporal content transformer (TCT). We also introduce the temporal sampling on features to reduce the input length for the TCT, so as to improve the learning effectiveness and efficiency of this module. Consisting of the STDE and the TCT, the proposed Temporal Distortion-Content Transformers for Video Quality Assessment (DisCoVQA) reaches state-of-the-art performance on several VQA benchmarks without any extra pre-training datasets and up to 10% better generalization ability than existing methods. We also conduct extensive ablation experiments to prove the effectiveness of each part in our proposed model, and provide visualizations to prove that the proposed modules achieve our intention on modeling these temporal issues. Our code is published at https://github.com/QualityAssessment/DisCoVQA. |
| first_indexed | 2024-10-01T05:51:20Z |
| format | Journal Article |
| id | ntu-10356/178454 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T05:51:20Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1784542024-06-20T07:26:29Z DisCoVQA: temporal distortion-content transformers for video quality assessment Wu, Haoning Chen, Chaofeng Liao, Liang Hou, Jingwen Sun, Wenxiu Yan, Qiong Lin, Weisi College of Computing and Data Science School of Computer Science and Engineering S-Lab Computer and Information Science Deep learning Video quality assessment Compared with spatial counterparts, temporal relationships between frames and their influences on video quality assessment (VQA) are still relatively under-studied in existing works. These relationships lead to two important types of effects for video quality. Firstly, some meaningless temporal variations (such as shaking, flicker, and unsmooth scene transitions) cause temporal distortions that degrade quality of videos. Secondly, the human visual system often has different attention to frames with different contents, resulting in their different importance to the overall video quality. Based on prominent time-series modeling ability of transformers, we propose a novel and effective transformer-based VQA method to tackle these two issues. To better differentiate temporal variations and thus capture the temporal distortions, we design the Spatial-Temporal Distortion Extraction (STDE) module that extracts multi-level spatial-temporal features with a video swin transformer tiny (Swin-T) backbone and uses temporal difference layer to further capture these distortions. To tackle with temporal quality attention, we propose the encoder-decoder-like temporal content transformer (TCT). We also introduce the temporal sampling on features to reduce the input length for the TCT, so as to improve the learning effectiveness and efficiency of this module. Consisting of the STDE and the TCT, the proposed Temporal Distortion-Content Transformers for Video Quality Assessment (DisCoVQA) reaches state-of-the-art performance on several VQA benchmarks without any extra pre-training datasets and up to 10% better generalization ability than existing methods. We also conduct extensive ablation experiments to prove the effectiveness of each part in our proposed model, and provide visualizations to prove that the proposed modules achieve our intention on modeling these temporal issues. Our code is published at https://github.com/QualityAssessment/DisCoVQA. This work was supported in part by the RIE2020 Industry Alignment Fund–Industry Collaboration Projects (IAF-ICP) Funding Initiative, and in part by the Cash and In-Kind Contribution From the Industry Partner’s. 2024-06-20T07:26:29Z 2024-06-20T07:26:29Z 2023 Journal Article Wu, H., Chen, C., Liao, L., Hou, J., Sun, W., Yan, Q. & Lin, W. (2023). DisCoVQA: temporal distortion-content transformers for video quality assessment. IEEE Transactions On Circuits and Systems for Video Technology, 33(9), 4840-4854. https://dx.doi.org/10.1109/TCSVT.2023.3249741 1051-8215 https://hdl.handle.net/10356/178454 10.1109/TCSVT.2023.3249741 2-s2.0-85149392109 9 33 4840 4854 en IEEE Transactions on Circuits and Systems for Video Technology © 2023 IEEE. All rights reserved. |
| spellingShingle | Computer and Information Science Deep learning Video quality assessment Wu, Haoning Chen, Chaofeng Liao, Liang Hou, Jingwen Sun, Wenxiu Yan, Qiong Lin, Weisi DisCoVQA: temporal distortion-content transformers for video quality assessment |
| title | DisCoVQA: temporal distortion-content transformers for video quality assessment |
| title_full | DisCoVQA: temporal distortion-content transformers for video quality assessment |
| title_fullStr | DisCoVQA: temporal distortion-content transformers for video quality assessment |
| title_full_unstemmed | DisCoVQA: temporal distortion-content transformers for video quality assessment |
| title_short | DisCoVQA: temporal distortion-content transformers for video quality assessment |
| title_sort | discovqa temporal distortion content transformers for video quality assessment |
| topic | Computer and Information Science Deep learning Video quality assessment |
| url | https://hdl.handle.net/10356/178454 |
| work_keys_str_mv | AT wuhaoning discovqatemporaldistortioncontenttransformersforvideoqualityassessment AT chenchaofeng discovqatemporaldistortioncontenttransformersforvideoqualityassessment AT liaoliang discovqatemporaldistortioncontenttransformersforvideoqualityassessment AT houjingwen discovqatemporaldistortioncontenttransformersforvideoqualityassessment AT sunwenxiu discovqatemporaldistortioncontenttransformersforvideoqualityassessment AT yanqiong discovqatemporaldistortioncontenttransformersforvideoqualityassessment AT linweisi discovqatemporaldistortioncontenttransformersforvideoqualityassessment |