Exploring the effectiveness of video perceptual representation in blind video quality assessment
With the rapid growth of in-The-wild videos taken by non-specialists, blind video quality assessment (VQA) has become a challenging and demanding problem. Although lots of efforts have been made to solve this problem, it remains unclear how the human visual system (HVS) relates to the temporal quali...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Conference Paper |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/178457 |
| _version_ | 1811694507115151360 |
|---|---|
| author | Liao, Liang Xu, Kangmin Wu, Haoning Chen, Chaofeng Sun, Wenxiu Yan, Qiong Lin, Weisi |
| author2 | College of Computing and Data Science |
| author_facet | College of Computing and Data Science Liao, Liang Xu, Kangmin Wu, Haoning Chen, Chaofeng Sun, Wenxiu Yan, Qiong Lin, Weisi |
| author_sort | Liao, Liang |
| collection | NTU |
| description | With the rapid growth of in-The-wild videos taken by non-specialists, blind video quality assessment (VQA) has become a challenging and demanding problem. Although lots of efforts have been made to solve this problem, it remains unclear how the human visual system (HVS) relates to the temporal quality of videos. Meanwhile, recent work has found that the frames of natural video transformed into the perceptual domain of the HVS tend to form a straight trajectory of the representations. With the obtained insight that distortion impairs the perceived video quality and results in a curved trajectory of the perceptual representation, we propose a temporal perceptual quality index (TPQI) to measure the temporal distortion by describing the graphic morphology of the representation. Specifically, we first extract the video perceptual representations from the lateral geniculate nucleus (LGN) and primary visual area (V1) of the HVS, and then measure the straightness and compactness of their trajectories to quantify the degradation in naturalness and content continuity of video. Experiments show that the perceptual representation in the HVS is an effective way of predicting subjective temporal quality, and thus TPQI can, for the first time, achieve comparable performance to the spatial quality metric and be even more effective in assessing videos with large temporal variations. We further demonstrate that by combining with NIQE, a spatial quality metric, TPQI can achieve top performance over popular in-The-wild video datasets. More importantly, TPQI does not require any additional information beyond the video being evaluated and thus can be applied to any datasets without parameter tuning. Source code is available at https://github.com/UoLMM/TPQI-VQA. |
| first_indexed | 2024-10-01T07:08:40Z |
| format | Conference Paper |
| id | ntu-10356/178457 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:08:40Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1784572024-06-21T01:03:41Z Exploring the effectiveness of video perceptual representation in blind video quality assessment Liao, Liang Xu, Kangmin Wu, Haoning Chen, Chaofeng Sun, Wenxiu Yan, Qiong Lin, Weisi College of Computing and Data Science School of Computer Science and Engineering 30th ACM International Conference on Multimedia (MM '22) S-Lab Computer and Information Science Perceptual trajectories Primary visual cortex With the rapid growth of in-The-wild videos taken by non-specialists, blind video quality assessment (VQA) has become a challenging and demanding problem. Although lots of efforts have been made to solve this problem, it remains unclear how the human visual system (HVS) relates to the temporal quality of videos. Meanwhile, recent work has found that the frames of natural video transformed into the perceptual domain of the HVS tend to form a straight trajectory of the representations. With the obtained insight that distortion impairs the perceived video quality and results in a curved trajectory of the perceptual representation, we propose a temporal perceptual quality index (TPQI) to measure the temporal distortion by describing the graphic morphology of the representation. Specifically, we first extract the video perceptual representations from the lateral geniculate nucleus (LGN) and primary visual area (V1) of the HVS, and then measure the straightness and compactness of their trajectories to quantify the degradation in naturalness and content continuity of video. Experiments show that the perceptual representation in the HVS is an effective way of predicting subjective temporal quality, and thus TPQI can, for the first time, achieve comparable performance to the spatial quality metric and be even more effective in assessing videos with large temporal variations. We further demonstrate that by combining with NIQE, a spatial quality metric, TPQI can achieve top performance over popular in-The-wild video datasets. More importantly, TPQI does not require any additional information beyond the video being evaluated and thus can be applied to any datasets without parameter tuning. Source code is available at https://github.com/UoLMM/TPQI-VQA. This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). 2024-06-21T01:03:41Z 2024-06-21T01:03:41Z 2022 Conference Paper Liao, L., Xu, K., Wu, H., Chen, C., Sun, W., Yan, Q. & Lin, W. (2022). Exploring the effectiveness of video perceptual representation in blind video quality assessment. 30th ACM International Conference on Multimedia (MM '22), 837-846. https://dx.doi.org/10.1145/3503161.3547849 9781450392037 https://hdl.handle.net/10356/178457 10.1145/3503161.3547849 2-s2.0-85140428186 837 846 en © 2022 Association for Computing Machinery. All rights reserved. |
| spellingShingle | Computer and Information Science Perceptual trajectories Primary visual cortex Liao, Liang Xu, Kangmin Wu, Haoning Chen, Chaofeng Sun, Wenxiu Yan, Qiong Lin, Weisi Exploring the effectiveness of video perceptual representation in blind video quality assessment |
| title | Exploring the effectiveness of video perceptual representation in blind video quality assessment |
| title_full | Exploring the effectiveness of video perceptual representation in blind video quality assessment |
| title_fullStr | Exploring the effectiveness of video perceptual representation in blind video quality assessment |
| title_full_unstemmed | Exploring the effectiveness of video perceptual representation in blind video quality assessment |
| title_short | Exploring the effectiveness of video perceptual representation in blind video quality assessment |
| title_sort | exploring the effectiveness of video perceptual representation in blind video quality assessment |
| topic | Computer and Information Science Perceptual trajectories Primary visual cortex |
| url | https://hdl.handle.net/10356/178457 |
| work_keys_str_mv | AT liaoliang exploringtheeffectivenessofvideoperceptualrepresentationinblindvideoqualityassessment AT xukangmin exploringtheeffectivenessofvideoperceptualrepresentationinblindvideoqualityassessment AT wuhaoning exploringtheeffectivenessofvideoperceptualrepresentationinblindvideoqualityassessment AT chenchaofeng exploringtheeffectivenessofvideoperceptualrepresentationinblindvideoqualityassessment AT sunwenxiu exploringtheeffectivenessofvideoperceptualrepresentationinblindvideoqualityassessment AT yanqiong exploringtheeffectivenessofvideoperceptualrepresentationinblindvideoqualityassessment AT linweisi exploringtheeffectivenessofvideoperceptualrepresentationinblindvideoqualityassessment |