Attention Networks for the Quality Enhancement of Light Field Images
In this paper, we propose a novel filtering method based on deep attention networks for the quality enhancement of light field (LF) images captured by plenoptic cameras and compressed using the High Efficiency Video Coding (HEVC) standard. The proposed architecture was built using efficient complex...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-05-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/9/3246 |
| _version_ | 1797534962347409408 |
|---|---|
| author | Ionut Schiopu Adrian Munteanu |
| author_facet | Ionut Schiopu Adrian Munteanu |
| author_sort | Ionut Schiopu |
| collection | DOAJ |
| description | In this paper, we propose a novel filtering method based on deep attention networks for the quality enhancement of light field (LF) images captured by plenoptic cameras and compressed using the High Efficiency Video Coding (HEVC) standard. The proposed architecture was built using efficient complex processing blocks and novel attention-based residual blocks. The network takes advantage of the macro-pixel (MP) structure, specific to LF images, and processes each reconstructed MP in the luminance (Y) channel. The input patch is represented as a tensor that collects, from an MP neighbourhood, four Epipolar Plane Images (EPIs) at four different angles. The experimental results on a common LF image database showed high improvements over HEVC in terms of the structural similarity index (SSIM), with an average Y-Bjøntegaard Delta (BD)-rate savings of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>36.57</mn><mo>%</mo><mo>,</mo></mrow></semantics></math></inline-formula> and an average Y-BD-PSNR improvement of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.301</mn></mrow></semantics></math></inline-formula> dB. Increased performance was achieved when the HEVC built-in filtering methods were skipped. The visual results illustrate that the enhanced image contains sharper edges and more texture details. The ablation study provides two robust solutions to reduce the inference time by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>44.6</mn><mo>%</mo></mrow></semantics></math></inline-formula> and the network complexity by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>74.7</mn><mo>%</mo></mrow></semantics></math></inline-formula>. The results demonstrate the potential of attention networks for the quality enhancement of LF images encoded by HEVC. |
| first_indexed | 2024-03-10T11:37:56Z |
| format | Article |
| id | doaj.art-d4aa81f7c58a4b858f473f8a4e2c2531 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T11:37:56Z |
| publishDate | 2021-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-d4aa81f7c58a4b858f473f8a4e2c25312023-11-21T18:44:11ZengMDPI AGSensors1424-82202021-05-01219324610.3390/s21093246Attention Networks for the Quality Enhancement of Light Field ImagesIonut Schiopu0Adrian Munteanu1Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, 1050 Brussels, BelgiumDepartment of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, 1050 Brussels, BelgiumIn this paper, we propose a novel filtering method based on deep attention networks for the quality enhancement of light field (LF) images captured by plenoptic cameras and compressed using the High Efficiency Video Coding (HEVC) standard. The proposed architecture was built using efficient complex processing blocks and novel attention-based residual blocks. The network takes advantage of the macro-pixel (MP) structure, specific to LF images, and processes each reconstructed MP in the luminance (Y) channel. The input patch is represented as a tensor that collects, from an MP neighbourhood, four Epipolar Plane Images (EPIs) at four different angles. The experimental results on a common LF image database showed high improvements over HEVC in terms of the structural similarity index (SSIM), with an average Y-Bjøntegaard Delta (BD)-rate savings of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>36.57</mn><mo>%</mo><mo>,</mo></mrow></semantics></math></inline-formula> and an average Y-BD-PSNR improvement of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.301</mn></mrow></semantics></math></inline-formula> dB. Increased performance was achieved when the HEVC built-in filtering methods were skipped. The visual results illustrate that the enhanced image contains sharper edges and more texture details. The ablation study provides two robust solutions to reduce the inference time by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>44.6</mn><mo>%</mo></mrow></semantics></math></inline-formula> and the network complexity by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>74.7</mn><mo>%</mo></mrow></semantics></math></inline-formula>. The results demonstrate the potential of attention networks for the quality enhancement of LF images encoded by HEVC.https://www.mdpi.com/1424-8220/21/9/3246attention networkquality enhancementlight field imagesvideo coding |
| spellingShingle | Ionut Schiopu Adrian Munteanu Attention Networks for the Quality Enhancement of Light Field Images Sensors attention network quality enhancement light field images video coding |
| title | Attention Networks for the Quality Enhancement of Light Field Images |
| title_full | Attention Networks for the Quality Enhancement of Light Field Images |
| title_fullStr | Attention Networks for the Quality Enhancement of Light Field Images |
| title_full_unstemmed | Attention Networks for the Quality Enhancement of Light Field Images |
| title_short | Attention Networks for the Quality Enhancement of Light Field Images |
| title_sort | attention networks for the quality enhancement of light field images |
| topic | attention network quality enhancement light field images video coding |
| url | https://www.mdpi.com/1424-8220/21/9/3246 |
| work_keys_str_mv | AT ionutschiopu attentionnetworksforthequalityenhancementoflightfieldimages AT adrianmunteanu attentionnetworksforthequalityenhancementoflightfieldimages |