Nighttime Thermal Infrared Image Translation Integrating Visible Images
Nighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn th...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-02-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/16/4/666 |
| _version_ | 1797297080965791744 |
|---|---|
| author | Shihao Yang Min Sun Xiayin Lou Hanjun Yang Dong Liu |
| author_facet | Shihao Yang Min Sun Xiayin Lou Hanjun Yang Dong Liu |
| author_sort | Shihao Yang |
| collection | DOAJ |
| description | Nighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn the mapping relationship between temperature and color domains, and translate NTIR images with one channel into DCV images with three channels. However, this mapping relationship is an ill-posed problem with multiple solutions without constraints, resulting in blurred edges, color disorder, and semantic errors. To solve this problem, an NTIR2DCV method that includes two steps is proposed: firstly, fuse Nighttime Color Visible (NCV) images with NTIR images based on an Illumination-Aware, Multilevel Decomposition Latent Low-Rank Representation (IA-MDLatLRR) method, which considers the differences in illumination conditions during image fusion and adjusts the fusion strategy of MDLatLRR accordingly to suppress the adverse effects of nighttime lights; secondly, translate the Nighttime Fused (NF) image to DCV image based on HyperDimensional Computing Generative Adversarial Network (HDC-GAN), which ensures feature-level semantic consistency between the source image (NF image) and the translated image (DCV image) without creating semantic label maps. Extensive comparative experiments and the evaluation metrics values show that the proposed algorithms perform better than other State-Of-The-Art (SOTA) image fusion and translation methods, such as FID and KID, which decreased by 14.1 and 18.9, respectively. |
| first_indexed | 2024-03-07T22:16:07Z |
| format | Article |
| id | doaj.art-8ffa7c5e6e7a427ab5ac1d8f1d5bc764 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-07T22:16:07Z |
| publishDate | 2024-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-8ffa7c5e6e7a427ab5ac1d8f1d5bc7642024-02-23T15:33:02ZengMDPI AGRemote Sensing2072-42922024-02-0116466610.3390/rs16040666Nighttime Thermal Infrared Image Translation Integrating Visible ImagesShihao Yang0Min Sun1Xiayin Lou2Hanjun Yang3Dong Liu4Institute of Remote Sensing and Geographic Information Systems, Peking University, 5 Summer Palace Road, Beijing 100871, ChinaInstitute of Remote Sensing and Geographic Information Systems, Peking University, 5 Summer Palace Road, Beijing 100871, ChinaInstitute of Remote Sensing and Geographic Information Systems, Peking University, 5 Summer Palace Road, Beijing 100871, ChinaInstitute of Remote Sensing and Geographic Information Systems, Peking University, 5 Summer Palace Road, Beijing 100871, ChinaInstitute of Remote Sensing and Geographic Information Systems, Peking University, 5 Summer Palace Road, Beijing 100871, ChinaNighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn the mapping relationship between temperature and color domains, and translate NTIR images with one channel into DCV images with three channels. However, this mapping relationship is an ill-posed problem with multiple solutions without constraints, resulting in blurred edges, color disorder, and semantic errors. To solve this problem, an NTIR2DCV method that includes two steps is proposed: firstly, fuse Nighttime Color Visible (NCV) images with NTIR images based on an Illumination-Aware, Multilevel Decomposition Latent Low-Rank Representation (IA-MDLatLRR) method, which considers the differences in illumination conditions during image fusion and adjusts the fusion strategy of MDLatLRR accordingly to suppress the adverse effects of nighttime lights; secondly, translate the Nighttime Fused (NF) image to DCV image based on HyperDimensional Computing Generative Adversarial Network (HDC-GAN), which ensures feature-level semantic consistency between the source image (NF image) and the translated image (DCV image) without creating semantic label maps. Extensive comparative experiments and the evaluation metrics values show that the proposed algorithms perform better than other State-Of-The-Art (SOTA) image fusion and translation methods, such as FID and KID, which decreased by 14.1 and 18.9, respectively.https://www.mdpi.com/2072-4292/16/4/666nighttime thermal infrared image translationimage fusionillumination maphyperdimensional computing |
| spellingShingle | Shihao Yang Min Sun Xiayin Lou Hanjun Yang Dong Liu Nighttime Thermal Infrared Image Translation Integrating Visible Images Remote Sensing nighttime thermal infrared image translation image fusion illumination map hyperdimensional computing |
| title | Nighttime Thermal Infrared Image Translation Integrating Visible Images |
| title_full | Nighttime Thermal Infrared Image Translation Integrating Visible Images |
| title_fullStr | Nighttime Thermal Infrared Image Translation Integrating Visible Images |
| title_full_unstemmed | Nighttime Thermal Infrared Image Translation Integrating Visible Images |
| title_short | Nighttime Thermal Infrared Image Translation Integrating Visible Images |
| title_sort | nighttime thermal infrared image translation integrating visible images |
| topic | nighttime thermal infrared image translation image fusion illumination map hyperdimensional computing |
| url | https://www.mdpi.com/2072-4292/16/4/666 |
| work_keys_str_mv | AT shihaoyang nighttimethermalinfraredimagetranslationintegratingvisibleimages AT minsun nighttimethermalinfraredimagetranslationintegratingvisibleimages AT xiayinlou nighttimethermalinfraredimagetranslationintegratingvisibleimages AT hanjunyang nighttimethermalinfraredimagetranslationintegratingvisibleimages AT dongliu nighttimethermalinfraredimagetranslationintegratingvisibleimages |