Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image Synthesis
In the field of SAR-to-optical image synthesis, current methods based on conditional generative adversarial networks (CGANs) have satisfying performance under simple scenarios, but the performance drops severely under complicated scenarios. Considering that SAR images can form a robust time series d...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-03-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/7/1863 |
| _version_ | 1797607152898015232 |
|---|---|
| author | Yongchun Weng Yong Ma Fu Chen Erping Shang Wutao Yao Shuyan Zhang Jin Yang Jianbo Liu |
| author_facet | Yongchun Weng Yong Ma Fu Chen Erping Shang Wutao Yao Shuyan Zhang Jin Yang Jianbo Liu |
| author_sort | Yongchun Weng |
| collection | DOAJ |
| description | In the field of SAR-to-optical image synthesis, current methods based on conditional generative adversarial networks (CGANs) have satisfying performance under simple scenarios, but the performance drops severely under complicated scenarios. Considering that SAR images can form a robust time series due to SAR’s all-weather imaging ability, we take advantage of this and extract a temporal correlation from bi-temporal SAR images to guide the translation. To achieve this, we introduce a co-attention mechanism into the CGAN that learns the correlation between optically-available and optically-absent time points, selectively enhances the features of the former time point, and eventually guides the model to a better optical image synthesis on the latter time point. Additionally, we adopt a strategy to balance the weight of optical and SAR features to extract better features from the SAR input. With these strategies, the quality of synthesized images is notably improved in complicated scenarios. The synthesized images can increase the spatial and temporal resolution of optical imagery, greatly improving the availability of data for the applications of crop monitoring, change detection, and visual interpretation. |
| first_indexed | 2024-03-11T05:26:12Z |
| format | Article |
| id | doaj.art-d06f6a8ac21a410289d2fdbb1f5715f4 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-11T05:26:12Z |
| publishDate | 2023-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-d06f6a8ac21a410289d2fdbb1f5715f42023-11-17T17:30:04ZengMDPI AGRemote Sensing2072-42922023-03-01157186310.3390/rs15071863Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image SynthesisYongchun Weng0Yong Ma1Fu Chen2Erping Shang3Wutao Yao4Shuyan Zhang5Jin Yang6Jianbo Liu7Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, ChinaIn the field of SAR-to-optical image synthesis, current methods based on conditional generative adversarial networks (CGANs) have satisfying performance under simple scenarios, but the performance drops severely under complicated scenarios. Considering that SAR images can form a robust time series due to SAR’s all-weather imaging ability, we take advantage of this and extract a temporal correlation from bi-temporal SAR images to guide the translation. To achieve this, we introduce a co-attention mechanism into the CGAN that learns the correlation between optically-available and optically-absent time points, selectively enhances the features of the former time point, and eventually guides the model to a better optical image synthesis on the latter time point. Additionally, we adopt a strategy to balance the weight of optical and SAR features to extract better features from the SAR input. With these strategies, the quality of synthesized images is notably improved in complicated scenarios. The synthesized images can increase the spatial and temporal resolution of optical imagery, greatly improving the availability of data for the applications of crop monitoring, change detection, and visual interpretation.https://www.mdpi.com/2072-4292/15/7/1863attention mechanismgenerative adversarial networks (GANs)image-to-image translationsynthetic aperture radar (SAR) |
| spellingShingle | Yongchun Weng Yong Ma Fu Chen Erping Shang Wutao Yao Shuyan Zhang Jin Yang Jianbo Liu Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image Synthesis Remote Sensing attention mechanism generative adversarial networks (GANs) image-to-image translation synthetic aperture radar (SAR) |
| title | Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image Synthesis |
| title_full | Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image Synthesis |
| title_fullStr | Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image Synthesis |
| title_full_unstemmed | Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image Synthesis |
| title_short | Temporal Co-Attention Guided Conditional Generative Adversarial Network for Optical Image Synthesis |
| title_sort | temporal co attention guided conditional generative adversarial network for optical image synthesis |
| topic | attention mechanism generative adversarial networks (GANs) image-to-image translation synthetic aperture radar (SAR) |
| url | https://www.mdpi.com/2072-4292/15/7/1863 |
| work_keys_str_mv | AT yongchunweng temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis AT yongma temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis AT fuchen temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis AT erpingshang temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis AT wutaoyao temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis AT shuyanzhang temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis AT jinyang temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis AT jianboliu temporalcoattentionguidedconditionalgenerativeadversarialnetworkforopticalimagesynthesis |