SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog Detection
Sea fog detection (SFD) presents a significant challenge in the field of intelligent Earth observation, particularly in analyzing meteorological satellite imagery. Akin to various vision tasks, ImageNet pre-training is commonly used for pre-training SFD. However, in the context of multi-spectral met...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-08-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/16/4102 |
| _version_ | 1827728760943673344 |
|---|---|
| author | Haotian Yan Sundingkai Su Ming Wu Mengqiu Xu Yihao Zuo Chuang Zhang Bin Huang |
| author_facet | Haotian Yan Sundingkai Su Ming Wu Mengqiu Xu Yihao Zuo Chuang Zhang Bin Huang |
| author_sort | Haotian Yan |
| collection | DOAJ |
| description | Sea fog detection (SFD) presents a significant challenge in the field of intelligent Earth observation, particularly in analyzing meteorological satellite imagery. Akin to various vision tasks, ImageNet pre-training is commonly used for pre-training SFD. However, in the context of multi-spectral meteorological satellite imagery, the initial step of deep learning has received limited attention. Recently, pre-training with Very High-Resolution (VHR) satellite imagery has gained increased popularity in remote-sensing vision tasks, showing the potential to replace ImageNet pre-training. However, it is worth noting that the meteorological satellite imagery applied in SFD, despite being an application of computer vision in remote sensing, differs greatly from VHR satellite imagery. To address the limitation of pre-training for SFD, this paper introduces a novel deep-learning paradigm to the meteorological domain driven by Masked Image Modeling (MIM). Our research reveals two key insights: (1) Pre-training with meteorological satellite imagery yields superior SFD performance compared to pre-training with nature imagery and VHR satellite imagery. (2) Incorporating the architectural characteristics of SFD models into a vanilla masked autoencoder (MAE) can augment the effectiveness of meteorological pre-training. To facilitate this research, we curate a pre-training dataset comprising 514,655 temporal multi-spectral meteorological satellite images, covering the Bohai Sea and Yellow Sea regions, which have the most sea fog occurrence. The longitude ranges from 115.00E to 128.75E, and the latitude ranges from 27.60N to 41.35N. Moreover, we introduce SeaMAE, a novel MAE that utilizes a Vision Transformer as the encoder and a convolutional hierarchical decoder, to learn meteorological representations. SeaMAE is pre-trained on this dataset and fine-tuned for SFD, resulting in state-of-the-art performance. For instance, using the ViT-Base as the backbone, SeaMAE pre-training which achieves <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>64.18</mn><mo>%</mo></mrow></semantics></math></inline-formula> surpasses from-scratch learning, natural imagery pre-training, and VRH satellite imagery pre-training by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>5.53</mn><mo>%</mo></mrow></semantics></math></inline-formula>, <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.49</mn><mo>%</mo></mrow></semantics></math></inline-formula>, and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.21</mn><mo>%</mo></mrow></semantics></math></inline-formula>, respectively, in terms of Intersection over Union of SFD. |
| first_indexed | 2024-03-10T23:36:29Z |
| format | Article |
| id | doaj.art-481e5a0757884e8da95dccb67db6202e |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T23:36:29Z |
| publishDate | 2023-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-481e5a0757884e8da95dccb67db6202e2023-11-19T02:54:34ZengMDPI AGRemote Sensing2072-42922023-08-011516410210.3390/rs15164102SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog DetectionHaotian Yan0Sundingkai Su1Ming Wu2Mengqiu Xu3Yihao Zuo4Chuang Zhang5Bin Huang6Artificial Intelligence School, Beijing University of Posts and Telecommunications, Beijing 100876, ChinaArtificial Intelligence School, Beijing University of Posts and Telecommunications, Beijing 100876, ChinaArtificial Intelligence School, Beijing University of Posts and Telecommunications, Beijing 100876, ChinaArtificial Intelligence School, Beijing University of Posts and Telecommunications, Beijing 100876, ChinaArtificial Intelligence School, Beijing University of Posts and Telecommunications, Beijing 100876, ChinaArtificial Intelligence School, Beijing University of Posts and Telecommunications, Beijing 100876, ChinaNational Meteorological Center, China Meteorological Administration, Beijing 100081, ChinaSea fog detection (SFD) presents a significant challenge in the field of intelligent Earth observation, particularly in analyzing meteorological satellite imagery. Akin to various vision tasks, ImageNet pre-training is commonly used for pre-training SFD. However, in the context of multi-spectral meteorological satellite imagery, the initial step of deep learning has received limited attention. Recently, pre-training with Very High-Resolution (VHR) satellite imagery has gained increased popularity in remote-sensing vision tasks, showing the potential to replace ImageNet pre-training. However, it is worth noting that the meteorological satellite imagery applied in SFD, despite being an application of computer vision in remote sensing, differs greatly from VHR satellite imagery. To address the limitation of pre-training for SFD, this paper introduces a novel deep-learning paradigm to the meteorological domain driven by Masked Image Modeling (MIM). Our research reveals two key insights: (1) Pre-training with meteorological satellite imagery yields superior SFD performance compared to pre-training with nature imagery and VHR satellite imagery. (2) Incorporating the architectural characteristics of SFD models into a vanilla masked autoencoder (MAE) can augment the effectiveness of meteorological pre-training. To facilitate this research, we curate a pre-training dataset comprising 514,655 temporal multi-spectral meteorological satellite images, covering the Bohai Sea and Yellow Sea regions, which have the most sea fog occurrence. The longitude ranges from 115.00E to 128.75E, and the latitude ranges from 27.60N to 41.35N. Moreover, we introduce SeaMAE, a novel MAE that utilizes a Vision Transformer as the encoder and a convolutional hierarchical decoder, to learn meteorological representations. SeaMAE is pre-trained on this dataset and fine-tuned for SFD, resulting in state-of-the-art performance. For instance, using the ViT-Base as the backbone, SeaMAE pre-training which achieves <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>64.18</mn><mo>%</mo></mrow></semantics></math></inline-formula> surpasses from-scratch learning, natural imagery pre-training, and VRH satellite imagery pre-training by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>5.53</mn><mo>%</mo></mrow></semantics></math></inline-formula>, <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.49</mn><mo>%</mo></mrow></semantics></math></inline-formula>, and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.21</mn><mo>%</mo></mrow></semantics></math></inline-formula>, respectively, in terms of Intersection over Union of SFD.https://www.mdpi.com/2072-4292/15/16/4102sea fog detectionpre-trainingmasked autoencodersmeteorological satellite imagery |
| spellingShingle | Haotian Yan Sundingkai Su Ming Wu Mengqiu Xu Yihao Zuo Chuang Zhang Bin Huang SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog Detection Remote Sensing sea fog detection pre-training masked autoencoders meteorological satellite imagery |
| title | SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog Detection |
| title_full | SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog Detection |
| title_fullStr | SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog Detection |
| title_full_unstemmed | SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog Detection |
| title_short | SeaMAE: Masked Pre-Training with Meteorological Satellite Imagery for Sea Fog Detection |
| title_sort | seamae masked pre training with meteorological satellite imagery for sea fog detection |
| topic | sea fog detection pre-training masked autoencoders meteorological satellite imagery |
| url | https://www.mdpi.com/2072-4292/15/16/4102 |
| work_keys_str_mv | AT haotianyan seamaemaskedpretrainingwithmeteorologicalsatelliteimageryforseafogdetection AT sundingkaisu seamaemaskedpretrainingwithmeteorologicalsatelliteimageryforseafogdetection AT mingwu seamaemaskedpretrainingwithmeteorologicalsatelliteimageryforseafogdetection AT mengqiuxu seamaemaskedpretrainingwithmeteorologicalsatelliteimageryforseafogdetection AT yihaozuo seamaemaskedpretrainingwithmeteorologicalsatelliteimageryforseafogdetection AT chuangzhang seamaemaskedpretrainingwithmeteorologicalsatelliteimageryforseafogdetection AT binhuang seamaemaskedpretrainingwithmeteorologicalsatelliteimageryforseafogdetection |