Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding
Multi-spectral satellite remote sensing is a primary way to monitor planet-scale events such as deforestation, land-cover change, fire, and flooding. Unfortunately, incomplete spatial coverage and sparse temporal sampling make it challenging to develop a unified understanding of the environment. We...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/150309 |
| _version_ | 1826206888627273728 |
|---|---|
| author | Huang, Kuan Wei |
| author2 | Freeman, William T. |
| author_facet | Freeman, William T. Huang, Kuan Wei |
| author_sort | Huang, Kuan Wei |
| collection | MIT |
| description | Multi-spectral satellite remote sensing is a primary way to monitor planet-scale events such as deforestation, land-cover change, fire, and flooding. Unfortunately, incomplete spatial coverage and sparse temporal sampling make it challenging to develop a unified understanding of the environment. We aim to solve these challenges by creating a curated multi-modal satellite remote sensing dataset and presenting a novel architecture that learns a unified representation across large-scale heterogeneous remote sensing data by solving an image completion task. We equip our model with temporal, spectral, and global positioning information in addition to local positional encoding. This allows our algorithm to learn a unified, high-resolution, and time-varying representation across the entire survey area. Unlike the prior work, our architecture does not require data with uniform coverage, temporal resolution, or paired bands, and through prompting, it can act as a method for satellite infilling, temporal prediction, and cross-band translation. We train and evaluate our approach on a multi-modal remote sensing dataset and show that it outperforms baselines across satellite completion and cross-band translation tasks. In addition, we show that the neural feature field learned by our method is more effective than baselines for transfer learning to predict Amazon rainforest deforestation. |
| first_indexed | 2024-09-23T13:39:52Z |
| format | Thesis |
| id | mit-1721.1/150309 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T13:39:52Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1503092023-04-01T03:27:19Z Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding Huang, Kuan Wei Freeman, William T. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Multi-spectral satellite remote sensing is a primary way to monitor planet-scale events such as deforestation, land-cover change, fire, and flooding. Unfortunately, incomplete spatial coverage and sparse temporal sampling make it challenging to develop a unified understanding of the environment. We aim to solve these challenges by creating a curated multi-modal satellite remote sensing dataset and presenting a novel architecture that learns a unified representation across large-scale heterogeneous remote sensing data by solving an image completion task. We equip our model with temporal, spectral, and global positioning information in addition to local positional encoding. This allows our algorithm to learn a unified, high-resolution, and time-varying representation across the entire survey area. Unlike the prior work, our architecture does not require data with uniform coverage, temporal resolution, or paired bands, and through prompting, it can act as a method for satellite infilling, temporal prediction, and cross-band translation. We train and evaluate our approach on a multi-modal remote sensing dataset and show that it outperforms baselines across satellite completion and cross-band translation tasks. In addition, we show that the neural feature field learned by our method is more effective than baselines for transfer learning to predict Amazon rainforest deforestation. M.Eng. 2023-03-31T14:46:47Z 2023-03-31T14:46:47Z 2023-02 2023-02-27T18:43:22.491Z Thesis https://hdl.handle.net/1721.1/150309 In Copyright - Educational Use Permitted Copyright MIT http://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Huang, Kuan Wei Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding |
| title | Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding |
| title_full | Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding |
| title_fullStr | Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding |
| title_full_unstemmed | Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding |
| title_short | Combining Masked Autoencoding and Neural Fields for Multi-band Satellite Understanding |
| title_sort | combining masked autoencoding and neural fields for multi band satellite understanding |
| url | https://hdl.handle.net/1721.1/150309 |
| work_keys_str_mv | AT huangkuanwei combiningmaskedautoencodingandneuralfieldsformultibandsatelliteunderstanding |