NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping
NASA NeMO-Net, The Neural Multimodal Observation and Training Network for global coral reef assessment, is a convolutional neural network (CNN) that generates benthic habitat maps of coral reefs and other shallow marine ecosystems. To segment and classify imagery accurately, CNNs require curated tra...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2021-04-01
|
| Series: | Frontiers in Marine Science |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fmars.2021.645408/full |
| _version_ | 1818956387615506432 |
|---|---|
| author | Jarrett van den Bergh Ved Chirayath Alan Li Juan L. Torres-Pérez Michal Segal-Rozenhaimer Michal Segal-Rozenhaimer |
| author_facet | Jarrett van den Bergh Ved Chirayath Alan Li Juan L. Torres-Pérez Michal Segal-Rozenhaimer Michal Segal-Rozenhaimer |
| author_sort | Jarrett van den Bergh |
| collection | DOAJ |
| description | NASA NeMO-Net, The Neural Multimodal Observation and Training Network for global coral reef assessment, is a convolutional neural network (CNN) that generates benthic habitat maps of coral reefs and other shallow marine ecosystems. To segment and classify imagery accurately, CNNs require curated training datasets of considerable volume and accuracy. Here, we present a citizen science approach to create these training datasets through a novel 3D classification game for mobile and desktop devices. Leveraging citizen science, the NeMO-Net video game generates high-resolution 3D benthic habitat labels at the subcentimeter to meter scales. The video game trains users to accurately identify benthic categories and semantically segment 3D scenes captured using NASA airborne fluid lensing, the first remote sensing technology capable of mitigating ocean wave distortions, as well as in situ 3D photogrammetry and 2D satellite remote sensing. An active learning framework is used in the game to allow users to rate and edit other user classifications, dynamically improving segmentation accuracy. Refined and aggregated data labels from the game are used to train NeMO-Net’s supercomputer-based CNN to autonomously map shallow marine systems and augment satellite habitat mapping accuracy in these regions. We share the NeMO-Net game approach to user training and retention, outline the 3D labeling technique developed to accurately label complex coral reef imagery, and present preliminary results from over 70,000 user classifications. To overcome the inherent variability of citizen science, we analyze criteria and metrics for evaluating and filtering user data. Finally, we examine how future citizen science and machine learning approaches might benefit from label training in 3D space using an active learning framework. Within 7 months of launch, NeMO-Net has reached over 300 million people globally and directly engaged communities in coral reef mapping and conservation through ongoing scientific field campaigns, uninhibited by geography, language, or physical ability. As more user data are fed into NeMO-Net’s CNN, it will produce the first shallow-marine habitat mapping products trained on 3D subcm-scale label data and merged with m-scale satellite data that could be applied globally when data sets are available. |
| first_indexed | 2024-12-20T10:53:09Z |
| format | Article |
| id | doaj.art-5e2ede03068841858ce44cc2a99e0967 |
| institution | Directory Open Access Journal |
| issn | 2296-7745 |
| language | English |
| last_indexed | 2024-12-20T10:53:09Z |
| publishDate | 2021-04-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Marine Science |
| spelling | doaj.art-5e2ede03068841858ce44cc2a99e09672022-12-21T19:43:12ZengFrontiers Media S.A.Frontiers in Marine Science2296-77452021-04-01810.3389/fmars.2021.645408645408NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat MappingJarrett van den Bergh0Ved Chirayath1Alan Li2Juan L. Torres-Pérez3Michal Segal-Rozenhaimer4Michal Segal-Rozenhaimer5NASA Laboratory for Advanced Sensing, Earth Science Division, NASA Silicon Valley Ames Research Center, Mountain View, CA, United StatesNASA Laboratory for Advanced Sensing, Earth Science Division, NASA Silicon Valley Ames Research Center, Mountain View, CA, United StatesNASA Laboratory for Advanced Sensing, Earth Science Division, NASA Silicon Valley Ames Research Center, Mountain View, CA, United StatesNASA Laboratory for Advanced Sensing, Earth Science Division, NASA Silicon Valley Ames Research Center, Mountain View, CA, United StatesNASA Laboratory for Advanced Sensing, Earth Science Division, NASA Silicon Valley Ames Research Center, Mountain View, CA, United StatesDepartment of Geophysics, Porter School of the Environment and Earth Sciences, Tel Aviv University, Tel-Aviv, IsraelNASA NeMO-Net, The Neural Multimodal Observation and Training Network for global coral reef assessment, is a convolutional neural network (CNN) that generates benthic habitat maps of coral reefs and other shallow marine ecosystems. To segment and classify imagery accurately, CNNs require curated training datasets of considerable volume and accuracy. Here, we present a citizen science approach to create these training datasets through a novel 3D classification game for mobile and desktop devices. Leveraging citizen science, the NeMO-Net video game generates high-resolution 3D benthic habitat labels at the subcentimeter to meter scales. The video game trains users to accurately identify benthic categories and semantically segment 3D scenes captured using NASA airborne fluid lensing, the first remote sensing technology capable of mitigating ocean wave distortions, as well as in situ 3D photogrammetry and 2D satellite remote sensing. An active learning framework is used in the game to allow users to rate and edit other user classifications, dynamically improving segmentation accuracy. Refined and aggregated data labels from the game are used to train NeMO-Net’s supercomputer-based CNN to autonomously map shallow marine systems and augment satellite habitat mapping accuracy in these regions. We share the NeMO-Net game approach to user training and retention, outline the 3D labeling technique developed to accurately label complex coral reef imagery, and present preliminary results from over 70,000 user classifications. To overcome the inherent variability of citizen science, we analyze criteria and metrics for evaluating and filtering user data. Finally, we examine how future citizen science and machine learning approaches might benefit from label training in 3D space using an active learning framework. Within 7 months of launch, NeMO-Net has reached over 300 million people globally and directly engaged communities in coral reef mapping and conservation through ongoing scientific field campaigns, uninhibited by geography, language, or physical ability. As more user data are fed into NeMO-Net’s CNN, it will produce the first shallow-marine habitat mapping products trained on 3D subcm-scale label data and merged with m-scale satellite data that could be applied globally when data sets are available.https://www.frontiersin.org/articles/10.3389/fmars.2021.645408/fullcoral reefsremote sensingmachine learningcitizen sciencefluid lensingvideo game |
| spellingShingle | Jarrett van den Bergh Ved Chirayath Alan Li Juan L. Torres-Pérez Michal Segal-Rozenhaimer Michal Segal-Rozenhaimer NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping Frontiers in Marine Science coral reefs remote sensing machine learning citizen science fluid lensing video game |
| title | NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping |
| title_full | NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping |
| title_fullStr | NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping |
| title_full_unstemmed | NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping |
| title_short | NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping |
| title_sort | nemo net gamifying 3d labeling of multi modal reference datasets to support automated marine habitat mapping |
| topic | coral reefs remote sensing machine learning citizen science fluid lensing video game |
| url | https://www.frontiersin.org/articles/10.3389/fmars.2021.645408/full |
| work_keys_str_mv | AT jarrettvandenbergh nemonetgamifying3dlabelingofmultimodalreferencedatasetstosupportautomatedmarinehabitatmapping AT vedchirayath nemonetgamifying3dlabelingofmultimodalreferencedatasetstosupportautomatedmarinehabitatmapping AT alanli nemonetgamifying3dlabelingofmultimodalreferencedatasetstosupportautomatedmarinehabitatmapping AT juanltorresperez nemonetgamifying3dlabelingofmultimodalreferencedatasetstosupportautomatedmarinehabitatmapping AT michalsegalrozenhaimer nemonetgamifying3dlabelingofmultimodalreferencedatasetstosupportautomatedmarinehabitatmapping AT michalsegalrozenhaimer nemonetgamifying3dlabelingofmultimodalreferencedatasetstosupportautomatedmarinehabitatmapping |