Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling
The small sizes of most marine plankton necessitate that plankton sampling occur on fine spatial scales, yet our questions often span large spatial areas. Underwater imaging can provide a solution to this sampling conundrum but collects large quantities of data that require an automated approach to...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-06-01
|
| Series: | Frontiers in Marine Science |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fmars.2023.1187771/full |
| _version_ | 1797808876737789952 |
|---|---|
| author | Moritz S. Schmid Dominic Daprano Malhar M. Damle Christopher M. Sullivan Christopher M. Sullivan Su Sponaugle Su Sponaugle Charles Cousin Cedric Guigand Robert K. Cowen |
| author_facet | Moritz S. Schmid Dominic Daprano Malhar M. Damle Christopher M. Sullivan Christopher M. Sullivan Su Sponaugle Su Sponaugle Charles Cousin Cedric Guigand Robert K. Cowen |
| author_sort | Moritz S. Schmid |
| collection | DOAJ |
| description | The small sizes of most marine plankton necessitate that plankton sampling occur on fine spatial scales, yet our questions often span large spatial areas. Underwater imaging can provide a solution to this sampling conundrum but collects large quantities of data that require an automated approach to image analysis. Machine learning for plankton classification, and high-performance computing (HPC) infrastructure, are critical to rapid image processing; however, these assets, especially HPC infrastructure, are only available post-cruise leading to an ‘after-the-fact’ view of plankton community structure. To be responsive to the often-ephemeral nature of oceanographic features and species assemblages in highly dynamic current systems, real-time data are key for adaptive oceanographic sampling. Here we used the new In-situ Ichthyoplankton Imaging System-3 (ISIIS-3) in the Northern California Current (NCC) in conjunction with an edge server to classify imaged plankton in real-time into 170 classes. This capability together with data visualization in a heavy.ai dashboard makes adaptive real-time decision-making and sampling at sea possible. Dual ISIIS-Deep-focus Particle Imager (DPI) cameras sample 180 L s-1, leading to >10 GB of video per min. Imaged organisms are in the size range of 250 µm to 15 cm and include abundant crustaceans, fragile taxa (e.g., hydromedusae, salps), faster swimmers (e.g., krill), and rarer taxa (e.g., larval fishes). A deep learning pipeline deployed on the edge server used multithreaded CPU-based segmentation and GPU-based classification to process the imagery. AVI videos contain 50 sec of data and can contain between 23,000 - 225,000 particle and plankton segments. Processing one AVI through segmentation and classification takes on average 3.75 mins, depending on biological productivity. A heavyDB database monitors for newly processed data and is linked to a heavy.ai dashboard for interactive data visualization. We describe several examples where imaging, AI, and data visualization enable adaptive sampling that can have a transformative effect on oceanography. We envision AI-enabled adaptive sampling to have a high impact on our ability to resolve biological responses to important oceanographic features in the NCC, such as oxygen minimum zones, or harmful algal bloom thin layers, which affect the health of the ecosystem, fisheries, and local communities. |
| first_indexed | 2024-03-13T06:44:12Z |
| format | Article |
| id | doaj.art-5906644abb3942f58c6bd86a69ba7df0 |
| institution | Directory Open Access Journal |
| issn | 2296-7745 |
| language | English |
| last_indexed | 2024-03-13T06:44:12Z |
| publishDate | 2023-06-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Marine Science |
| spelling | doaj.art-5906644abb3942f58c6bd86a69ba7df02023-06-08T11:44:18ZengFrontiers Media S.A.Frontiers in Marine Science2296-77452023-06-011010.3389/fmars.2023.11877711187771Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive samplingMoritz S. Schmid0Dominic Daprano1Malhar M. Damle2Christopher M. Sullivan3Christopher M. Sullivan4Su Sponaugle5Su Sponaugle6Charles Cousin7Cedric Guigand8Robert K. Cowen9Hatfield Marine Science Center, Oregon State University, Newport, OR, United StatesCenter for Quantitative and Life Sciences, Oregon State University, Corvallis, OR, United StatesCenter for Quantitative and Life Sciences, Oregon State University, Corvallis, OR, United StatesCenter for Quantitative and Life Sciences, Oregon State University, Corvallis, OR, United StatesCollege of Earth, Ocean, and Atmospheric Sciences, Oregon State University, Corvallis, OR, United StatesHatfield Marine Science Center, Oregon State University, Newport, OR, United StatesDepartment of Integrative Biology, Oregon State University, Corvallis, OR, United StatesBellamare LLC, San Diego, CA, United StatesBellamare LLC, San Diego, CA, United StatesHatfield Marine Science Center, Oregon State University, Newport, OR, United StatesThe small sizes of most marine plankton necessitate that plankton sampling occur on fine spatial scales, yet our questions often span large spatial areas. Underwater imaging can provide a solution to this sampling conundrum but collects large quantities of data that require an automated approach to image analysis. Machine learning for plankton classification, and high-performance computing (HPC) infrastructure, are critical to rapid image processing; however, these assets, especially HPC infrastructure, are only available post-cruise leading to an ‘after-the-fact’ view of plankton community structure. To be responsive to the often-ephemeral nature of oceanographic features and species assemblages in highly dynamic current systems, real-time data are key for adaptive oceanographic sampling. Here we used the new In-situ Ichthyoplankton Imaging System-3 (ISIIS-3) in the Northern California Current (NCC) in conjunction with an edge server to classify imaged plankton in real-time into 170 classes. This capability together with data visualization in a heavy.ai dashboard makes adaptive real-time decision-making and sampling at sea possible. Dual ISIIS-Deep-focus Particle Imager (DPI) cameras sample 180 L s-1, leading to >10 GB of video per min. Imaged organisms are in the size range of 250 µm to 15 cm and include abundant crustaceans, fragile taxa (e.g., hydromedusae, salps), faster swimmers (e.g., krill), and rarer taxa (e.g., larval fishes). A deep learning pipeline deployed on the edge server used multithreaded CPU-based segmentation and GPU-based classification to process the imagery. AVI videos contain 50 sec of data and can contain between 23,000 - 225,000 particle and plankton segments. Processing one AVI through segmentation and classification takes on average 3.75 mins, depending on biological productivity. A heavyDB database monitors for newly processed data and is linked to a heavy.ai dashboard for interactive data visualization. We describe several examples where imaging, AI, and data visualization enable adaptive sampling that can have a transformative effect on oceanography. We envision AI-enabled adaptive sampling to have a high impact on our ability to resolve biological responses to important oceanographic features in the NCC, such as oxygen minimum zones, or harmful algal bloom thin layers, which affect the health of the ecosystem, fisheries, and local communities.https://www.frontiersin.org/articles/10.3389/fmars.2023.1187771/fulladaptive samplingedge computingocean technologyunderwater imagingplankton ecologymachine learning |
| spellingShingle | Moritz S. Schmid Dominic Daprano Malhar M. Damle Christopher M. Sullivan Christopher M. Sullivan Su Sponaugle Su Sponaugle Charles Cousin Cedric Guigand Robert K. Cowen Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling Frontiers in Marine Science adaptive sampling edge computing ocean technology underwater imaging plankton ecology machine learning |
| title | Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling |
| title_full | Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling |
| title_fullStr | Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling |
| title_full_unstemmed | Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling |
| title_short | Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling |
| title_sort | edge computing at sea high throughput classification of in situ plankton imagery for adaptive sampling |
| topic | adaptive sampling edge computing ocean technology underwater imaging plankton ecology machine learning |
| url | https://www.frontiersin.org/articles/10.3389/fmars.2023.1187771/full |
| work_keys_str_mv | AT moritzsschmid edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT dominicdaprano edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT malharmdamle edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT christophermsullivan edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT christophermsullivan edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT susponaugle edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT susponaugle edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT charlescousin edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT cedricguigand edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling AT robertkcowen edgecomputingatseahighthroughputclassificationofinsituplanktonimageryforadaptivesampling |