K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation
Deep learning self-supervised algorithms that can segment an image in a fixed number of hard clusters such as the k-means algorithm and with an end-to-end deep learning approach are still lacking. Here, we introduce the k-textures algorithm which provides self-supervised segmentation of a 4-band ima...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-09-01
|
| Series: | Frontiers in Environmental Science |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fenvs.2022.946729/full |
| _version_ | 1811266377398353920 |
|---|---|
| author | Fabien H. Wagner Fabien H. Wagner Ricardo Dalagnol Ricardo Dalagnol Alber H. Sánchez Mayumi C. M. Hirye Samuel Favrichon Jake H. Lee Steffen Mauceri Yan Yang Sassan Saatchi Sassan Saatchi |
| author_facet | Fabien H. Wagner Fabien H. Wagner Ricardo Dalagnol Ricardo Dalagnol Alber H. Sánchez Mayumi C. M. Hirye Samuel Favrichon Jake H. Lee Steffen Mauceri Yan Yang Sassan Saatchi Sassan Saatchi |
| author_sort | Fabien H. Wagner |
| collection | DOAJ |
| description | Deep learning self-supervised algorithms that can segment an image in a fixed number of hard clusters such as the k-means algorithm and with an end-to-end deep learning approach are still lacking. Here, we introduce the k-textures algorithm which provides self-supervised segmentation of a 4-band image (RGB-NIR) for a k number of classes. An example of its application on high-resolution Planet satellite imagery is given. Our algorithm shows that discrete search is feasible using convolutional neural networks (CNN) and gradient descent. The model detects k hard clustering classes represented in the model as k discrete binary masks and their associated k independently generated textures, which combined are a simulation of the original image. The similarity loss is the mean squared error between the features of the original and the simulated image, both extracted from the penultimate convolutional block of Keras “imagenet” pre-trained VGG-16 model and a custom feature extractor made with Planet data. The main advances of the k-textures model are: first, the k discrete binary masks are obtained inside the model using gradient descent. The model allows for the generation of discrete binary masks using a novel method using a hard sigmoid activation function. Second, it provides hard clustering classes–each pixel has only one class. Finally, in comparison to k-means, where each pixel is considered independently, here, contextual information is also considered and each class is not associated only with similar values in the color channels but with a texture. Our approach is designed to ease the production of training samples for satellite image segmentation and the k-textures architecture could be adapted to support different numbers of bands and for more complex self-segmentation tasks, such as object self-segmentation. The model codes and weights are available at https://doi.org/10.5281/zenodo.6359859. |
| first_indexed | 2024-04-12T20:42:07Z |
| format | Article |
| id | doaj.art-ed0fedee47964307a9693c96ee1a8733 |
| institution | Directory Open Access Journal |
| issn | 2296-665X |
| language | English |
| last_indexed | 2024-04-12T20:42:07Z |
| publishDate | 2022-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Environmental Science |
| spelling | doaj.art-ed0fedee47964307a9693c96ee1a87332022-12-22T03:17:24ZengFrontiers Media S.A.Frontiers in Environmental Science2296-665X2022-09-011010.3389/fenvs.2022.946729946729K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentationFabien H. Wagner0Fabien H. Wagner1Ricardo Dalagnol2Ricardo Dalagnol3Alber H. Sánchez4Mayumi C. M. Hirye5Samuel Favrichon6Jake H. Lee7Steffen Mauceri8Yan Yang9Sassan Saatchi10Sassan Saatchi11Institute of the Environment and Sustainability, University of California, Los Angeles, Los Angeles, CA, United StatesNASA-Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United StatesInstitute of the Environment and Sustainability, University of California, Los Angeles, Los Angeles, CA, United StatesNASA-Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United StatesEarth Observation and Geoinformatics Division, National Institute for Space Research — INPE, São José dos Campos, SP, BrazilQuapá Lab, Faculty of Architecture and Urbanism, University of São Paulo — USP, São Paulo, SP, BrazilNASA-Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United StatesNASA-Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United StatesNASA-Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United StatesNASA-Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United StatesInstitute of the Environment and Sustainability, University of California, Los Angeles, Los Angeles, CA, United StatesNASA-Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United StatesDeep learning self-supervised algorithms that can segment an image in a fixed number of hard clusters such as the k-means algorithm and with an end-to-end deep learning approach are still lacking. Here, we introduce the k-textures algorithm which provides self-supervised segmentation of a 4-band image (RGB-NIR) for a k number of classes. An example of its application on high-resolution Planet satellite imagery is given. Our algorithm shows that discrete search is feasible using convolutional neural networks (CNN) and gradient descent. The model detects k hard clustering classes represented in the model as k discrete binary masks and their associated k independently generated textures, which combined are a simulation of the original image. The similarity loss is the mean squared error between the features of the original and the simulated image, both extracted from the penultimate convolutional block of Keras “imagenet” pre-trained VGG-16 model and a custom feature extractor made with Planet data. The main advances of the k-textures model are: first, the k discrete binary masks are obtained inside the model using gradient descent. The model allows for the generation of discrete binary masks using a novel method using a hard sigmoid activation function. Second, it provides hard clustering classes–each pixel has only one class. Finally, in comparison to k-means, where each pixel is considered independently, here, contextual information is also considered and each class is not associated only with similar values in the color channels but with a texture. Our approach is designed to ease the production of training samples for satellite image segmentation and the k-textures architecture could be adapted to support different numbers of bands and for more complex self-segmentation tasks, such as object self-segmentation. The model codes and weights are available at https://doi.org/10.5281/zenodo.6359859.https://www.frontiersin.org/articles/10.3389/fenvs.2022.946729/fulldeep learning - artificial neural networksegmentation (image processing)tropical forestlandcovertensorflow (2)self-supervised segmentation |
| spellingShingle | Fabien H. Wagner Fabien H. Wagner Ricardo Dalagnol Ricardo Dalagnol Alber H. Sánchez Mayumi C. M. Hirye Samuel Favrichon Jake H. Lee Steffen Mauceri Yan Yang Sassan Saatchi Sassan Saatchi K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation Frontiers in Environmental Science deep learning - artificial neural network segmentation (image processing) tropical forest landcover tensorflow (2) self-supervised segmentation |
| title | K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation |
| title_full | K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation |
| title_fullStr | K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation |
| title_full_unstemmed | K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation |
| title_short | K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation |
| title_sort | k textures a self supervised hard clustering deep learning algorithm for satellite image segmentation |
| topic | deep learning - artificial neural network segmentation (image processing) tropical forest landcover tensorflow (2) self-supervised segmentation |
| url | https://www.frontiersin.org/articles/10.3389/fenvs.2022.946729/full |
| work_keys_str_mv | AT fabienhwagner ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT fabienhwagner ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT ricardodalagnol ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT ricardodalagnol ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT alberhsanchez ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT mayumicmhirye ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT samuelfavrichon ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT jakehlee ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT steffenmauceri ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT yanyang ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT sassansaatchi ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation AT sassansaatchi ktexturesaselfsupervisedhardclusteringdeeplearningalgorithmforsatelliteimagesegmentation |