Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual Data
Ocean surface monitoring, emphasizing oil slick detection, has become essential due to its importance for oil exploration and ecosystem risk prevention. Automation is now mandatory since the manual annotation process of oil by photo-interpreters is time-consuming and cannot process the data collecte...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-07-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/14/15/3565 |
| _version_ | 1827624722975686656 |
|---|---|
| author | Emna Amri Pierre Dardouillet Alexandre Benoit Hermann Courteille Philippe Bolon Dominique Dubucq Anthony Credoz |
| author_facet | Emna Amri Pierre Dardouillet Alexandre Benoit Hermann Courteille Philippe Bolon Dominique Dubucq Anthony Credoz |
| author_sort | Emna Amri |
| collection | DOAJ |
| description | Ocean surface monitoring, emphasizing oil slick detection, has become essential due to its importance for oil exploration and ecosystem risk prevention. Automation is now mandatory since the manual annotation process of oil by photo-interpreters is time-consuming and cannot process the data collected continuously by the available spaceborne sensors. Studies on automatic detection methods mainly focus on Synthetic Aperture Radar (SAR) data exclusively to detect anthropogenic (spills) or natural (seeps) oil slicks, all using limited datasets. The main goal is to maximize the detection of oil slicks of both natures while being robust to other phenomena that generate false alarms, called “lookalikes”. To this end, this paper presents the automation of offshore oil slick detection on an extensive database of real and recent oil slick monitoring scenarios, including both types of slicks. It relies on slick annotations performed by expert photo-interpreters on Sentinel-1 SAR data over four years and three areas worldwide. In addition, contextual data such as wind estimates and infrastructure positions are included in the database as they are relevant data for oil detection. The contributions of this paper are: (i) A comparative study of deep learning approaches using SAR data. A semantic and instance segmentation analysis via FC-DenseNet and Mask R-CNN, respectively. (ii) A proposal for Fuse-FC-DenseNet, an extension of FC-DenseNet that fuses heterogeneous SAR and wind speed data for enhanced oil slick segmentation. (iii) An improved set of evaluation metrics dedicated to the task that considers contextual information. (iv) A visual explanation of deep learning predictions based on the SHapley Additive exPlanation (SHAP) method adapted to semantic segmentation. The proposed approach yields a detection performance of up to <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>94</mn><mo>%</mo></mrow></semantics></math></inline-formula> of good detection with a false alarm reduction ranging from <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>14</mn><mo>%</mo></mrow></semantics></math></inline-formula> to <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>34</mn><mo>%</mo></mrow></semantics></math></inline-formula> compared to mono-modal models. These results provide new solutions to improve the detection of natural and anthropogenic oil slicks by providing tools that allow photo-interpreters to work more efficiently on a wide range of marine surfaces to be monitored worldwide. Such a tool will accelerate the oil slick detection task to keep up with the continuous sensor acquisition. This upstream work will allow us to study its possible integration into an industrial production pipeline. In addition, a prediction explanation is proposed, which can be integrated as a step to identify the appropriate methodology for presenting the predictions to the experts and understanding the obtained predictions and their sensitivity to contextual information. Thus it helps them to optimize their way of working. |
| first_indexed | 2024-03-09T12:14:41Z |
| format | Article |
| id | doaj.art-468593c5f651458ba9c2b9a117bdad3c |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-09T12:14:41Z |
| publishDate | 2022-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-468593c5f651458ba9c2b9a117bdad3c2023-11-30T22:48:03ZengMDPI AGRemote Sensing2072-42922022-07-011415356510.3390/rs14153565Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual DataEmna Amri0Pierre Dardouillet1Alexandre Benoit2Hermann Courteille3Philippe Bolon4Dominique Dubucq5Anthony Credoz6LISTIC Laboratory, Polytech Annecy-Chambery, University of Savoie Mont Blanc, F-74944 Annecy le Vieux, FranceLISTIC Laboratory, Polytech Annecy-Chambery, University of Savoie Mont Blanc, F-74944 Annecy le Vieux, FranceLISTIC Laboratory, Polytech Annecy-Chambery, University of Savoie Mont Blanc, F-74944 Annecy le Vieux, FranceLISTIC Laboratory, Polytech Annecy-Chambery, University of Savoie Mont Blanc, F-74944 Annecy le Vieux, FranceLISTIC Laboratory, Polytech Annecy-Chambery, University of Savoie Mont Blanc, F-74944 Annecy le Vieux, FranceTotalEnergies S.E., Avenue Larribau, F-64018 Pau, FranceTotalEnergies S.E., Avenue Larribau, F-64018 Pau, FranceOcean surface monitoring, emphasizing oil slick detection, has become essential due to its importance for oil exploration and ecosystem risk prevention. Automation is now mandatory since the manual annotation process of oil by photo-interpreters is time-consuming and cannot process the data collected continuously by the available spaceborne sensors. Studies on automatic detection methods mainly focus on Synthetic Aperture Radar (SAR) data exclusively to detect anthropogenic (spills) or natural (seeps) oil slicks, all using limited datasets. The main goal is to maximize the detection of oil slicks of both natures while being robust to other phenomena that generate false alarms, called “lookalikes”. To this end, this paper presents the automation of offshore oil slick detection on an extensive database of real and recent oil slick monitoring scenarios, including both types of slicks. It relies on slick annotations performed by expert photo-interpreters on Sentinel-1 SAR data over four years and three areas worldwide. In addition, contextual data such as wind estimates and infrastructure positions are included in the database as they are relevant data for oil detection. The contributions of this paper are: (i) A comparative study of deep learning approaches using SAR data. A semantic and instance segmentation analysis via FC-DenseNet and Mask R-CNN, respectively. (ii) A proposal for Fuse-FC-DenseNet, an extension of FC-DenseNet that fuses heterogeneous SAR and wind speed data for enhanced oil slick segmentation. (iii) An improved set of evaluation metrics dedicated to the task that considers contextual information. (iv) A visual explanation of deep learning predictions based on the SHapley Additive exPlanation (SHAP) method adapted to semantic segmentation. The proposed approach yields a detection performance of up to <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>94</mn><mo>%</mo></mrow></semantics></math></inline-formula> of good detection with a false alarm reduction ranging from <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>14</mn><mo>%</mo></mrow></semantics></math></inline-formula> to <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>34</mn><mo>%</mo></mrow></semantics></math></inline-formula> compared to mono-modal models. These results provide new solutions to improve the detection of natural and anthropogenic oil slicks by providing tools that allow photo-interpreters to work more efficiently on a wide range of marine surfaces to be monitored worldwide. Such a tool will accelerate the oil slick detection task to keep up with the continuous sensor acquisition. This upstream work will allow us to study its possible integration into an industrial production pipeline. In addition, a prediction explanation is proposed, which can be integrated as a step to identify the appropriate methodology for presenting the predictions to the experts and understanding the obtained predictions and their sensitivity to contextual information. Thus it helps them to optimize their way of working.https://www.mdpi.com/2072-4292/14/15/3565oil slicksdata fusionoffshore detectionSAR imagesmeteorological datadeep learning |
| spellingShingle | Emna Amri Pierre Dardouillet Alexandre Benoit Hermann Courteille Philippe Bolon Dominique Dubucq Anthony Credoz Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual Data Remote Sensing oil slicks data fusion offshore detection SAR images meteorological data deep learning |
| title | Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual Data |
| title_full | Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual Data |
| title_fullStr | Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual Data |
| title_full_unstemmed | Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual Data |
| title_short | Offshore Oil Slick Detection: From Photo-Interpreter to Explainable Multi-Modal Deep Learning Models Using SAR Images and Contextual Data |
| title_sort | offshore oil slick detection from photo interpreter to explainable multi modal deep learning models using sar images and contextual data |
| topic | oil slicks data fusion offshore detection SAR images meteorological data deep learning |
| url | https://www.mdpi.com/2072-4292/14/15/3565 |
| work_keys_str_mv | AT emnaamri offshoreoilslickdetectionfromphotointerpretertoexplainablemultimodaldeeplearningmodelsusingsarimagesandcontextualdata AT pierredardouillet offshoreoilslickdetectionfromphotointerpretertoexplainablemultimodaldeeplearningmodelsusingsarimagesandcontextualdata AT alexandrebenoit offshoreoilslickdetectionfromphotointerpretertoexplainablemultimodaldeeplearningmodelsusingsarimagesandcontextualdata AT hermanncourteille offshoreoilslickdetectionfromphotointerpretertoexplainablemultimodaldeeplearningmodelsusingsarimagesandcontextualdata AT philippebolon offshoreoilslickdetectionfromphotointerpretertoexplainablemultimodaldeeplearningmodelsusingsarimagesandcontextualdata AT dominiquedubucq offshoreoilslickdetectionfromphotointerpretertoexplainablemultimodaldeeplearningmodelsusingsarimagesandcontextualdata AT anthonycredoz offshoreoilslickdetectionfromphotointerpretertoexplainablemultimodaldeeplearningmodelsusingsarimagesandcontextualdata |