Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation
Deep learning models perform unreliably when the data come from a distribution different from the training one. In critical applications such as medical imaging, out-of-distribution (OOD) detection methods help to identify such data samples, preventing erroneous predictions. In this paper, we furthe...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Journal of Imaging |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2313-433X/9/9/191 |
| _version_ | 1797579407730147328 |
|---|---|
| author | Anton Vasiliuk Daria Frolova Mikhail Belyaev Boris Shirokikh |
| author_facet | Anton Vasiliuk Daria Frolova Mikhail Belyaev Boris Shirokikh |
| author_sort | Anton Vasiliuk |
| collection | DOAJ |
| description | Deep learning models perform unreliably when the data come from a distribution different from the training one. In critical applications such as medical imaging, out-of-distribution (OOD) detection methods help to identify such data samples, preventing erroneous predictions. In this paper, we further investigate OOD detection effectiveness when applied to 3D medical image segmentation. We designed several OOD challenges representing clinically occurring cases and found that none of the methods achieved acceptable performance. Methods not dedicated to segmentation severely failed to perform in the designed setups; the best mean false-positive rate at a 95% true-positive rate (FPR) was 0.59. Segmentation-dedicated methods still achieved suboptimal performance, with the best mean FPR being 0.31 (lower is better). To indicate this suboptimality, we developed a simple method called Intensity Histogram Features (IHF), which performed comparably or better in the same challenges, with a mean FPR of 0.25. Our findings highlight the limitations of the existing OOD detection methods with 3D medical images and present a promising avenue for improving them. To facilitate research in this area, we release the designed challenges as a publicly available benchmark and formulate practical criteria to test the generalization of OOD detection beyond the suggested benchmark. We also propose IHF as a solid baseline to contest emerging methods. |
| first_indexed | 2024-03-10T22:36:43Z |
| format | Article |
| id | doaj.art-e7f7874cf27040249260488c7964cdd8 |
| institution | Directory Open Access Journal |
| issn | 2313-433X |
| language | English |
| last_indexed | 2024-03-10T22:36:43Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Journal of Imaging |
| spelling | doaj.art-e7f7874cf27040249260488c7964cdd82023-11-19T11:24:50ZengMDPI AGJournal of Imaging2313-433X2023-09-019919110.3390/jimaging9090191Limitations of Out-of-Distribution Detection in 3D Medical Image SegmentationAnton Vasiliuk0Daria Frolova1Mikhail Belyaev2Boris Shirokikh3Moscow Institute of Physics and Technology, Moscow 141701, RussiaArtificial Intelligence Research Institute (AIRI), Moscow 105064, RussiaArtificial Intelligence Research Institute (AIRI), Moscow 105064, RussiaArtificial Intelligence Research Institute (AIRI), Moscow 105064, RussiaDeep learning models perform unreliably when the data come from a distribution different from the training one. In critical applications such as medical imaging, out-of-distribution (OOD) detection methods help to identify such data samples, preventing erroneous predictions. In this paper, we further investigate OOD detection effectiveness when applied to 3D medical image segmentation. We designed several OOD challenges representing clinically occurring cases and found that none of the methods achieved acceptable performance. Methods not dedicated to segmentation severely failed to perform in the designed setups; the best mean false-positive rate at a 95% true-positive rate (FPR) was 0.59. Segmentation-dedicated methods still achieved suboptimal performance, with the best mean FPR being 0.31 (lower is better). To indicate this suboptimality, we developed a simple method called Intensity Histogram Features (IHF), which performed comparably or better in the same challenges, with a mean FPR of 0.25. Our findings highlight the limitations of the existing OOD detection methods with 3D medical images and present a promising avenue for improving them. To facilitate research in this area, we release the designed challenges as a publicly available benchmark and formulate practical criteria to test the generalization of OOD detection beyond the suggested benchmark. We also propose IHF as a solid baseline to contest emerging methods.https://www.mdpi.com/2313-433X/9/9/191computed tomographymagnetic resonance imagingout-of-distribution detectionanomaly detectionsegmentation |
| spellingShingle | Anton Vasiliuk Daria Frolova Mikhail Belyaev Boris Shirokikh Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation Journal of Imaging computed tomography magnetic resonance imaging out-of-distribution detection anomaly detection segmentation |
| title | Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation |
| title_full | Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation |
| title_fullStr | Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation |
| title_full_unstemmed | Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation |
| title_short | Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation |
| title_sort | limitations of out of distribution detection in 3d medical image segmentation |
| topic | computed tomography magnetic resonance imaging out-of-distribution detection anomaly detection segmentation |
| url | https://www.mdpi.com/2313-433X/9/9/191 |
| work_keys_str_mv | AT antonvasiliuk limitationsofoutofdistributiondetectionin3dmedicalimagesegmentation AT dariafrolova limitationsofoutofdistributiondetectionin3dmedicalimagesegmentation AT mikhailbelyaev limitationsofoutofdistributiondetectionin3dmedicalimagesegmentation AT borisshirokikh limitationsofoutofdistributiondetectionin3dmedicalimagesegmentation |