Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge
Polyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, location, and surface largely affect identification, localisation, and characterisation. Moreover, colonoscopic surveillance and removal of polyps (referred to as polypectomy ) are highly operator-...
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Internet publication |
| Language: | English |
| Published: |
2022
|
| _version_ | 1797107978552213504 |
|---|---|
| author | Ali, S Ghatwary, N Jha, D Isik-Polat, E Polat, G Yang, C Li, W Galdran, A Ballester, M-ÁG Thambawita, V Hicks, S Poudel, S Lee, S-W Jin, Z Gan, T Yu, C Yan, J Yeo, D Lee, H Tomar, NK Haithmi, M Ahmed, A Riegler, MA Daul, C Halvorsen, P Rittscher, J Salem, OE Lamarque, D Cannizzaro, R Realdon, S Lange, TD East, JE |
| author_facet | Ali, S Ghatwary, N Jha, D Isik-Polat, E Polat, G Yang, C Li, W Galdran, A Ballester, M-ÁG Thambawita, V Hicks, S Poudel, S Lee, S-W Jin, Z Gan, T Yu, C Yan, J Yeo, D Lee, H Tomar, NK Haithmi, M Ahmed, A Riegler, MA Daul, C Halvorsen, P Rittscher, J Salem, OE Lamarque, D Cannizzaro, R Realdon, S Lange, TD East, JE |
| author_sort | Ali, S |
| collection | OXFORD |
| description | Polyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, location, and surface largely affect identification, localisation, and characterisation. Moreover, colonoscopic surveillance and removal of polyps (referred to as polypectomy ) are highly operator-dependent procedures. There exist a high missed detection rate and incomplete removal of colonic polyps due to their variable nature, the difficulties to delineate the abnormality, the high recurrence rates, and the anatomical topography of the colon. There have been several developments in realising automated methods for both detection and segmentation of these polyps using machine learning. However, the major drawback in most of these methods is their ability to generalise to out-of-sample unseen datasets that come from different centres, modalities and acquisition systems. To test this hypothesis rigorously we curated a multi-centre and multi-population dataset acquired from multiple colonoscopy systems and challenged teams comprising machine learning experts to develop robust automated detection and segmentation methods as part of our crowd-sourcing Endoscopic computer vision challenge (EndoCV) 2021. In this paper, we analyse the detection results of the four top (among seven) teams and the segmentation results of the five top teams (among 16). Our analyses demonstrate that the top-ranking teams concentrated on accuracy (i.e., accuracy > 80% on overall Dice score on different validation sets) over real-time performance required for clinical applicability. We further dissect the methods and provide an experiment-based hypothesis that reveals the need for improved generalisability to tackle diversity present in multi-centre datasets. |
| first_indexed | 2024-03-07T07:21:29Z |
| format | Internet publication |
| id | oxford-uuid:8c969a21-4b82-48a6-a166-2709d8c5c589 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:21:29Z |
| publishDate | 2022 |
| record_format | dspace |
| spelling | oxford-uuid:8c969a21-4b82-48a6-a166-2709d8c5c5892022-10-21T13:40:03ZAssessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challengeInternet publicationhttp://purl.org/coar/resource_type/c_7ad9uuid:8c969a21-4b82-48a6-a166-2709d8c5c589EnglishSymplectic Elements2022Ali, SGhatwary, NJha, DIsik-Polat, EPolat, GYang, CLi, WGaldran, ABallester, M-ÁGThambawita, VHicks, SPoudel, SLee, S-WJin, ZGan, TYu, CYan, JYeo, DLee, HTomar, NKHaithmi, MAhmed, ARiegler, MADaul, CHalvorsen, PRittscher, JSalem, OELamarque, DCannizzaro, RRealdon, SLange, TDEast, JEPolyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, location, and surface largely affect identification, localisation, and characterisation. Moreover, colonoscopic surveillance and removal of polyps (referred to as polypectomy ) are highly operator-dependent procedures. There exist a high missed detection rate and incomplete removal of colonic polyps due to their variable nature, the difficulties to delineate the abnormality, the high recurrence rates, and the anatomical topography of the colon. There have been several developments in realising automated methods for both detection and segmentation of these polyps using machine learning. However, the major drawback in most of these methods is their ability to generalise to out-of-sample unseen datasets that come from different centres, modalities and acquisition systems. To test this hypothesis rigorously we curated a multi-centre and multi-population dataset acquired from multiple colonoscopy systems and challenged teams comprising machine learning experts to develop robust automated detection and segmentation methods as part of our crowd-sourcing Endoscopic computer vision challenge (EndoCV) 2021. In this paper, we analyse the detection results of the four top (among seven) teams and the segmentation results of the five top teams (among 16). Our analyses demonstrate that the top-ranking teams concentrated on accuracy (i.e., accuracy > 80% on overall Dice score on different validation sets) over real-time performance required for clinical applicability. We further dissect the methods and provide an experiment-based hypothesis that reveals the need for improved generalisability to tackle diversity present in multi-centre datasets. |
| spellingShingle | Ali, S Ghatwary, N Jha, D Isik-Polat, E Polat, G Yang, C Li, W Galdran, A Ballester, M-ÁG Thambawita, V Hicks, S Poudel, S Lee, S-W Jin, Z Gan, T Yu, C Yan, J Yeo, D Lee, H Tomar, NK Haithmi, M Ahmed, A Riegler, MA Daul, C Halvorsen, P Rittscher, J Salem, OE Lamarque, D Cannizzaro, R Realdon, S Lange, TD East, JE Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge |
| title | Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge |
| title_full | Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge |
| title_fullStr | Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge |
| title_full_unstemmed | Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge |
| title_short | Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge |
| title_sort | assessing generalisability of deep learning based polyp detection and segmentation methods through a computer vision challenge |
| work_keys_str_mv | AT alis assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT ghatwaryn assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT jhad assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT isikpolate assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT polatg assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT yangc assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT liw assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT galdrana assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT ballestermag assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT thambawitav assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT hickss assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT poudels assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT leesw assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT jinz assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT gant assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT yuc assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT yanj assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT yeod assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT leeh assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT tomarnk assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT haithmim assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT ahmeda assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT rieglerma assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT daulc assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT halvorsenp assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT rittscherj assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT salemoe assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT lamarqued assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT cannizzaror assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT realdons assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT langetd assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge AT eastje assessinggeneralisabilityofdeeplearningbasedpolypdetectionandsegmentationmethodsthroughacomputervisionchallenge |