The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet search
Abstract Model Dermatology ( https://modelderm.com ; Build2021) is a publicly testable neural network that can classify 184 skin disorders. We aimed to investigate whether our algorithm can classify clinical images of an Internet community along with tertiary care center datasets. Consecutive images...
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-09-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-022-20632-7 |
| _version_ | 1828380836315004928 |
|---|---|
| author | Seung Seog Han Cristian Navarrete-Dechent Konstantinos Liopyris Myoung Shin Kim Gyeong Hun Park Sang Seok Woo Juhyun Park Jung Won Shin Bo Ri Kim Min Jae Kim Francisca Donoso Francisco Villanueva Cristian Ramirez Sung Eun Chang Allan Halpern Seong Hwan Kim Jung-Im Na |
| author_facet | Seung Seog Han Cristian Navarrete-Dechent Konstantinos Liopyris Myoung Shin Kim Gyeong Hun Park Sang Seok Woo Juhyun Park Jung Won Shin Bo Ri Kim Min Jae Kim Francisca Donoso Francisco Villanueva Cristian Ramirez Sung Eun Chang Allan Halpern Seong Hwan Kim Jung-Im Na |
| author_sort | Seung Seog Han |
| collection | DOAJ |
| description | Abstract Model Dermatology ( https://modelderm.com ; Build2021) is a publicly testable neural network that can classify 184 skin disorders. We aimed to investigate whether our algorithm can classify clinical images of an Internet community along with tertiary care center datasets. Consecutive images from an Internet skin cancer community (‘RD’ dataset, 1,282 images posted between 25 January 2020 to 30 July 2021; https://reddit.com/r/melanoma ) were analyzed retrospectively, along with hospital datasets (Edinburgh dataset, 1,300 images; SNU dataset, 2,101 images; TeleDerm dataset, 340 consecutive images). The algorithm’s performance was equivalent to that of dermatologists in the curated clinical datasets (Edinburgh and SNU datasets). However, its performance deteriorated in the RD and TeleDerm datasets because of insufficient image quality and the presence of out-of-distribution disorders, respectively. For the RD dataset, the algorithm’s Top-1/3 accuracy (39.2%/67.2%) and AUC (0.800) were equivalent to that of general physicians (36.8%/52.9%). It was more accurate than that of the laypersons using random Internet searches (19.2%/24.4%). The Top-1/3 accuracy was affected by inadequate image quality (adequate = 43.2%/71.3% versus inadequate = 32.9%/60.8%), whereas participant performance did not deteriorate (adequate = 35.8%/52.7% vs. inadequate = 38.4%/53.3%). In this report, the algorithm performance was significantly affected by the change of the intended settings, which implies that AI algorithms at dermatologist-level, in-distribution setting, may not be able to show the same level of performance in with out-of-distribution settings. |
| first_indexed | 2024-12-10T04:05:05Z |
| format | Article |
| id | doaj.art-805954cc2e8b49f6bedca86afd6fa9fe |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-12-10T04:05:05Z |
| publishDate | 2022-09-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-805954cc2e8b49f6bedca86afd6fa9fe2022-12-22T02:02:52ZengNature PortfolioScientific Reports2045-23222022-09-011211910.1038/s41598-022-20632-7The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet searchSeung Seog Han0Cristian Navarrete-Dechent1Konstantinos Liopyris2Myoung Shin Kim3Gyeong Hun Park4Sang Seok Woo5Juhyun Park6Jung Won Shin7Bo Ri Kim8Min Jae Kim9Francisca Donoso10Francisco Villanueva11Cristian Ramirez12Sung Eun Chang13Allan Halpern14Seong Hwan Kim15Jung-Im Na16Department of Dermatology, I Dermatology ClinicDepartment of Dermatology, School of Medicine, Pontificia Universidad Católica de ChileDepartment of Dermatology, University of Athens, Andreas Syggros Hospital of Skin and Venereal DiseasesDepartment of Dermatology, Sanggye Paik Hospital, Inje University College of MedicineDepartment of Dermatology, Dongtan Sacred Heart Hospital, Hallym University College of MedicineDepartment of Plastic and Reconstructive Surgery, Kangnam Sacred Heart Hospital, Hallym University College of MedicineDepartment of Dermatology, Seoul National University Bundang HospitalDepartment of Dermatology, Seoul National University Bundang HospitalDepartment of Dermatology, Seoul National University Bundang HospitalDepartment of Dermatology, Seoul National University Bundang HospitalDepartment of Dermatology, School of Medicine, Pontificia Universidad Católica de ChileDepartment of Dermatology, School of Medicine, Pontificia Universidad Católica de ChileDepartment of Dermatology, School of Medicine, Pontificia Universidad Católica de ChileDepartment of Dermatology, Asan Medical Center, Ulsan University College of MedicineDermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer CenterDepartment of Plastic and Reconstructive Surgery, Kangnam Sacred Heart Hospital, Hallym University College of MedicineDepartment of Dermatology, Seoul National University Bundang HospitalAbstract Model Dermatology ( https://modelderm.com ; Build2021) is a publicly testable neural network that can classify 184 skin disorders. We aimed to investigate whether our algorithm can classify clinical images of an Internet community along with tertiary care center datasets. Consecutive images from an Internet skin cancer community (‘RD’ dataset, 1,282 images posted between 25 January 2020 to 30 July 2021; https://reddit.com/r/melanoma ) were analyzed retrospectively, along with hospital datasets (Edinburgh dataset, 1,300 images; SNU dataset, 2,101 images; TeleDerm dataset, 340 consecutive images). The algorithm’s performance was equivalent to that of dermatologists in the curated clinical datasets (Edinburgh and SNU datasets). However, its performance deteriorated in the RD and TeleDerm datasets because of insufficient image quality and the presence of out-of-distribution disorders, respectively. For the RD dataset, the algorithm’s Top-1/3 accuracy (39.2%/67.2%) and AUC (0.800) were equivalent to that of general physicians (36.8%/52.9%). It was more accurate than that of the laypersons using random Internet searches (19.2%/24.4%). The Top-1/3 accuracy was affected by inadequate image quality (adequate = 43.2%/71.3% versus inadequate = 32.9%/60.8%), whereas participant performance did not deteriorate (adequate = 35.8%/52.7% vs. inadequate = 38.4%/53.3%). In this report, the algorithm performance was significantly affected by the change of the intended settings, which implies that AI algorithms at dermatologist-level, in-distribution setting, may not be able to show the same level of performance in with out-of-distribution settings.https://doi.org/10.1038/s41598-022-20632-7 |
| spellingShingle | Seung Seog Han Cristian Navarrete-Dechent Konstantinos Liopyris Myoung Shin Kim Gyeong Hun Park Sang Seok Woo Juhyun Park Jung Won Shin Bo Ri Kim Min Jae Kim Francisca Donoso Francisco Villanueva Cristian Ramirez Sung Eun Chang Allan Halpern Seong Hwan Kim Jung-Im Na The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet search Scientific Reports |
| title | The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet search |
| title_full | The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet search |
| title_fullStr | The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet search |
| title_full_unstemmed | The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet search |
| title_short | The degradation of performance of a state-of-the-art skin image classifier when applied to patient-driven internet search |
| title_sort | degradation of performance of a state of the art skin image classifier when applied to patient driven internet search |
| url | https://doi.org/10.1038/s41598-022-20632-7 |
| work_keys_str_mv | AT seungseoghan thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT cristiannavarretedechent thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT konstantinosliopyris thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT myoungshinkim thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT gyeonghunpark thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT sangseokwoo thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT juhyunpark thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT jungwonshin thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT borikim thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT minjaekim thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT franciscadonoso thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT franciscovillanueva thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT cristianramirez thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT sungeunchang thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT allanhalpern thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT seonghwankim thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT jungimna thedegradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT seungseoghan degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT cristiannavarretedechent degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT konstantinosliopyris degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT myoungshinkim degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT gyeonghunpark degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT sangseokwoo degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT juhyunpark degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT jungwonshin degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT borikim degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT minjaekim degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT franciscadonoso degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT franciscovillanueva degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT cristianramirez degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT sungeunchang degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT allanhalpern degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT seonghwankim degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch AT jungimna degradationofperformanceofastateoftheartskinimageclassifierwhenappliedtopatientdriveninternetsearch |