Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class Labels
Purpose: To compare the efficacy and efficiency of training neural networks for medical image classification using comparison labels indicating relative disease severity versus diagnostic class labels from a retinopathy of prematurity (ROP) image dataset. Design: Evaluation of diagnostic test or tec...
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2022-06-01
|
| Series: | Ophthalmology Science |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666914522000112 |
| _version_ | 1811332815549104128 |
|---|---|
| author | Adam Hanif, MD İlkay Yıldız, PhD Peng Tian, PhD Beyza Kalkanlı, BS Deniz Erdoğmuş, PhD Stratis Ioannidis, PhD Jennifer Dy, PhD Jayashree Kalpathy-Cramer, PhD Susan Ostmo, MS Karyn Jonas, BSN R. V. Paul Chan, MD, MBA Michael F. Chiang, MD J. Peter Campbell, MD, MPH |
| author_facet | Adam Hanif, MD İlkay Yıldız, PhD Peng Tian, PhD Beyza Kalkanlı, BS Deniz Erdoğmuş, PhD Stratis Ioannidis, PhD Jennifer Dy, PhD Jayashree Kalpathy-Cramer, PhD Susan Ostmo, MS Karyn Jonas, BSN R. V. Paul Chan, MD, MBA Michael F. Chiang, MD J. Peter Campbell, MD, MPH |
| author_sort | Adam Hanif, MD |
| collection | DOAJ |
| description | Purpose: To compare the efficacy and efficiency of training neural networks for medical image classification using comparison labels indicating relative disease severity versus diagnostic class labels from a retinopathy of prematurity (ROP) image dataset. Design: Evaluation of diagnostic test or technology. Participants: Deep learning neural networks trained on expert-labeled wide-angle retinal images obtained from patients undergoing diagnostic ROP examinations obtained as part of the Imaging and Informatics in ROP (i-ROP) cohort study. Methods: Neural networks were trained with either class or comparison labels indicating plus disease severity in ROP retinal fundus images from 2 datasets. After training and validation, all networks underwent evaluation using a separate test dataset in 1 of 2 binary classification tasks: normal versus abnormal or plus versus nonplus. Main Outcome Measures: Area under the receiver operating characteristic curve (AUC) values were measured to assess network performance. Results: Given the same number of labels, neural networks learned more efficiently by comparison, generating significantly higher AUCs in both classification tasks across both datasets. Similarly, given the same number of images, comparison learning developed networks with significantly higher AUCs across both classification tasks in 1 of 2 datasets. The difference in efficiency and accuracy between models trained on either label type decreased as the size of the training set increased. Conclusions: Comparison labels individually are more informative and more abundant per sample than class labels. These findings indicate a potential means of overcoming the common obstacle of data variability and scarcity when training neural networks for medical image classification tasks. |
| first_indexed | 2024-04-13T16:42:39Z |
| format | Article |
| id | doaj.art-44f5dcfdd4d54e9a881b0d64d2dae15c |
| institution | Directory Open Access Journal |
| issn | 2666-9145 |
| language | English |
| last_indexed | 2024-04-13T16:42:39Z |
| publishDate | 2022-06-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Ophthalmology Science |
| spelling | doaj.art-44f5dcfdd4d54e9a881b0d64d2dae15c2022-12-22T02:39:10ZengElsevierOphthalmology Science2666-91452022-06-0122100122Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class LabelsAdam Hanif, MD0İlkay Yıldız, PhD1Peng Tian, PhD2Beyza Kalkanlı, BS3Deniz Erdoğmuş, PhD4Stratis Ioannidis, PhD5Jennifer Dy, PhD6Jayashree Kalpathy-Cramer, PhD7Susan Ostmo, MS8Karyn Jonas, BSN9R. V. Paul Chan, MD, MBA10Michael F. Chiang, MD11J. Peter Campbell, MD, MPH12Department of Ophthalmology, Oregon Health & Science University, Portland, OregonDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MassachusettsDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MassachusettsDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MassachusettsDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MassachusettsDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MassachusettsDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MassachusettsDepartment of Radiology, Athinoula A. Martinos Center for Biomedical Imaging Clinical Computational Neuroimaging Group, Charlestown, MassachusettsDepartment of Ophthalmology, Oregon Health & Science University, Portland, OregonDepartment of Ophthalmology, University of Illinois at Chicago College of Medicine, Chicago, IllinoisDepartment of Ophthalmology, University of Illinois at Chicago College of Medicine, Chicago, IllinoisNational Eye Institute, National Institutes of Health, Bethesda, MarylandDepartment of Ophthalmology, Oregon Health & Science University, Portland, Oregon; Correspondence: J. Peter Campbell, MD, MPH, Department of Ophthalmology, Oregon Health & Science University, 515 SW Campus Drive, Portland, OR 97239.Purpose: To compare the efficacy and efficiency of training neural networks for medical image classification using comparison labels indicating relative disease severity versus diagnostic class labels from a retinopathy of prematurity (ROP) image dataset. Design: Evaluation of diagnostic test or technology. Participants: Deep learning neural networks trained on expert-labeled wide-angle retinal images obtained from patients undergoing diagnostic ROP examinations obtained as part of the Imaging and Informatics in ROP (i-ROP) cohort study. Methods: Neural networks were trained with either class or comparison labels indicating plus disease severity in ROP retinal fundus images from 2 datasets. After training and validation, all networks underwent evaluation using a separate test dataset in 1 of 2 binary classification tasks: normal versus abnormal or plus versus nonplus. Main Outcome Measures: Area under the receiver operating characteristic curve (AUC) values were measured to assess network performance. Results: Given the same number of labels, neural networks learned more efficiently by comparison, generating significantly higher AUCs in both classification tasks across both datasets. Similarly, given the same number of images, comparison learning developed networks with significantly higher AUCs across both classification tasks in 1 of 2 datasets. The difference in efficiency and accuracy between models trained on either label type decreased as the size of the training set increased. Conclusions: Comparison labels individually are more informative and more abundant per sample than class labels. These findings indicate a potential means of overcoming the common obstacle of data variability and scarcity when training neural networks for medical image classification tasks.http://www.sciencedirect.com/science/article/pii/S2666914522000112Artificial intelligenceDeep learningLabelsNeural networksRetinopathy of prematurity |
| spellingShingle | Adam Hanif, MD İlkay Yıldız, PhD Peng Tian, PhD Beyza Kalkanlı, BS Deniz Erdoğmuş, PhD Stratis Ioannidis, PhD Jennifer Dy, PhD Jayashree Kalpathy-Cramer, PhD Susan Ostmo, MS Karyn Jonas, BSN R. V. Paul Chan, MD, MBA Michael F. Chiang, MD J. Peter Campbell, MD, MPH Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class Labels Ophthalmology Science Artificial intelligence Deep learning Labels Neural networks Retinopathy of prematurity |
| title | Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class Labels |
| title_full | Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class Labels |
| title_fullStr | Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class Labels |
| title_full_unstemmed | Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class Labels |
| title_short | Improved Training Efficiency for Retinopathy of Prematurity Deep Learning Models Using Comparison versus Class Labels |
| title_sort | improved training efficiency for retinopathy of prematurity deep learning models using comparison versus class labels |
| topic | Artificial intelligence Deep learning Labels Neural networks Retinopathy of prematurity |
| url | http://www.sciencedirect.com/science/article/pii/S2666914522000112 |
| work_keys_str_mv | AT adamhanifmd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT ilkayyıldızphd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT pengtianphd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT beyzakalkanlıbs improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT denizerdogmusphd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT stratisioannidisphd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT jenniferdyphd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT jayashreekalpathycramerphd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT susanostmoms improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT karynjonasbsn improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT rvpaulchanmdmba improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT michaelfchiangmd improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels AT jpetercampbellmdmph improvedtrainingefficiencyforretinopathyofprematuritydeeplearningmodelsusingcomparisonversusclasslabels |