Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous Chorioretinopathy
BackgroundAlthough previous research has made substantial progress in developing high-performance artificial intelligence (AI)–based computer-aided diagnosis (AI-CAD) systems in various medical domains, little attention has been paid to developing and evaluating AI-CAD system...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-11-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2023/1/e48142 |
| _version_ | 1797448370030116864 |
|---|---|
| author | Jeewoo Yoon Jinyoung Han Junseo Ko Seong Choi Ji In Park Joon Seo Hwang Jeong Mo Han Daniel Duck-Jin Hwang |
| author_facet | Jeewoo Yoon Jinyoung Han Junseo Ko Seong Choi Ji In Park Joon Seo Hwang Jeong Mo Han Daniel Duck-Jin Hwang |
| author_sort | Jeewoo Yoon |
| collection | DOAJ |
| description |
BackgroundAlthough previous research has made substantial progress in developing high-performance artificial intelligence (AI)–based computer-aided diagnosis (AI-CAD) systems in various medical domains, little attention has been paid to developing and evaluating AI-CAD system in ophthalmology, particularly for diagnosing retinal diseases using optical coherence tomography (OCT) images.
ObjectiveThis diagnostic study aimed to determine the usefulness of a proposed AI-CAD system in assisting ophthalmologists with the diagnosis of central serous chorioretinopathy (CSC), which is known to be difficult to diagnose, using OCT images.
MethodsFor the training and evaluation of the proposed deep learning model, 1693 OCT images were collected and annotated. The data set included 929 and 764 cases of acute and chronic CSC, respectively. In total, 66 ophthalmologists (2 groups: 36 retina and 30 nonretina specialists) participated in the observer performance test. To evaluate the deep learning algorithm used in the proposed AI-CAD system, the training, validation, and test sets were split in an 8:1:1 ratio. Further, 100 randomly sampled OCT images from the test set were used for the observer performance test, and the participants were instructed to select a CSC subtype for each of these images. Each image was provided under different conditions: (1) without AI assistance, (2) with AI assistance with a probability score, and (3) with AI assistance with a probability score and visual evidence heatmap. The sensitivity, specificity, and area under the receiver operating characteristic curve were used to measure the diagnostic performance of the model and ophthalmologists.
ResultsThe proposed system achieved a high detection performance (99% of the area under the curve) for CSC, outperforming the 66 ophthalmologists who participated in the observer performance test. In both groups, ophthalmologists with the support of AI assistance with a probability score and visual evidence heatmap achieved the highest mean diagnostic performance compared with that of those subjected to other conditions (without AI assistance or with AI assistance with a probability score). Nonretina specialists achieved expert-level diagnostic performance with the support of the proposed AI-CAD system.
ConclusionsOur proposed AI-CAD system improved the diagnosis of CSC by ophthalmologists, which may support decision-making regarding retinal disease detection and alleviate the workload of ophthalmologists. |
| first_indexed | 2024-03-09T14:10:25Z |
| format | Article |
| id | doaj.art-f9b6cb09418447c49cae0dabc82193e7 |
| institution | Directory Open Access Journal |
| issn | 1438-8871 |
| language | English |
| last_indexed | 2024-03-09T14:10:25Z |
| publishDate | 2023-11-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj.art-f9b6cb09418447c49cae0dabc82193e72023-11-29T15:15:42ZengJMIR PublicationsJournal of Medical Internet Research1438-88712023-11-0125e4814210.2196/48142Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous ChorioretinopathyJeewoo Yoonhttps://orcid.org/0000-0002-9067-8653Jinyoung Hanhttps://orcid.org/0000-0002-8911-2791Junseo Kohttps://orcid.org/0000-0002-4458-3987Seong Choihttps://orcid.org/0000-0003-2721-4706Ji In Parkhttps://orcid.org/0000-0003-4662-3759Joon Seo Hwanghttps://orcid.org/0000-0003-1175-7693Jeong Mo Hanhttps://orcid.org/0000-0002-6379-4536Daniel Duck-Jin Hwanghttps://orcid.org/0000-0003-1808-3169 BackgroundAlthough previous research has made substantial progress in developing high-performance artificial intelligence (AI)–based computer-aided diagnosis (AI-CAD) systems in various medical domains, little attention has been paid to developing and evaluating AI-CAD system in ophthalmology, particularly for diagnosing retinal diseases using optical coherence tomography (OCT) images. ObjectiveThis diagnostic study aimed to determine the usefulness of a proposed AI-CAD system in assisting ophthalmologists with the diagnosis of central serous chorioretinopathy (CSC), which is known to be difficult to diagnose, using OCT images. MethodsFor the training and evaluation of the proposed deep learning model, 1693 OCT images were collected and annotated. The data set included 929 and 764 cases of acute and chronic CSC, respectively. In total, 66 ophthalmologists (2 groups: 36 retina and 30 nonretina specialists) participated in the observer performance test. To evaluate the deep learning algorithm used in the proposed AI-CAD system, the training, validation, and test sets were split in an 8:1:1 ratio. Further, 100 randomly sampled OCT images from the test set were used for the observer performance test, and the participants were instructed to select a CSC subtype for each of these images. Each image was provided under different conditions: (1) without AI assistance, (2) with AI assistance with a probability score, and (3) with AI assistance with a probability score and visual evidence heatmap. The sensitivity, specificity, and area under the receiver operating characteristic curve were used to measure the diagnostic performance of the model and ophthalmologists. ResultsThe proposed system achieved a high detection performance (99% of the area under the curve) for CSC, outperforming the 66 ophthalmologists who participated in the observer performance test. In both groups, ophthalmologists with the support of AI assistance with a probability score and visual evidence heatmap achieved the highest mean diagnostic performance compared with that of those subjected to other conditions (without AI assistance or with AI assistance with a probability score). Nonretina specialists achieved expert-level diagnostic performance with the support of the proposed AI-CAD system. ConclusionsOur proposed AI-CAD system improved the diagnosis of CSC by ophthalmologists, which may support decision-making regarding retinal disease detection and alleviate the workload of ophthalmologists.https://www.jmir.org/2023/1/e48142 |
| spellingShingle | Jeewoo Yoon Jinyoung Han Junseo Ko Seong Choi Ji In Park Joon Seo Hwang Jeong Mo Han Daniel Duck-Jin Hwang Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous Chorioretinopathy Journal of Medical Internet Research |
| title | Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous Chorioretinopathy |
| title_full | Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous Chorioretinopathy |
| title_fullStr | Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous Chorioretinopathy |
| title_full_unstemmed | Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous Chorioretinopathy |
| title_short | Developing and Evaluating an AI-Based Computer-Aided Diagnosis System for Retinal Disease: Diagnostic Study for Central Serous Chorioretinopathy |
| title_sort | developing and evaluating an ai based computer aided diagnosis system for retinal disease diagnostic study for central serous chorioretinopathy |
| url | https://www.jmir.org/2023/1/e48142 |
| work_keys_str_mv | AT jeewooyoon developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy AT jinyounghan developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy AT junseoko developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy AT seongchoi developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy AT jiinpark developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy AT joonseohwang developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy AT jeongmohan developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy AT danielduckjinhwang developingandevaluatinganaibasedcomputeraideddiagnosissystemforretinaldiseasediagnosticstudyforcentralserouschorioretinopathy |