Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial
BACKGROUND: The automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes. OBJECTIVE: This stud...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications Inc.
2022
|
| Online Access: | https://hdl.handle.net/1721.1/144059 |
| _version_ | 1826190016845447168 |
|---|---|
| author | Noriega, Alejandro Meizner, Daniela Camacho, Dalia Enciso, Jennifer Quiroz-Mercado, Hugo Morales-Canton, Virgilio Almaatouq, Abdullah Pentland, Alex |
| author2 | Massachusetts Institute of Technology. Media Laboratory |
| author_facet | Massachusetts Institute of Technology. Media Laboratory Noriega, Alejandro Meizner, Daniela Camacho, Dalia Enciso, Jennifer Quiroz-Mercado, Hugo Morales-Canton, Virgilio Almaatouq, Abdullah Pentland, Alex |
| author_sort | Noriega, Alejandro |
| collection | MIT |
| description | BACKGROUND: The automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes. OBJECTIVE: This study aimed to develop and evaluate the performance of an automated deep learning-based system to classify retinal fundus images as referable and nonreferable diabetic retinopathy cases, from international and Mexican patients. In particular, we aimed to evaluate the performance of the automated retina image analysis (ARIA) system under an independent scheme (ie, only ARIA screening) and 2 assistive schemes (ie, hybrid ARIA plus ophthalmologist screening), using a web-based platform for remote image analysis to determine and compare the sensibility and specificity of the 3 schemes. METHODS: A randomized controlled experiment was performed where 17 ophthalmologists were asked to classify a series of retinal fundus images under 3 different conditions. The conditions were to (1) screen the fundus image by themselves (solo); (2) screen the fundus image after exposure to the retina image classification of the ARIA system (ARIA answer); and (3) screen the fundus image after exposure to the classification of the ARIA system, as well as its level of confidence and an attention map highlighting the most important areas of interest in the image according to the ARIA system (ARIA explanation). The ophthalmologists' classification in each condition and the result from the ARIA system were compared against a gold standard generated by consulting and aggregating the opinion of 3 retina specialists for each fundus image. RESULTS: The ARIA system was able to classify referable vs nonreferable cases with an area under the receiver operating characteristic curve of 98%, a sensitivity of 95.1%, and a specificity of 91.5% for international patient cases. There was an area under the receiver operating characteristic curve of 98.3%, a sensitivity of 95.2%, and a specificity of 90% for Mexican patient cases. The ARIA system performance was more successful than the average performance of the 17 ophthalmologists enrolled in the study. Additionally, the results suggest that the ARIA system can be useful as an assistive tool, as sensitivity was significantly higher in the experimental condition where ophthalmologists were exposed to the ARIA system's answer prior to their own classification (93.3%), compared with the sensitivity of the condition where participants assessed the images independently (87.3%; P=.05). CONCLUSIONS: These results demonstrate that both independent and assistive use cases of the ARIA system present, for Latin American countries such as Mexico, a substantial opportunity toward expanding the monitoring capacity for the early detection of diabetes-related blindness. |
| first_indexed | 2024-09-23T08:33:44Z |
| format | Article |
| id | mit-1721.1/144059 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T08:33:44Z |
| publishDate | 2022 |
| publisher | JMIR Publications Inc. |
| record_format | dspace |
| spelling | mit-1721.1/1440592023-04-19T19:13:34Z Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial Noriega, Alejandro Meizner, Daniela Camacho, Dalia Enciso, Jennifer Quiroz-Mercado, Hugo Morales-Canton, Virgilio Almaatouq, Abdullah Pentland, Alex Massachusetts Institute of Technology. Media Laboratory BACKGROUND: The automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes. OBJECTIVE: This study aimed to develop and evaluate the performance of an automated deep learning-based system to classify retinal fundus images as referable and nonreferable diabetic retinopathy cases, from international and Mexican patients. In particular, we aimed to evaluate the performance of the automated retina image analysis (ARIA) system under an independent scheme (ie, only ARIA screening) and 2 assistive schemes (ie, hybrid ARIA plus ophthalmologist screening), using a web-based platform for remote image analysis to determine and compare the sensibility and specificity of the 3 schemes. METHODS: A randomized controlled experiment was performed where 17 ophthalmologists were asked to classify a series of retinal fundus images under 3 different conditions. The conditions were to (1) screen the fundus image by themselves (solo); (2) screen the fundus image after exposure to the retina image classification of the ARIA system (ARIA answer); and (3) screen the fundus image after exposure to the classification of the ARIA system, as well as its level of confidence and an attention map highlighting the most important areas of interest in the image according to the ARIA system (ARIA explanation). The ophthalmologists' classification in each condition and the result from the ARIA system were compared against a gold standard generated by consulting and aggregating the opinion of 3 retina specialists for each fundus image. RESULTS: The ARIA system was able to classify referable vs nonreferable cases with an area under the receiver operating characteristic curve of 98%, a sensitivity of 95.1%, and a specificity of 91.5% for international patient cases. There was an area under the receiver operating characteristic curve of 98.3%, a sensitivity of 95.2%, and a specificity of 90% for Mexican patient cases. The ARIA system performance was more successful than the average performance of the 17 ophthalmologists enrolled in the study. Additionally, the results suggest that the ARIA system can be useful as an assistive tool, as sensitivity was significantly higher in the experimental condition where ophthalmologists were exposed to the ARIA system's answer prior to their own classification (93.3%), compared with the sensitivity of the condition where participants assessed the images independently (87.3%; P=.05). CONCLUSIONS: These results demonstrate that both independent and assistive use cases of the ARIA system present, for Latin American countries such as Mexico, a substantial opportunity toward expanding the monitoring capacity for the early detection of diabetes-related blindness. 2022-07-26T18:25:50Z 2022-07-26T18:25:50Z 2021 2022-07-26T18:16:41Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/144059 Noriega, Alejandro, Meizner, Daniela, Camacho, Dalia, Enciso, Jennifer, Quiroz-Mercado, Hugo et al. 2021. "Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial." JMIR Formative Research, 5 (8). en 10.2196/25290 JMIR Formative Research Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf JMIR Publications Inc. JMIR Publications |
| spellingShingle | Noriega, Alejandro Meizner, Daniela Camacho, Dalia Enciso, Jennifer Quiroz-Mercado, Hugo Morales-Canton, Virgilio Almaatouq, Abdullah Pentland, Alex Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_full | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_fullStr | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_full_unstemmed | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_short | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_sort | screening diabetic retinopathy using an automated retinal image analysis system in independent and assistive use cases in mexico randomized controlled trial |
| url | https://hdl.handle.net/1721.1/144059 |
| work_keys_str_mv | AT noriegaalejandro screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT meiznerdaniela screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT camachodalia screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT encisojennifer screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT quirozmercadohugo screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT moralescantonvirgilio screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT almaatouqabdullah screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT pentlandalex screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial |