Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial
BackgroundThe automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2021-08-01
|
| Series: | JMIR Formative Research |
| Online Access: | https://formative.jmir.org/2021/8/e25290 |
| _version_ | 1797735785661726720 |
|---|---|
| author | Alejandro Noriega Daniela Meizner Dalia Camacho Jennifer Enciso Hugo Quiroz-Mercado Virgilio Morales-Canton Abdullah Almaatouq Alex Pentland |
| author_facet | Alejandro Noriega Daniela Meizner Dalia Camacho Jennifer Enciso Hugo Quiroz-Mercado Virgilio Morales-Canton Abdullah Almaatouq Alex Pentland |
| author_sort | Alejandro Noriega |
| collection | DOAJ |
| description |
BackgroundThe automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes.
ObjectiveThis study aimed to develop and evaluate the performance of an automated deep learning–based system to classify retinal fundus images as referable and nonreferable diabetic retinopathy cases, from international and Mexican patients. In particular, we aimed to evaluate the performance of the automated retina image analysis (ARIA) system under an independent scheme (ie, only ARIA screening) and 2 assistive schemes (ie, hybrid ARIA plus ophthalmologist screening), using a web-based platform for remote image analysis to determine and compare the sensibility and specificity of the 3 schemes.
MethodsA randomized controlled experiment was performed where 17 ophthalmologists were asked to classify a series of retinal fundus images under 3 different conditions. The conditions were to (1) screen the fundus image by themselves (solo); (2) screen the fundus image after exposure to the retina image classification of the ARIA system (ARIA answer); and (3) screen the fundus image after exposure to the classification of the ARIA system, as well as its level of confidence and an attention map highlighting the most important areas of interest in the image according to the ARIA system (ARIA explanation). The ophthalmologists’ classification in each condition and the result from the ARIA system were compared against a gold standard generated by consulting and aggregating the opinion of 3 retina specialists for each fundus image.
ResultsThe ARIA system was able to classify referable vs nonreferable cases with an area under the receiver operating characteristic curve of 98%, a sensitivity of 95.1%, and a specificity of 91.5% for international patient cases. There was an area under the receiver operating characteristic curve of 98.3%, a sensitivity of 95.2%, and a specificity of 90% for Mexican patient cases. The ARIA system performance was more successful than the average performance of the 17 ophthalmologists enrolled in the study. Additionally, the results suggest that the ARIA system can be useful as an assistive tool, as sensitivity was significantly higher in the experimental condition where ophthalmologists were exposed to the ARIA system’s answer prior to their own classification (93.3%), compared with the sensitivity of the condition where participants assessed the images independently (87.3%; P=.05).
ConclusionsThese results demonstrate that both independent and assistive use cases of the ARIA system present, for Latin American countries such as Mexico, a substantial opportunity toward expanding the monitoring capacity for the early detection of diabetes-related blindness. |
| first_indexed | 2024-03-12T13:03:09Z |
| format | Article |
| id | doaj.art-299b5e99b74947098338d9663a270ed9 |
| institution | Directory Open Access Journal |
| issn | 2561-326X |
| language | English |
| last_indexed | 2024-03-12T13:03:09Z |
| publishDate | 2021-08-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Formative Research |
| spelling | doaj.art-299b5e99b74947098338d9663a270ed92023-08-28T18:33:32ZengJMIR PublicationsJMIR Formative Research2561-326X2021-08-0158e2529010.2196/25290Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled TrialAlejandro Noriegahttps://orcid.org/0000-0003-0106-392XDaniela Meiznerhttps://orcid.org/0000-0002-1500-8887Dalia Camachohttps://orcid.org/0000-0001-7493-7507Jennifer Encisohttps://orcid.org/0000-0003-1289-9730Hugo Quiroz-Mercadohttps://orcid.org/0000-0003-0063-9552Virgilio Morales-Cantonhttps://orcid.org/0000-0002-0454-3841Abdullah Almaatouqhttps://orcid.org/0000-0002-8467-9123Alex Pentlandhttps://orcid.org/0000-0002-8053-9983 BackgroundThe automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes. ObjectiveThis study aimed to develop and evaluate the performance of an automated deep learning–based system to classify retinal fundus images as referable and nonreferable diabetic retinopathy cases, from international and Mexican patients. In particular, we aimed to evaluate the performance of the automated retina image analysis (ARIA) system under an independent scheme (ie, only ARIA screening) and 2 assistive schemes (ie, hybrid ARIA plus ophthalmologist screening), using a web-based platform for remote image analysis to determine and compare the sensibility and specificity of the 3 schemes. MethodsA randomized controlled experiment was performed where 17 ophthalmologists were asked to classify a series of retinal fundus images under 3 different conditions. The conditions were to (1) screen the fundus image by themselves (solo); (2) screen the fundus image after exposure to the retina image classification of the ARIA system (ARIA answer); and (3) screen the fundus image after exposure to the classification of the ARIA system, as well as its level of confidence and an attention map highlighting the most important areas of interest in the image according to the ARIA system (ARIA explanation). The ophthalmologists’ classification in each condition and the result from the ARIA system were compared against a gold standard generated by consulting and aggregating the opinion of 3 retina specialists for each fundus image. ResultsThe ARIA system was able to classify referable vs nonreferable cases with an area under the receiver operating characteristic curve of 98%, a sensitivity of 95.1%, and a specificity of 91.5% for international patient cases. There was an area under the receiver operating characteristic curve of 98.3%, a sensitivity of 95.2%, and a specificity of 90% for Mexican patient cases. The ARIA system performance was more successful than the average performance of the 17 ophthalmologists enrolled in the study. Additionally, the results suggest that the ARIA system can be useful as an assistive tool, as sensitivity was significantly higher in the experimental condition where ophthalmologists were exposed to the ARIA system’s answer prior to their own classification (93.3%), compared with the sensitivity of the condition where participants assessed the images independently (87.3%; P=.05). ConclusionsThese results demonstrate that both independent and assistive use cases of the ARIA system present, for Latin American countries such as Mexico, a substantial opportunity toward expanding the monitoring capacity for the early detection of diabetes-related blindness.https://formative.jmir.org/2021/8/e25290 |
| spellingShingle | Alejandro Noriega Daniela Meizner Dalia Camacho Jennifer Enciso Hugo Quiroz-Mercado Virgilio Morales-Canton Abdullah Almaatouq Alex Pentland Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial JMIR Formative Research |
| title | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_full | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_fullStr | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_full_unstemmed | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_short | Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial |
| title_sort | screening diabetic retinopathy using an automated retinal image analysis system in independent and assistive use cases in mexico randomized controlled trial |
| url | https://formative.jmir.org/2021/8/e25290 |
| work_keys_str_mv | AT alejandronoriega screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT danielameizner screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT daliacamacho screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT jenniferenciso screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT hugoquirozmercado screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT virgiliomoralescanton screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT abdullahalmaatouq screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial AT alexpentland screeningdiabeticretinopathyusinganautomatedretinalimageanalysissysteminindependentandassistiveusecasesinmexicorandomizedcontrolledtrial |