Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings
Abstract Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorith...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-03-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-024-55792-1 |
| _version_ | 1827327740763701248 |
|---|---|
| author | Queralt Miró Catalina Josep Vidal-Alaball Aïna Fuster-Casanovas Anna Escalé-Besa Anna Ruiz Comellas Jordi Solé-Casals |
| author_facet | Queralt Miró Catalina Josep Vidal-Alaball Aïna Fuster-Casanovas Anna Escalé-Besa Anna Ruiz Comellas Jordi Solé-Casals |
| author_sort | Queralt Miró Catalina |
| collection | DOAJ |
| description | Abstract Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorithm’s diagnoses in real clinical practice, comparing them to a radiologist’s diagnoses. The aim is also to identify diagnoses the algorithm may not have been trained for. A prospective observational study for the external validation of the AI algorithm in a region of Catalonia, comparing the AI algorithm’s diagnosis with that of the reference radiologist, considered the gold standard. The external validation was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist's report. Analysing the validity of the AI algorithm, the average accuracy was 0.95 (95% CI 0.92; 0.98), the sensitivity was 0.48 (95% CI 0.30; 0.66) and the specificity was 0.98 (95% CI 0.97; 0.99). The conditions where the algorithm was most sensitive were external, upper abdominal and cardiac and/or valvular implants. On the other hand, the conditions where the algorithm was less sensitive were in the mediastinum, vessels and bone. The algorithm has been validated in the primary care setting and has proven to be useful when identifying images with or without conditions. However, in order to be a valuable tool to help and support experts, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analysed. Our study emphasizes the need for continuous improvement to ensure the algorithm’s effectiveness in primary care. |
| first_indexed | 2024-03-07T15:06:41Z |
| format | Article |
| id | doaj.art-e5096b10326f4cfe82990142e7dbd71a |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-03-07T15:06:41Z |
| publishDate | 2024-03-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-e5096b10326f4cfe82990142e7dbd71a2024-03-05T18:53:51ZengNature PortfolioScientific Reports2045-23222024-03-0114111110.1038/s41598-024-55792-1Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settingsQueralt Miró Catalina0Josep Vidal-Alaball1Aïna Fuster-Casanovas2Anna Escalé-Besa3Anna Ruiz Comellas4Jordi Solé-Casals5Unitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut Jordi Gol i GurinaUnitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut Jordi Gol i GurinaUnitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut Jordi Gol i GurinaUnitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut Jordi Gol i GurinaUnitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut Jordi Gol i GurinaData and Signal Processing Group, Faculty of Science, Technology and Engineering, University of Vic-Central University of CataloniaAbstract Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorithm’s diagnoses in real clinical practice, comparing them to a radiologist’s diagnoses. The aim is also to identify diagnoses the algorithm may not have been trained for. A prospective observational study for the external validation of the AI algorithm in a region of Catalonia, comparing the AI algorithm’s diagnosis with that of the reference radiologist, considered the gold standard. The external validation was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist's report. Analysing the validity of the AI algorithm, the average accuracy was 0.95 (95% CI 0.92; 0.98), the sensitivity was 0.48 (95% CI 0.30; 0.66) and the specificity was 0.98 (95% CI 0.97; 0.99). The conditions where the algorithm was most sensitive were external, upper abdominal and cardiac and/or valvular implants. On the other hand, the conditions where the algorithm was less sensitive were in the mediastinum, vessels and bone. The algorithm has been validated in the primary care setting and has proven to be useful when identifying images with or without conditions. However, in order to be a valuable tool to help and support experts, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analysed. Our study emphasizes the need for continuous improvement to ensure the algorithm’s effectiveness in primary care.https://doi.org/10.1038/s41598-024-55792-1 |
| spellingShingle | Queralt Miró Catalina Josep Vidal-Alaball Aïna Fuster-Casanovas Anna Escalé-Besa Anna Ruiz Comellas Jordi Solé-Casals Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings Scientific Reports |
| title | Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings |
| title_full | Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings |
| title_fullStr | Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings |
| title_full_unstemmed | Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings |
| title_short | Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings |
| title_sort | real world testing of an artificial intelligence algorithm for the analysis of chest x rays in primary care settings |
| url | https://doi.org/10.1038/s41598-024-55792-1 |
| work_keys_str_mv | AT queraltmirocatalina realworldtestingofanartificialintelligencealgorithmfortheanalysisofchestxraysinprimarycaresettings AT josepvidalalaball realworldtestingofanartificialintelligencealgorithmfortheanalysisofchestxraysinprimarycaresettings AT ainafustercasanovas realworldtestingofanartificialintelligencealgorithmfortheanalysisofchestxraysinprimarycaresettings AT annaescalebesa realworldtestingofanartificialintelligencealgorithmfortheanalysisofchestxraysinprimarycaresettings AT annaruizcomellas realworldtestingofanartificialintelligencealgorithmfortheanalysisofchestxraysinprimarycaresettings AT jordisolecasals realworldtestingofanartificialintelligencealgorithmfortheanalysisofchestxraysinprimarycaresettings |