Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional Data
The diagnosis of severe COVID-19 lung infection is important because it carries a higher risk for the patient and requires prompt treatment with oxygen therapy and hospitalization while those with less severe lung infection often stay on observation. Also, severe infections are more likely to have l...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-02-01
|
| Series: | Diagnostics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2075-4418/14/3/341 |
| _version_ | 1797318850586345472 |
|---|---|
| author | André Sobiecki Lubomir M. Hadjiiski Heang-Ping Chan Ravi K. Samala Chuan Zhou Jadranka Stojanovska Prachi P. Agarwal |
| author_facet | André Sobiecki Lubomir M. Hadjiiski Heang-Ping Chan Ravi K. Samala Chuan Zhou Jadranka Stojanovska Prachi P. Agarwal |
| author_sort | André Sobiecki |
| collection | DOAJ |
| description | The diagnosis of severe COVID-19 lung infection is important because it carries a higher risk for the patient and requires prompt treatment with oxygen therapy and hospitalization while those with less severe lung infection often stay on observation. Also, severe infections are more likely to have long-standing residual changes in their lungs and may need follow-up imaging. We have developed deep learning neural network models for classifying severe vs. non-severe lung infections in COVID-19 patients on chest radiographs (CXR). A deep learning U-Net model was developed to segment the lungs. Inception-v1 and Inception-v4 models were trained for the classification of severe vs. non-severe COVID-19 infection. Four CXR datasets from multi-country and multi-institutional sources were used to develop and evaluate the models. The combined dataset consisted of 5748 cases and 6193 CXR images with physicians’ severity ratings as reference standard. The area under the receiver operating characteristic curve (AUC) was used to evaluate model performance. We studied the reproducibility of classification performance using the different combinations of training and validation data sets. We also evaluated the generalizability of the trained deep learning models using both independent internal and external test sets. The Inception-v1 based models achieved AUC ranging between 0.81 ± 0.02 and 0.84 ± 0.0, while the Inception-v4 models achieved AUC in the range of 0.85 ± 0.06 and 0.89 ± 0.01, on the independent test sets, respectively. These results demonstrate the promise of using deep learning models in differentiating COVID-19 patients with severe from non-severe lung infection on chest radiographs. |
| first_indexed | 2024-03-08T03:58:22Z |
| format | Article |
| id | doaj.art-22937aa61e4e47deadc1aee2e96c563f |
| institution | Directory Open Access Journal |
| issn | 2075-4418 |
| language | English |
| last_indexed | 2024-03-08T03:58:22Z |
| publishDate | 2024-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Diagnostics |
| spelling | doaj.art-22937aa61e4e47deadc1aee2e96c563f2024-02-09T15:10:17ZengMDPI AGDiagnostics2075-44182024-02-0114334110.3390/diagnostics14030341Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional DataAndré Sobiecki0Lubomir M. Hadjiiski1Heang-Ping Chan2Ravi K. Samala3Chuan Zhou4Jadranka Stojanovska5Prachi P. Agarwal6Department of Radiology, University of Michigan, Ann Arbor, MI 48109, USADepartment of Radiology, University of Michigan, Ann Arbor, MI 48109, USADepartment of Radiology, University of Michigan, Ann Arbor, MI 48109, USAOffice of Science and Engineering Laboratories, Center for Devices and Radiological Health, U.S. Food and Drug Administration, Silver Spring, MD 20993, USADepartment of Radiology, University of Michigan, Ann Arbor, MI 48109, USADepartment of Radiology, New York University, New York, NY 10016, USADepartment of Radiology, University of Michigan, Ann Arbor, MI 48109, USAThe diagnosis of severe COVID-19 lung infection is important because it carries a higher risk for the patient and requires prompt treatment with oxygen therapy and hospitalization while those with less severe lung infection often stay on observation. Also, severe infections are more likely to have long-standing residual changes in their lungs and may need follow-up imaging. We have developed deep learning neural network models for classifying severe vs. non-severe lung infections in COVID-19 patients on chest radiographs (CXR). A deep learning U-Net model was developed to segment the lungs. Inception-v1 and Inception-v4 models were trained for the classification of severe vs. non-severe COVID-19 infection. Four CXR datasets from multi-country and multi-institutional sources were used to develop and evaluate the models. The combined dataset consisted of 5748 cases and 6193 CXR images with physicians’ severity ratings as reference standard. The area under the receiver operating characteristic curve (AUC) was used to evaluate model performance. We studied the reproducibility of classification performance using the different combinations of training and validation data sets. We also evaluated the generalizability of the trained deep learning models using both independent internal and external test sets. The Inception-v1 based models achieved AUC ranging between 0.81 ± 0.02 and 0.84 ± 0.0, while the Inception-v4 models achieved AUC in the range of 0.85 ± 0.06 and 0.89 ± 0.01, on the independent test sets, respectively. These results demonstrate the promise of using deep learning models in differentiating COVID-19 patients with severe from non-severe lung infection on chest radiographs.https://www.mdpi.com/2075-4418/14/3/341severe lung infectionCOVID-19deep learningdiagnosisclassification |
| spellingShingle | André Sobiecki Lubomir M. Hadjiiski Heang-Ping Chan Ravi K. Samala Chuan Zhou Jadranka Stojanovska Prachi P. Agarwal Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional Data Diagnostics severe lung infection COVID-19 deep learning diagnosis classification |
| title | Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional Data |
| title_full | Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional Data |
| title_fullStr | Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional Data |
| title_full_unstemmed | Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional Data |
| title_short | Detection of Severe Lung Infection on Chest Radiographs of COVID-19 Patients: Robustness of AI Models across Multi-Institutional Data |
| title_sort | detection of severe lung infection on chest radiographs of covid 19 patients robustness of ai models across multi institutional data |
| topic | severe lung infection COVID-19 deep learning diagnosis classification |
| url | https://www.mdpi.com/2075-4418/14/3/341 |
| work_keys_str_mv | AT andresobiecki detectionofseverelunginfectiononchestradiographsofcovid19patientsrobustnessofaimodelsacrossmultiinstitutionaldata AT lubomirmhadjiiski detectionofseverelunginfectiononchestradiographsofcovid19patientsrobustnessofaimodelsacrossmultiinstitutionaldata AT heangpingchan detectionofseverelunginfectiononchestradiographsofcovid19patientsrobustnessofaimodelsacrossmultiinstitutionaldata AT raviksamala detectionofseverelunginfectiononchestradiographsofcovid19patientsrobustnessofaimodelsacrossmultiinstitutionaldata AT chuanzhou detectionofseverelunginfectiononchestradiographsofcovid19patientsrobustnessofaimodelsacrossmultiinstitutionaldata AT jadrankastojanovska detectionofseverelunginfectiononchestradiographsofcovid19patientsrobustnessofaimodelsacrossmultiinstitutionaldata AT prachipagarwal detectionofseverelunginfectiononchestradiographsofcovid19patientsrobustnessofaimodelsacrossmultiinstitutionaldata |