Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays
Classification of drug-resistant tuberculosis (DR-TB) and drug-sensitive tuberculosis (DS-TB) from chest radiographs remains an open problem. Our previous cross validation performance on publicly available chest X-ray (CXR) data combined with image augmentation, the addition of synthetically generat...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-01-01
|
| Series: | Diagnostics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2075-4418/12/1/188 |
| _version_ | 1797494674356699136 |
|---|---|
| author | Manohar Karki Karthik Kantipudi Feng Yang Hang Yu Yi Xiang J. Wang Ziv Yaniv Stefan Jaeger |
| author_facet | Manohar Karki Karthik Kantipudi Feng Yang Hang Yu Yi Xiang J. Wang Ziv Yaniv Stefan Jaeger |
| author_sort | Manohar Karki |
| collection | DOAJ |
| description | Classification of drug-resistant tuberculosis (DR-TB) and drug-sensitive tuberculosis (DS-TB) from chest radiographs remains an open problem. Our previous cross validation performance on publicly available chest X-ray (CXR) data combined with image augmentation, the addition of synthetically generated and publicly available images achieved a performance of 85% AUC with a deep convolutional neural network (CNN). However, when we evaluated the CNN model trained to classify DR-TB and DS-TB on unseen data, significant performance degradation was observed (65% AUC). Hence, in this paper, we investigate the generalizability of our models on images from a held out country’s dataset. We explore the extent of the problem and the possible reasons behind the lack of good generalization. A comparison of radiologist-annotated lesion locations in the lung and the trained model’s localization of areas of interest, using GradCAM, did not show much overlap. Using the same network architecture, a multi-country classifier was able to identify the country of origin of the X-ray with high accuracy (86%), suggesting that image acquisition differences and the distribution of non-pathological and non-anatomical aspects of the images are affecting the generalization and localization of the drug resistance classification model as well. When CXR images were severely corrupted, the performance on the validation set was still better than 60% AUC. The model overfitted to the data from countries in the cross validation set but did not generalize to the held out country. Finally, we applied a multi-task based approach that uses prior TB lesions location information to guide the classifier network to focus its attention on improving the generalization performance on the held out set from another country to 68% AUC. |
| first_indexed | 2024-03-10T01:37:44Z |
| format | Article |
| id | doaj.art-b9e33de7977b49e1b1c0edcc6a6f4cdc |
| institution | Directory Open Access Journal |
| issn | 2075-4418 |
| language | English |
| last_indexed | 2024-03-10T01:37:44Z |
| publishDate | 2022-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Diagnostics |
| spelling | doaj.art-b9e33de7977b49e1b1c0edcc6a6f4cdc2023-11-23T13:29:48ZengMDPI AGDiagnostics2075-44182022-01-0112118810.3390/diagnostics12010188Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-raysManohar Karki0Karthik Kantipudi1Feng Yang2Hang Yu3Yi Xiang J. Wang4Ziv Yaniv5Stefan Jaeger6Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, Bethesda, MD 20894, USAOffice of Cyber Infrastructure and Computational Biology, National Institute of Allergy and Infectious Diseases, Bethesda, MD 20894, USALister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, Bethesda, MD 20894, USALister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, Bethesda, MD 20894, USALister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, Bethesda, MD 20894, USAOffice of Cyber Infrastructure and Computational Biology, National Institute of Allergy and Infectious Diseases, Bethesda, MD 20894, USALister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, Bethesda, MD 20894, USAClassification of drug-resistant tuberculosis (DR-TB) and drug-sensitive tuberculosis (DS-TB) from chest radiographs remains an open problem. Our previous cross validation performance on publicly available chest X-ray (CXR) data combined with image augmentation, the addition of synthetically generated and publicly available images achieved a performance of 85% AUC with a deep convolutional neural network (CNN). However, when we evaluated the CNN model trained to classify DR-TB and DS-TB on unseen data, significant performance degradation was observed (65% AUC). Hence, in this paper, we investigate the generalizability of our models on images from a held out country’s dataset. We explore the extent of the problem and the possible reasons behind the lack of good generalization. A comparison of radiologist-annotated lesion locations in the lung and the trained model’s localization of areas of interest, using GradCAM, did not show much overlap. Using the same network architecture, a multi-country classifier was able to identify the country of origin of the X-ray with high accuracy (86%), suggesting that image acquisition differences and the distribution of non-pathological and non-anatomical aspects of the images are affecting the generalization and localization of the drug resistance classification model as well. When CXR images were severely corrupted, the performance on the validation set was still better than 60% AUC. The model overfitted to the data from countries in the cross validation set but did not generalize to the held out country. Finally, we applied a multi-task based approach that uses prior TB lesions location information to guide the classifier network to focus its attention on improving the generalization performance on the held out set from another country to 68% AUC.https://www.mdpi.com/2075-4418/12/1/188Tuberculosis (TB)drug resistancedeep learningchest X-raysgeneralizationlocalization |
| spellingShingle | Manohar Karki Karthik Kantipudi Feng Yang Hang Yu Yi Xiang J. Wang Ziv Yaniv Stefan Jaeger Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays Diagnostics Tuberculosis (TB) drug resistance deep learning chest X-rays generalization localization |
| title | Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays |
| title_full | Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays |
| title_fullStr | Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays |
| title_full_unstemmed | Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays |
| title_short | Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays |
| title_sort | generalization challenges in drug resistant tuberculosis detection from chest x rays |
| topic | Tuberculosis (TB) drug resistance deep learning chest X-rays generalization localization |
| url | https://www.mdpi.com/2075-4418/12/1/188 |
| work_keys_str_mv | AT manoharkarki generalizationchallengesindrugresistanttuberculosisdetectionfromchestxrays AT karthikkantipudi generalizationchallengesindrugresistanttuberculosisdetectionfromchestxrays AT fengyang generalizationchallengesindrugresistanttuberculosisdetectionfromchestxrays AT hangyu generalizationchallengesindrugresistanttuberculosisdetectionfromchestxrays AT yixiangjwang generalizationchallengesindrugresistanttuberculosisdetectionfromchestxrays AT zivyaniv generalizationchallengesindrugresistanttuberculosisdetectionfromchestxrays AT stefanjaeger generalizationchallengesindrugresistanttuberculosisdetectionfromchestxrays |