Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification
With the increasing prevalence of AI-assisted decision-making in the healthcare domain, evaluating fairness of machine learning models is more central than ever. Measuring the fairness of medical decision-support systems has enormous impacts on patients of different backgrounds and can influence how...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2024
|
| Online Access: | https://hdl.handle.net/1721.1/156974 |
| _version_ | 1824458266591100928 |
|---|---|
| author | Sandadi, Varsha |
| author2 | Ghassemi, Marzyeh |
| author_facet | Ghassemi, Marzyeh Sandadi, Varsha |
| author_sort | Sandadi, Varsha |
| collection | MIT |
| description | With the increasing prevalence of AI-assisted decision-making in the healthcare domain, evaluating fairness of machine learning models is more central than ever. Measuring the fairness of medical decision-support systems has enormous impacts on patients of different backgrounds and can influence how clinicians make decisions. In this study, we conduct a fairness analysis on the top 8-10 performing machine learning and artificial intelligence models from the Radiological Society of North America cervical spine fracture detection challenge and abdominal trauma detection challenge. Seven metrics are used for a more comprehensive assessment on fairness. Our findings indicate that cervical spine fracture detection models exhibit overall fairness, while abdominal trauma detection models demonstrate some unfairness in specific injury regions, possibly due to limited sample size. We also explore the performance of top models from the intracranial hemorrhage detection challenge across clinician-labeled "easy," "medium," and "hard" cases, revealing a lower accuracy rate on hard cases. This study underscores the need for additional model testing and comprehensive data representation to ensure fairness before real-world deployment in healthcare systems. |
| first_indexed | 2025-02-19T04:23:10Z |
| format | Thesis |
| id | mit-1721.1/156974 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2025-02-19T04:23:10Z |
| publishDate | 2024 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1569742024-09-25T03:26:16Z Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification Sandadi, Varsha Ghassemi, Marzyeh Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences With the increasing prevalence of AI-assisted decision-making in the healthcare domain, evaluating fairness of machine learning models is more central than ever. Measuring the fairness of medical decision-support systems has enormous impacts on patients of different backgrounds and can influence how clinicians make decisions. In this study, we conduct a fairness analysis on the top 8-10 performing machine learning and artificial intelligence models from the Radiological Society of North America cervical spine fracture detection challenge and abdominal trauma detection challenge. Seven metrics are used for a more comprehensive assessment on fairness. Our findings indicate that cervical spine fracture detection models exhibit overall fairness, while abdominal trauma detection models demonstrate some unfairness in specific injury regions, possibly due to limited sample size. We also explore the performance of top models from the intracranial hemorrhage detection challenge across clinician-labeled "easy," "medium," and "hard" cases, revealing a lower accuracy rate on hard cases. This study underscores the need for additional model testing and comprehensive data representation to ensure fairness before real-world deployment in healthcare systems. M.Eng. 2024-09-24T18:24:04Z 2024-09-24T18:24:04Z 2024-05 2024-07-11T15:31:08.480Z Thesis https://hdl.handle.net/1721.1/156974 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Sandadi, Varsha Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification |
| title | Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification |
| title_full | Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification |
| title_fullStr | Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification |
| title_full_unstemmed | Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification |
| title_short | Evaluating Fairness of Artificial Intelligence Models for Radiology Image Classification |
| title_sort | evaluating fairness of artificial intelligence models for radiology image classification |
| url | https://hdl.handle.net/1721.1/156974 |
| work_keys_str_mv | AT sandadivarsha evaluatingfairnessofartificialintelligencemodelsforradiologyimageclassification |