Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review
<h4>Background</h4> While artificial intelligence (AI) offers possibilities of advanced clinical prediction and decision-making in healthcare, models trained on relatively homogeneous datasets, and populations poorly-representative of underlying diversity, limits generalisability and ris...
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2022-03-01
|
| Series: | PLOS Digital Health |
| Online Access: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9931338/?tool=EBI |
| _version_ | 1797703272517074944 |
|---|---|
| author | Leo Anthony Celi Jacqueline Cellini Marie-Laure Charpignon Edward Christopher Dee Franck Dernoncourt Rene Eber William Greig Mitchell Lama Moukheiber Julian Schirmer Julia Situ Joseph Paguio Joel Park Judy Gichoya Wawira Seth Yao |
| author_facet | Leo Anthony Celi Jacqueline Cellini Marie-Laure Charpignon Edward Christopher Dee Franck Dernoncourt Rene Eber William Greig Mitchell Lama Moukheiber Julian Schirmer Julia Situ Joseph Paguio Joel Park Judy Gichoya Wawira Seth Yao |
| author_sort | Leo Anthony Celi |
| collection | DOAJ |
| description | <h4>Background</h4> While artificial intelligence (AI) offers possibilities of advanced clinical prediction and decision-making in healthcare, models trained on relatively homogeneous datasets, and populations poorly-representative of underlying diversity, limits generalisability and risks biased AI-based decisions. Here, we describe the landscape of AI in clinical medicine to delineate population and data-source disparities. <h4>Methods</h4> We performed a scoping review of clinical papers published in PubMed in 2019 using AI techniques. We assessed differences in dataset country source, clinical specialty, and author nationality, sex, and expertise. A manually tagged subsample of PubMed articles was used to train a model, leveraging transfer-learning techniques (building upon an existing BioBERT model) to predict eligibility for inclusion (original, human, clinical AI literature). Of all eligible articles, database country source and clinical specialty were manually labelled. A BioBERT-based model predicted first/last author expertise. Author nationality was determined using corresponding affiliated institution information using Entrez Direct. And first/last author sex was evaluated using the Gendarize.io API. <h4>Results</h4> Our search yielded 30,576 articles, of which 7,314 (23.9%) were eligible for further analysis. Most databases came from the US (40.8%) and China (13.7%). Radiology was the most represented clinical specialty (40.4%), followed by pathology (9.1%). Authors were primarily from either China (24.0%) or the US (18.4%). First and last authors were predominately data experts (i.e., statisticians) (59.6% and 53.9% respectively) rather than clinicians. And the majority of first/last authors were male (74.1%). <h4>Interpretation</h4> U.S. and Chinese datasets and authors were disproportionately overrepresented in clinical AI, and almost all of the top 10 databases and author nationalities were from high income countries (HICs). AI techniques were most commonly employed for image-rich specialties, and authors were predominantly male, with non-clinical backgrounds. Development of technological infrastructure in data-poor regions, and diligence in external validation and model re-calibration prior to clinical implementation in the short-term, are crucial in ensuring clinical AI is meaningful for broader populations, and to avoid perpetuating global health inequity. Author summary Artificial Intelligence (AI) creates opportunities for accurate, objective and immediate decision support in healthcare with little expert input–especially valuable in resource-poor settings where there is shortage of specialist care. Given that AI poorly generalises to cohorts outside those whose data was used to train and validate the algorithms, populations in data-rich regions stand to benefit substantially more vs data-poor regions, entrenching existing healthcare disparities. Here, we show that more than half of the datasets used for clinical AI originate from either the US or China. In addition, the U.S. and China contribute over 40% of the authors of the publications. While the models may perform on-par/better than clinician decision-making in the well-represented regions, benefits elsewhere are not guaranteed. Further, we show discrepancies in gender and specialty representation–notably that almost three-quarters of the coveted first/senior authorship positions were held by men, and radiology accounted for 40% of all clinical AI manuscripts. We emphasize that building equitable sociodemographic representation in data repositories, in author nationality, gender and expertise, and in clinical specialties is crucial in ameliorating health inequities. |
| first_indexed | 2024-03-12T05:02:10Z |
| format | Article |
| id | doaj.art-cced3c08553244ca8cb277a24cb43975 |
| institution | Directory Open Access Journal |
| issn | 2767-3170 |
| language | English |
| last_indexed | 2024-03-12T05:02:10Z |
| publishDate | 2022-03-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLOS Digital Health |
| spelling | doaj.art-cced3c08553244ca8cb277a24cb439752023-09-03T09:09:46ZengPublic Library of Science (PLoS)PLOS Digital Health2767-31702022-03-0113Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global reviewLeo Anthony CeliJacqueline CelliniMarie-Laure CharpignonEdward Christopher DeeFranck DernoncourtRene EberWilliam Greig MitchellLama MoukheiberJulian SchirmerJulia SituJoseph PaguioJoel ParkJudy Gichoya WawiraSeth Yao<h4>Background</h4> While artificial intelligence (AI) offers possibilities of advanced clinical prediction and decision-making in healthcare, models trained on relatively homogeneous datasets, and populations poorly-representative of underlying diversity, limits generalisability and risks biased AI-based decisions. Here, we describe the landscape of AI in clinical medicine to delineate population and data-source disparities. <h4>Methods</h4> We performed a scoping review of clinical papers published in PubMed in 2019 using AI techniques. We assessed differences in dataset country source, clinical specialty, and author nationality, sex, and expertise. A manually tagged subsample of PubMed articles was used to train a model, leveraging transfer-learning techniques (building upon an existing BioBERT model) to predict eligibility for inclusion (original, human, clinical AI literature). Of all eligible articles, database country source and clinical specialty were manually labelled. A BioBERT-based model predicted first/last author expertise. Author nationality was determined using corresponding affiliated institution information using Entrez Direct. And first/last author sex was evaluated using the Gendarize.io API. <h4>Results</h4> Our search yielded 30,576 articles, of which 7,314 (23.9%) were eligible for further analysis. Most databases came from the US (40.8%) and China (13.7%). Radiology was the most represented clinical specialty (40.4%), followed by pathology (9.1%). Authors were primarily from either China (24.0%) or the US (18.4%). First and last authors were predominately data experts (i.e., statisticians) (59.6% and 53.9% respectively) rather than clinicians. And the majority of first/last authors were male (74.1%). <h4>Interpretation</h4> U.S. and Chinese datasets and authors were disproportionately overrepresented in clinical AI, and almost all of the top 10 databases and author nationalities were from high income countries (HICs). AI techniques were most commonly employed for image-rich specialties, and authors were predominantly male, with non-clinical backgrounds. Development of technological infrastructure in data-poor regions, and diligence in external validation and model re-calibration prior to clinical implementation in the short-term, are crucial in ensuring clinical AI is meaningful for broader populations, and to avoid perpetuating global health inequity. Author summary Artificial Intelligence (AI) creates opportunities for accurate, objective and immediate decision support in healthcare with little expert input–especially valuable in resource-poor settings where there is shortage of specialist care. Given that AI poorly generalises to cohorts outside those whose data was used to train and validate the algorithms, populations in data-rich regions stand to benefit substantially more vs data-poor regions, entrenching existing healthcare disparities. Here, we show that more than half of the datasets used for clinical AI originate from either the US or China. In addition, the U.S. and China contribute over 40% of the authors of the publications. While the models may perform on-par/better than clinician decision-making in the well-represented regions, benefits elsewhere are not guaranteed. Further, we show discrepancies in gender and specialty representation–notably that almost three-quarters of the coveted first/senior authorship positions were held by men, and radiology accounted for 40% of all clinical AI manuscripts. We emphasize that building equitable sociodemographic representation in data repositories, in author nationality, gender and expertise, and in clinical specialties is crucial in ameliorating health inequities.https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9931338/?tool=EBI |
| spellingShingle | Leo Anthony Celi Jacqueline Cellini Marie-Laure Charpignon Edward Christopher Dee Franck Dernoncourt Rene Eber William Greig Mitchell Lama Moukheiber Julian Schirmer Julia Situ Joseph Paguio Joel Park Judy Gichoya Wawira Seth Yao Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review PLOS Digital Health |
| title | Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review |
| title_full | Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review |
| title_fullStr | Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review |
| title_full_unstemmed | Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review |
| title_short | Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review |
| title_sort | sources of bias in artificial intelligence that perpetuate healthcare disparities a global review |
| url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9931338/?tool=EBI |
| work_keys_str_mv | AT leoanthonyceli sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT jacquelinecellini sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT marielaurecharpignon sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT edwardchristopherdee sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT franckdernoncourt sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT reneeber sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT williamgreigmitchell sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT lamamoukheiber sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT julianschirmer sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT juliasitu sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT josephpaguio sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT joelpark sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT judygichoyawawira sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview AT sethyao sourcesofbiasinartificialintelligencethatperpetuatehealthcaredisparitiesaglobalreview |