AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening
Advances in artificial intelligence in healthcare are frequently promoted as ‘solutions’ to improve the accuracy, safety, and quality of clinical decisions, treatments, and care. Despite some diagnostic success, however, AI systems rely on forms of reductive reasoning and computational determinism t...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-03-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/7/3341 |
| _version_ | 1797440506496548864 |
|---|---|
| author | Saleem Ameen Ming-Chao Wong Kwang-Chien Yee Paul Turner |
| author_facet | Saleem Ameen Ming-Chao Wong Kwang-Chien Yee Paul Turner |
| author_sort | Saleem Ameen |
| collection | DOAJ |
| description | Advances in artificial intelligence in healthcare are frequently promoted as ‘solutions’ to improve the accuracy, safety, and quality of clinical decisions, treatments, and care. Despite some diagnostic success, however, AI systems rely on forms of reductive reasoning and computational determinism that embed problematic assumptions about clinical decision-making and clinical practice. Clinician autonomy, experience, and judgement are reduced to inputs and outputs framed as binary or multi-class classification problems benchmarked against a clinician’s capacity to identify or predict disease states. This paper examines this reductive reasoning in AI systems for colorectal cancer (CRC) to highlight their limitations and risks: (1) in AI systems themselves due to inherent biases in (a) retrospective training datasets and (b) embedded assumptions in underlying AI architectures and algorithms; (2) in the problematic and limited evaluations being conducted on AI systems prior to system integration in clinical practice; and (3) in marginalising socio-technical factors in the context-dependent interactions between clinicians, their patients, and the broader health system. The paper argues that to optimise benefits from AI systems and to avoid negative unintended consequences for clinical decision-making and patient care, there is a need for more nuanced and balanced approaches to AI system deployment and evaluation in CRC. |
| first_indexed | 2024-03-09T12:09:09Z |
| format | Article |
| id | doaj.art-e6d08f70eff94569bca630a893ab84b9 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T12:09:09Z |
| publishDate | 2022-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-e6d08f70eff94569bca630a893ab84b92023-11-30T22:54:24ZengMDPI AGApplied Sciences2076-34172022-03-01127334110.3390/app12073341AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer ScreeningSaleem Ameen0Ming-Chao Wong1Kwang-Chien Yee2Paul Turner3School of Medicine, College of Health and Medicine, University of Tasmania, Hobart 7000, AustraliaCollege of Sciences and Engineering, Information and Communication Technology, University of Tasmania, Hobart 7000, AustraliaSchool of Medicine, College of Health and Medicine, University of Tasmania, Hobart 7000, AustraliaCollege of Sciences and Engineering, Information and Communication Technology, University of Tasmania, Hobart 7000, AustraliaAdvances in artificial intelligence in healthcare are frequently promoted as ‘solutions’ to improve the accuracy, safety, and quality of clinical decisions, treatments, and care. Despite some diagnostic success, however, AI systems rely on forms of reductive reasoning and computational determinism that embed problematic assumptions about clinical decision-making and clinical practice. Clinician autonomy, experience, and judgement are reduced to inputs and outputs framed as binary or multi-class classification problems benchmarked against a clinician’s capacity to identify or predict disease states. This paper examines this reductive reasoning in AI systems for colorectal cancer (CRC) to highlight their limitations and risks: (1) in AI systems themselves due to inherent biases in (a) retrospective training datasets and (b) embedded assumptions in underlying AI architectures and algorithms; (2) in the problematic and limited evaluations being conducted on AI systems prior to system integration in clinical practice; and (3) in marginalising socio-technical factors in the context-dependent interactions between clinicians, their patients, and the broader health system. The paper argues that to optimise benefits from AI systems and to avoid negative unintended consequences for clinical decision-making and patient care, there is a need for more nuanced and balanced approaches to AI system deployment and evaluation in CRC.https://www.mdpi.com/2076-3417/12/7/3341artificial intelligencemachine learningpatient outcomessocio-technical designalgorithmic biasclinical interaction |
| spellingShingle | Saleem Ameen Ming-Chao Wong Kwang-Chien Yee Paul Turner AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening Applied Sciences artificial intelligence machine learning patient outcomes socio-technical design algorithmic bias clinical interaction |
| title | AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening |
| title_full | AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening |
| title_fullStr | AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening |
| title_full_unstemmed | AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening |
| title_short | AI and Clinical Decision Making: The Limitations and Risks of Computational Reductionism in Bowel Cancer Screening |
| title_sort | ai and clinical decision making the limitations and risks of computational reductionism in bowel cancer screening |
| topic | artificial intelligence machine learning patient outcomes socio-technical design algorithmic bias clinical interaction |
| url | https://www.mdpi.com/2076-3417/12/7/3341 |
| work_keys_str_mv | AT saleemameen aiandclinicaldecisionmakingthelimitationsandrisksofcomputationalreductionisminbowelcancerscreening AT mingchaowong aiandclinicaldecisionmakingthelimitationsandrisksofcomputationalreductionisminbowelcancerscreening AT kwangchienyee aiandclinicaldecisionmakingthelimitationsandrisksofcomputationalreductionisminbowelcancerscreening AT paulturner aiandclinicaldecisionmakingthelimitationsandrisksofcomputationalreductionisminbowelcancerscreening |