Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm”
One concern about the application of medical artificial intelligence (AI) regards the “black box” feature which can only be viewed in terms of its inputs and outputs, with no way to understand the AI's algorithm. This is problematic because patients, physicians, and even designers, do not under...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-02-01
|
| Series: | Intelligent Medicine |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2667102623000578 |
| _version_ | 1797220494150205440 |
|---|---|
| author | Hanhui Xu Kyle Michael James Shuttleworth |
| author_facet | Hanhui Xu Kyle Michael James Shuttleworth |
| author_sort | Hanhui Xu |
| collection | DOAJ |
| description | One concern about the application of medical artificial intelligence (AI) regards the “black box” feature which can only be viewed in terms of its inputs and outputs, with no way to understand the AI's algorithm. This is problematic because patients, physicians, and even designers, do not understand why or how a treatment recommendation is produced by AI technologies. One view claims that the worry about black-box medicine is unreasonable because AI systems outperform human doctors in identifying the disease. Furthermore, under the medical AI-physician-patient model, the physician can undertake the responsibility of interpreting the medical AI's diagnosis. In this study, we focus on the potential harm caused by the unexplainability feature of medical AI and try to show that such possible harm is underestimated. We will seek to contribute to the literature from three aspects. First, we appealed to a thought experiment to show that although the medical AI systems perform better on accuracy, the harm caused by medical AI's misdiagnoses may be more serious than that caused by human doctors’ misdiagnoses in some cases. Second, in patient-centered medicine, physicians were obligated to provide adequate information to their patients in medical decision-making. However, the unexplainability feature of medical AI systems would limit the patient's autonomy. Last, we tried to illustrate the psychological and financial burdens that may be caused by the unexplainablity feature of medical AI systems, which seems to be ignored by the previous ethical discussions. |
| first_indexed | 2024-04-24T12:50:26Z |
| format | Article |
| id | doaj.art-ce53bd5cea7140c484263ea250f7dcd3 |
| institution | Directory Open Access Journal |
| issn | 2667-1026 |
| language | English |
| last_indexed | 2024-04-24T12:50:26Z |
| publishDate | 2024-02-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Intelligent Medicine |
| spelling | doaj.art-ce53bd5cea7140c484263ea250f7dcd32024-04-06T04:40:41ZengElsevierIntelligent Medicine2667-10262024-02-01415257Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm”Hanhui Xu0Kyle Michael James Shuttleworth1School of Medicine, Nankai University, Tianjin, 300071, China; Corresponding author: Hanhui Xu, School of Medicine, Nankai University, Tianjin 300071, China (Email: 018205@nankai.edu.cn).Department of Global Connectivity, Akita International University, Okutsubakidai-193-2 Yuwatsubakigawa, Akita, 010-1211, JapanOne concern about the application of medical artificial intelligence (AI) regards the “black box” feature which can only be viewed in terms of its inputs and outputs, with no way to understand the AI's algorithm. This is problematic because patients, physicians, and even designers, do not understand why or how a treatment recommendation is produced by AI technologies. One view claims that the worry about black-box medicine is unreasonable because AI systems outperform human doctors in identifying the disease. Furthermore, under the medical AI-physician-patient model, the physician can undertake the responsibility of interpreting the medical AI's diagnosis. In this study, we focus on the potential harm caused by the unexplainability feature of medical AI and try to show that such possible harm is underestimated. We will seek to contribute to the literature from three aspects. First, we appealed to a thought experiment to show that although the medical AI systems perform better on accuracy, the harm caused by medical AI's misdiagnoses may be more serious than that caused by human doctors’ misdiagnoses in some cases. Second, in patient-centered medicine, physicians were obligated to provide adequate information to their patients in medical decision-making. However, the unexplainability feature of medical AI systems would limit the patient's autonomy. Last, we tried to illustrate the psychological and financial burdens that may be caused by the unexplainablity feature of medical AI systems, which seems to be ignored by the previous ethical discussions.http://www.sciencedirect.com/science/article/pii/S2667102623000578Medical artificial intelligenceBlack box problemDo no harmPaternalism |
| spellingShingle | Hanhui Xu Kyle Michael James Shuttleworth Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm” Intelligent Medicine Medical artificial intelligence Black box problem Do no harm Paternalism |
| title | Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm” |
| title_full | Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm” |
| title_fullStr | Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm” |
| title_full_unstemmed | Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm” |
| title_short | Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm” |
| title_sort | medical artificial intelligence and the black box problem a view based on the ethical principle of do no harm |
| topic | Medical artificial intelligence Black box problem Do no harm Paternalism |
| url | http://www.sciencedirect.com/science/article/pii/S2667102623000578 |
| work_keys_str_mv | AT hanhuixu medicalartificialintelligenceandtheblackboxproblemaviewbasedontheethicalprincipleofdonoharm AT kylemichaeljamesshuttleworth medicalartificialintelligenceandtheblackboxproblemaviewbasedontheethicalprincipleofdonoharm |