Psychological assessment of AI-based decision support systems: tool development and expected benefits
This study aimed to develop an evaluation tool that assesses the use of AI-based decision support systems (DSSs) in professional practice from a human-centered perspective. Following the International Organization for Standardization, this perspective aims to ensure that the use of interactive techn...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-09-01
|
| Series: | Frontiers in Artificial Intelligence |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frai.2023.1249322/full |
| _version_ | 1797674015672762368 |
|---|---|
| author | Katharina Buschmeyer Sarah Hatfield Julie Zenner |
| author_facet | Katharina Buschmeyer Sarah Hatfield Julie Zenner |
| author_sort | Katharina Buschmeyer |
| collection | DOAJ |
| description | This study aimed to develop an evaluation tool that assesses the use of AI-based decision support systems (DSSs) in professional practice from a human-centered perspective. Following the International Organization for Standardization, this perspective aims to ensure that the use of interactive technologies improves users' psychological load experience and behavior, e.g., in the form of reduced stress experience or increased performance. Concomitantly, this perspective attempts to proactively prevent or detect and correct the potential negative effects of these technologies on user load, such as impaired satisfaction and engagement, as early as possible. Based on this perspective, we developed and validated a questionnaire instrument, the Psychological Assessment of AI-based DSSs (PAAI), for the user-centered evaluation of the use of AI-based DSSs in practice. In particular, the instrument considers central design characteristics of AI-based DSSs and the corresponding work situation, which have a significant impact on users' psychological load. The instrument was tested in two independent studies. In Study 1, N = 223 individuals were recruited. Based on the results of item and scale analyses and an exploratory factor analysis, the newly developed instrument was refined, and the final version was tested using a confirmatory factor analysis. Findings showed acceptable-to-good fit indices, confirming the factorial validity of the PAAI. This was confirmed in a second study, which had N = 471 participants. Again, the CFA yielded acceptable-to-good fit indices. The validity was further confirmed using convergent and criterion validity analyses. |
| first_indexed | 2024-03-11T21:53:01Z |
| format | Article |
| id | doaj.art-cb1f86b82f2b49d2835efa308b935a1d |
| institution | Directory Open Access Journal |
| issn | 2624-8212 |
| language | English |
| last_indexed | 2024-03-11T21:53:01Z |
| publishDate | 2023-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Artificial Intelligence |
| spelling | doaj.art-cb1f86b82f2b49d2835efa308b935a1d2023-09-26T07:05:19ZengFrontiers Media S.A.Frontiers in Artificial Intelligence2624-82122023-09-01610.3389/frai.2023.12493221249322Psychological assessment of AI-based decision support systems: tool development and expected benefitsKatharina Buschmeyer0Sarah Hatfield1Julie Zenner2Faculty of Business, Augsburg Technical University of Applied Science, Augsburg, GermanyFaculty of Business, Augsburg Technical University of Applied Science, Augsburg, GermanyFaculty of Liberal Arts and Science, Augsburg Technical University of Applied Science, Augsburg, GermanyThis study aimed to develop an evaluation tool that assesses the use of AI-based decision support systems (DSSs) in professional practice from a human-centered perspective. Following the International Organization for Standardization, this perspective aims to ensure that the use of interactive technologies improves users' psychological load experience and behavior, e.g., in the form of reduced stress experience or increased performance. Concomitantly, this perspective attempts to proactively prevent or detect and correct the potential negative effects of these technologies on user load, such as impaired satisfaction and engagement, as early as possible. Based on this perspective, we developed and validated a questionnaire instrument, the Psychological Assessment of AI-based DSSs (PAAI), for the user-centered evaluation of the use of AI-based DSSs in practice. In particular, the instrument considers central design characteristics of AI-based DSSs and the corresponding work situation, which have a significant impact on users' psychological load. The instrument was tested in two independent studies. In Study 1, N = 223 individuals were recruited. Based on the results of item and scale analyses and an exploratory factor analysis, the newly developed instrument was refined, and the final version was tested using a confirmatory factor analysis. Findings showed acceptable-to-good fit indices, confirming the factorial validity of the PAAI. This was confirmed in a second study, which had N = 471 participants. Again, the CFA yielded acceptable-to-good fit indices. The validity was further confirmed using convergent and criterion validity analyses.https://www.frontiersin.org/articles/10.3389/frai.2023.1249322/fullAI-based decision support systemsworkhuman-centered evaluationsurvey inventorysystem propertiescharacteristics of the supported task |
| spellingShingle | Katharina Buschmeyer Sarah Hatfield Julie Zenner Psychological assessment of AI-based decision support systems: tool development and expected benefits Frontiers in Artificial Intelligence AI-based decision support systems work human-centered evaluation survey inventory system properties characteristics of the supported task |
| title | Psychological assessment of AI-based decision support systems: tool development and expected benefits |
| title_full | Psychological assessment of AI-based decision support systems: tool development and expected benefits |
| title_fullStr | Psychological assessment of AI-based decision support systems: tool development and expected benefits |
| title_full_unstemmed | Psychological assessment of AI-based decision support systems: tool development and expected benefits |
| title_short | Psychological assessment of AI-based decision support systems: tool development and expected benefits |
| title_sort | psychological assessment of ai based decision support systems tool development and expected benefits |
| topic | AI-based decision support systems work human-centered evaluation survey inventory system properties characteristics of the supported task |
| url | https://www.frontiersin.org/articles/10.3389/frai.2023.1249322/full |
| work_keys_str_mv | AT katharinabuschmeyer psychologicalassessmentofaibaseddecisionsupportsystemstooldevelopmentandexpectedbenefits AT sarahhatfield psychologicalassessmentofaibaseddecisionsupportsystemstooldevelopmentandexpectedbenefits AT juliezenner psychologicalassessmentofaibaseddecisionsupportsystemstooldevelopmentandexpectedbenefits |