A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process
Despite the unabated growth of algorithmic decision-making in organizations, there is a growing consensus that numerous situations will continue to require humans in the loop. However, the blending of a formal machine and bounded human rationality also amplifies the risk of what is known as local ra...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-11-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/22/11642 |
| _version_ | 1797465966520565760 |
|---|---|
| author | Ganesh Sankaran Marco A. Palomino Martin Knahl Guido Siestrup |
| author_facet | Ganesh Sankaran Marco A. Palomino Martin Knahl Guido Siestrup |
| author_sort | Ganesh Sankaran |
| collection | DOAJ |
| description | Despite the unabated growth of algorithmic decision-making in organizations, there is a growing consensus that numerous situations will continue to require humans in the loop. However, the blending of a formal machine and bounded human rationality also amplifies the risk of what is known as local rationality. Therefore, it is crucial, especially in a data-abundant environment that characterizes algorithmic decision-making, to devise means to assess performance holistically. In this paper, we propose a simulation-based model to address the current lack of research on quantifying algorithmic interventions in a broader organizational context. Our approach allows the combining of causal modeling and data science algorithms to represent decision settings involving a mix of machine and human rationality to measure performance. As a testbed, we consider the case of a fictitious company trying to improve its forecasting process with the help of a machine learning approach. The example demonstrates that a myopic assessment obscures problems that only a broader framing reveals. It highlights the value of a systems view since the effects of the interplay between human and algorithmic decisions can be largely unintuitive. Such a simulation-based approach can be an effective tool in efforts to delineate roles for humans and algorithms in hybrid contexts. |
| first_indexed | 2024-03-09T18:29:07Z |
| format | Article |
| id | doaj.art-3285b53d68ca481ea0503a34d03308b8 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T18:29:07Z |
| publishDate | 2022-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-3285b53d68ca481ea0503a34d03308b82023-11-24T07:38:34ZengMDPI AGApplied Sciences2076-34172022-11-0112221164210.3390/app122211642A Modeling Approach for Measuring the Performance of a Human-AI Collaborative ProcessGanesh Sankaran0Marco A. Palomino1Martin Knahl2Guido Siestrup3School of Engineering, Computing and Mathematics, University of Plymouth, Plymouth PL4 8AA, UKSchool of Engineering, Computing and Mathematics, University of Plymouth, Plymouth PL4 8AA, UKBusiness Information Systems, Hochschule Furtwangen University, 78120 Furtwangen, GermanyBusiness Information Systems, Hochschule Furtwangen University, 78120 Furtwangen, GermanyDespite the unabated growth of algorithmic decision-making in organizations, there is a growing consensus that numerous situations will continue to require humans in the loop. However, the blending of a formal machine and bounded human rationality also amplifies the risk of what is known as local rationality. Therefore, it is crucial, especially in a data-abundant environment that characterizes algorithmic decision-making, to devise means to assess performance holistically. In this paper, we propose a simulation-based model to address the current lack of research on quantifying algorithmic interventions in a broader organizational context. Our approach allows the combining of causal modeling and data science algorithms to represent decision settings involving a mix of machine and human rationality to measure performance. As a testbed, we consider the case of a fictitious company trying to improve its forecasting process with the help of a machine learning approach. The example demonstrates that a myopic assessment obscures problems that only a broader framing reveals. It highlights the value of a systems view since the effects of the interplay between human and algorithmic decisions can be largely unintuitive. Such a simulation-based approach can be an effective tool in efforts to delineate roles for humans and algorithms in hybrid contexts.https://www.mdpi.com/2076-3417/12/22/11642machine learningsystem dynamicssimulation modelingalgorithmic decision-makingbounded rationalitysupply chain planning |
| spellingShingle | Ganesh Sankaran Marco A. Palomino Martin Knahl Guido Siestrup A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process Applied Sciences machine learning system dynamics simulation modeling algorithmic decision-making bounded rationality supply chain planning |
| title | A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process |
| title_full | A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process |
| title_fullStr | A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process |
| title_full_unstemmed | A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process |
| title_short | A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process |
| title_sort | modeling approach for measuring the performance of a human ai collaborative process |
| topic | machine learning system dynamics simulation modeling algorithmic decision-making bounded rationality supply chain planning |
| url | https://www.mdpi.com/2076-3417/12/22/11642 |
| work_keys_str_mv | AT ganeshsankaran amodelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess AT marcoapalomino amodelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess AT martinknahl amodelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess AT guidosiestrup amodelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess AT ganeshsankaran modelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess AT marcoapalomino modelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess AT martinknahl modelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess AT guidosiestrup modelingapproachformeasuringtheperformanceofahumanaicollaborativeprocess |