Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and Trust
A novel set of system-state and control-action penalty functions are introduced as an alternative to traditional performance index contingency ranking. The novel system state penalty metrics are formulated based on piecewise linear functions of the system voltage and branch flow, guided by Weber&...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10273136/ |
| _version_ | 1827794582989963264 |
|---|---|
| author | Alexander A. Anderson Brett A. Jefferson Slaven Kincic John E. Wenskovitch Corey K. Fallon Jessica A. Baweja Yousu Chen |
| author_facet | Alexander A. Anderson Brett A. Jefferson Slaven Kincic John E. Wenskovitch Corey K. Fallon Jessica A. Baweja Yousu Chen |
| author_sort | Alexander A. Anderson |
| collection | DOAJ |
| description | A novel set of system-state and control-action penalty functions are introduced as an alternative to traditional performance index contingency ranking. The novel system state penalty metrics are formulated based on piecewise linear functions of the system voltage and branch flow, guided by Weber’s Law of human cognition. Novel continuous and discrete control action metrics are also developed to measure the inherent cost and risk associated with every action taken by human power system operator to resolve violations on a pre-contingent basis. These new metrics are combined with traditional human factors indices for measuring human-machine trust and cognitive workload to create a systematic framework for measuring and evaluating operator trust and reliance on artificial intelligence (AI) algorithms for control room use. An existing AI-based contingency analysis recommender tool using a semi-supervised action algorithm is selected for a series of experiments with operations engineering staff using the IEEE 118 Bus System. The penalty metrics presented are demonstrated for both steady-state contingency analysis and transient stability studies, with the operations participants able to reduce the total system penalty in 85% of scenarios through remedial actions. A human-machine team was able to achieve equal or lower continuous control action penalty scores than the participant without availability of the recommender in 57% of experiment scenarios and lower continuous control action penalty scores than the AI tool alone in 83% of scenarios. |
| first_indexed | 2024-03-11T18:35:58Z |
| format | Article |
| id | doaj.art-85a55fa7f3f64565ae6eeccd45f1dca7 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T18:35:58Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-85a55fa7f3f64565ae6eeccd45f1dca72023-10-12T23:01:26ZengIEEEIEEE Access2169-35362023-01-011110968910970710.1109/ACCESS.2023.332213310273136Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and TrustAlexander A. Anderson0https://orcid.org/0000-0001-9678-9963Brett A. Jefferson1Slaven Kincic2https://orcid.org/0000-0002-2003-0499John E. Wenskovitch3https://orcid.org/0000-0002-0573-6442Corey K. Fallon4Jessica A. Baweja5https://orcid.org/0000-0001-8466-8611Yousu Chen6https://orcid.org/0000-0001-7591-9597Energy, Infrastructure, and Buildings Division, Pacific Northwest National Laboratory, Richland, WA, USAComputing and Analytics Division, Pacific Northwest National Laboratory, Richland, WA, USAEnergy, Infrastructure, and Buildings Division, Pacific Northwest National Laboratory, Richland, WA, USAComputing and Analytics Division, Pacific Northwest National Laboratory, Richland, WA, USAComputing and Analytics Division, Pacific Northwest National Laboratory, Richland, WA, USAComputing and Analytics Division, Pacific Northwest National Laboratory, Richland, WA, USAEnergy, Infrastructure, and Buildings Division, Pacific Northwest National Laboratory, Richland, WA, USAA novel set of system-state and control-action penalty functions are introduced as an alternative to traditional performance index contingency ranking. The novel system state penalty metrics are formulated based on piecewise linear functions of the system voltage and branch flow, guided by Weber’s Law of human cognition. Novel continuous and discrete control action metrics are also developed to measure the inherent cost and risk associated with every action taken by human power system operator to resolve violations on a pre-contingent basis. These new metrics are combined with traditional human factors indices for measuring human-machine trust and cognitive workload to create a systematic framework for measuring and evaluating operator trust and reliance on artificial intelligence (AI) algorithms for control room use. An existing AI-based contingency analysis recommender tool using a semi-supervised action algorithm is selected for a series of experiments with operations engineering staff using the IEEE 118 Bus System. The penalty metrics presented are demonstrated for both steady-state contingency analysis and transient stability studies, with the operations participants able to reduce the total system penalty in 85% of scenarios through remedial actions. A human-machine team was able to achieve equal or lower continuous control action penalty scores than the participant without availability of the recommender in 57% of experiment scenarios and lower continuous control action penalty scores than the AI tool alone in 83% of scenarios.https://ieeexplore.ieee.org/document/10273136/Artificial intelligenceenergy management systemshuman factorspower system reliabilitypower system stabilitytechnology acceptance |
| spellingShingle | Alexander A. Anderson Brett A. Jefferson Slaven Kincic John E. Wenskovitch Corey K. Fallon Jessica A. Baweja Yousu Chen Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and Trust IEEE Access Artificial intelligence energy management systems human factors power system reliability power system stability technology acceptance |
| title | Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and Trust |
| title_full | Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and Trust |
| title_fullStr | Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and Trust |
| title_full_unstemmed | Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and Trust |
| title_short | Human-Centric Contingency Analysis Metrics for Evaluating Operator Performance and Trust |
| title_sort | human centric contingency analysis metrics for evaluating operator performance and trust |
| topic | Artificial intelligence energy management systems human factors power system reliability power system stability technology acceptance |
| url | https://ieeexplore.ieee.org/document/10273136/ |
| work_keys_str_mv | AT alexanderaanderson humancentriccontingencyanalysismetricsforevaluatingoperatorperformanceandtrust AT brettajefferson humancentriccontingencyanalysismetricsforevaluatingoperatorperformanceandtrust AT slavenkincic humancentriccontingencyanalysismetricsforevaluatingoperatorperformanceandtrust AT johnewenskovitch humancentriccontingencyanalysismetricsforevaluatingoperatorperformanceandtrust AT coreykfallon humancentriccontingencyanalysismetricsforevaluatingoperatorperformanceandtrust AT jessicaabaweja humancentriccontingencyanalysismetricsforevaluatingoperatorperformanceandtrust AT yousuchen humancentriccontingencyanalysismetricsforevaluatingoperatorperformanceandtrust |