NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness
A novel approach presented herein transforms the Human Machine Interface (HMI) states, as a pattern of visual feedback states that encompass both operator actions and process states, from a multi-variate time-series to a natural language processing (NLP) modeling domain. The goal of this approach is...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-06-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/20/11/3228 |
| _version_ | 1797566048212353024 |
|---|---|
| author | Harsh V. P. Singh Qusay H. Mahmoud |
| author_facet | Harsh V. P. Singh Qusay H. Mahmoud |
| author_sort | Harsh V. P. Singh |
| collection | DOAJ |
| description | A novel approach presented herein transforms the Human Machine Interface (HMI) states, as a pattern of visual feedback states that encompass both operator actions and process states, from a multi-variate time-series to a natural language processing (NLP) modeling domain. The goal of this approach is to predict operator response patterns for <inline-formula> <math display="inline"> <semantics> <mrow> <mi>n</mi> <mo>−</mo> <mi>a</mi> <mi>h</mi> <mi>e</mi> <mi>a</mi> <mi>d</mi> </mrow> </semantics> </math> </inline-formula> time-step window given <inline-formula> <math display="inline"> <semantics> <mrow> <mi>k</mi> <mo>−</mo> <mi>l</mi> <mi>a</mi> <mi>g</mi> <mi>g</mi> <mi>e</mi> <mi>d</mi> </mrow> </semantics> </math> </inline-formula> past HMI state patterns. The NLP approach offers the possibility of encoding (semantic) contextual relations within HMI state patterns. Towards which, a technique for framing raw HMI data for supervised training using sequence-to-sequence (<i>seq2seq</i>) deep-learning machine translation algorithms is presented. In addition, a custom <i>Seq2Seq</i> convolutional neural network (CNN) NLP model based on current state-of-the-art design elements such as attention, is compared against a standard recurrent neural network (RNN) based NLP model. Results demonstrate comparable effectiveness of both the designs of NLP models evaluated for modeling HMI states. RNN NLP models showed higher (<inline-formula> <math display="inline"> <semantics> <mrow> <mo>≈</mo> <mn>26</mn> <mo>%</mo> </mrow> </semantics> </math> </inline-formula>) forecast accuracy, in general for both in-sample and out-of-sample test datasets. However, custom CNN NLP model showed higher (<inline-formula> <math display="inline"> <semantics> <mrow> <mo>≈</mo> <mn>53</mn> <mo>%</mo> </mrow> </semantics> </math> </inline-formula>) validation accuracy indicative of less over-fitting with the same amount of available training data. The real-world application of the proposed NLP modeling of industrial HMIs, such as in power generating stations control rooms, aviation (cockpits), and so forth, is towards the realization of a non-intrusive operator situational awareness monitoring framework through prediction of HMI states. |
| first_indexed | 2024-03-10T19:20:21Z |
| format | Article |
| id | doaj.art-e0a3854628b84ea6ba524eab1bf688a4 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T19:20:21Z |
| publishDate | 2020-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-e0a3854628b84ea6ba524eab1bf688a42023-11-20T03:01:46ZengMDPI AGSensors1424-82202020-06-012011322810.3390/s20113228NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational AwarenessHarsh V. P. Singh0Qusay H. Mahmoud1Department of Electrical, Computer and Software Engineering, Ontario Tech University, Oshawa, ON L1G 0C5, CanadaDepartment of Electrical, Computer and Software Engineering, Ontario Tech University, Oshawa, ON L1G 0C5, CanadaA novel approach presented herein transforms the Human Machine Interface (HMI) states, as a pattern of visual feedback states that encompass both operator actions and process states, from a multi-variate time-series to a natural language processing (NLP) modeling domain. The goal of this approach is to predict operator response patterns for <inline-formula> <math display="inline"> <semantics> <mrow> <mi>n</mi> <mo>−</mo> <mi>a</mi> <mi>h</mi> <mi>e</mi> <mi>a</mi> <mi>d</mi> </mrow> </semantics> </math> </inline-formula> time-step window given <inline-formula> <math display="inline"> <semantics> <mrow> <mi>k</mi> <mo>−</mo> <mi>l</mi> <mi>a</mi> <mi>g</mi> <mi>g</mi> <mi>e</mi> <mi>d</mi> </mrow> </semantics> </math> </inline-formula> past HMI state patterns. The NLP approach offers the possibility of encoding (semantic) contextual relations within HMI state patterns. Towards which, a technique for framing raw HMI data for supervised training using sequence-to-sequence (<i>seq2seq</i>) deep-learning machine translation algorithms is presented. In addition, a custom <i>Seq2Seq</i> convolutional neural network (CNN) NLP model based on current state-of-the-art design elements such as attention, is compared against a standard recurrent neural network (RNN) based NLP model. Results demonstrate comparable effectiveness of both the designs of NLP models evaluated for modeling HMI states. RNN NLP models showed higher (<inline-formula> <math display="inline"> <semantics> <mrow> <mo>≈</mo> <mn>26</mn> <mo>%</mo> </mrow> </semantics> </math> </inline-formula>) forecast accuracy, in general for both in-sample and out-of-sample test datasets. However, custom CNN NLP model showed higher (<inline-formula> <math display="inline"> <semantics> <mrow> <mo>≈</mo> <mn>53</mn> <mo>%</mo> </mrow> </semantics> </math> </inline-formula>) validation accuracy indicative of less over-fitting with the same amount of available training data. The real-world application of the proposed NLP modeling of industrial HMIs, such as in power generating stations control rooms, aviation (cockpits), and so forth, is towards the realization of a non-intrusive operator situational awareness monitoring framework through prediction of HMI states.https://www.mdpi.com/1424-8220/20/11/3228Human Machine Interface (HMI)human-in-the-loop (HITL)natural language processing (NLP)situational awareness (SA)sequence-to-sequence (seq2seq) |
| spellingShingle | Harsh V. P. Singh Qusay H. Mahmoud NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness Sensors Human Machine Interface (HMI) human-in-the-loop (HITL) natural language processing (NLP) situational awareness (SA) sequence-to-sequence (seq2seq) |
| title | NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness |
| title_full | NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness |
| title_fullStr | NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness |
| title_full_unstemmed | NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness |
| title_short | NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness |
| title_sort | nlp based approach for predicting hmi state sequences towards monitoring operator situational awareness |
| topic | Human Machine Interface (HMI) human-in-the-loop (HITL) natural language processing (NLP) situational awareness (SA) sequence-to-sequence (seq2seq) |
| url | https://www.mdpi.com/1424-8220/20/11/3228 |
| work_keys_str_mv | AT harshvpsingh nlpbasedapproachforpredictinghmistatesequencestowardsmonitoringoperatorsituationalawareness AT qusayhmahmoud nlpbasedapproachforpredictinghmistatesequencestowardsmonitoringoperatorsituationalawareness |