| Summary: | A novel approach presented herein transforms the Human Machine Interface (HMI) states, as a pattern of visual feedback states that encompass both operator actions and process states, from a multi-variate time-series to a natural language processing (NLP) modeling domain. The goal of this approach is to predict operator response patterns for <inline-formula> <math display="inline"> <semantics> <mrow> <mi>n</mi> <mo>−</mo> <mi>a</mi> <mi>h</mi> <mi>e</mi> <mi>a</mi> <mi>d</mi> </mrow> </semantics> </math> </inline-formula> time-step window given <inline-formula> <math display="inline"> <semantics> <mrow> <mi>k</mi> <mo>−</mo> <mi>l</mi> <mi>a</mi> <mi>g</mi> <mi>g</mi> <mi>e</mi> <mi>d</mi> </mrow> </semantics> </math> </inline-formula> past HMI state patterns. The NLP approach offers the possibility of encoding (semantic) contextual relations within HMI state patterns. Towards which, a technique for framing raw HMI data for supervised training using sequence-to-sequence (<i>seq2seq</i>) deep-learning machine translation algorithms is presented. In addition, a custom <i>Seq2Seq</i> convolutional neural network (CNN) NLP model based on current state-of-the-art design elements such as attention, is compared against a standard recurrent neural network (RNN) based NLP model. Results demonstrate comparable effectiveness of both the designs of NLP models evaluated for modeling HMI states. RNN NLP models showed higher (<inline-formula> <math display="inline"> <semantics> <mrow> <mo>≈</mo> <mn>26</mn> <mo>%</mo> </mrow> </semantics> </math> </inline-formula>) forecast accuracy, in general for both in-sample and out-of-sample test datasets. However, custom CNN NLP model showed higher (<inline-formula> <math display="inline"> <semantics> <mrow> <mo>≈</mo> <mn>53</mn> <mo>%</mo> </mrow> </semantics> </math> </inline-formula>) validation accuracy indicative of less over-fitting with the same amount of available training data. The real-world application of the proposed NLP modeling of industrial HMIs, such as in power generating stations control rooms, aviation (cockpits), and so forth, is towards the realization of a non-intrusive operator situational awareness monitoring framework through prediction of HMI states.
|
|---|