The problem of intellectual support of human-machine interface (HMI) designers for Process Control Systems is considered. A method for solving this problem in the form of a set of models, technologies and knowledge, forming a new information technology for Control System HMI design is proposed. Software in the form of an Expert System that implements the proposed technology is developed. The use of an Expert System in the HMI design process will improve the efficiency, quality, reliability and safety of Control System operator's activity.
The introduction of new information technologies in production processes,
improvement of automation equipment leads to the fact that the human-operator
becomes the “narrowest” link in modern Control Systems. It was shown [
For today Automatic Control System development is mostly powered by Supervisory Control And Data Acquisition (SCADA) technology with wide variety of special component libraries as well as graphic editors used. However, they include no modules or add-ons to provide intellectual support for HMI ergonomic design. Hereinafter HMI determines Control System HMI developed by SCADA technology. The main purpose of our research is intellectual support of Control System HMI designers. 2
The use of intelligent systems as a component of SCADA systems or in conjunction with them is known [e.g. 7–9]. Such integration provides intelligent support of operator in the analyzing data process, diagnosing the state of technological equipment, forecasting the development of situations and making appropriate decisions in normal and emergency situations.
Operator is often remote from objects of observation and control (OOC) and performs remote control using the information model (IM) in modern Control Systems. There are many types of IM. A special case of IM is the HMI of the operator developed by SCADA technology.
There are a lot of techniques for ergonomically driven the IM design process for
Control System operators [e.g. 10]. Their disadvantage is the insufficient
formalization of the design stages. The analysis [
- software is being developed for displaying operational dispatch information on
the Internet, as well as archived data [
- basic requirements for the organization of HMI for process engineers of NPPs
with VVER-1000 reactors and a set of HMI elements that is sufficient for organizing
the control and diagnostics of a power unit is defined [
- software development of the system for improving the quality of interfaces of
computer simulation systems is under development [
- currently there are domestic standards that regulate the requirements for
ergonomic design of such types of IM as a mnemonic diagram [
There are also many HMI design guides for designing of task-based interface (Ecological Interface) (e.g. [18, 19]), human-centred design (e.g. [20]) and HMI for Process Control Applications (e.g. [21]), but they give only general guidelines for interfaces designing.
On the basis of the foregoing, we can conclude that there are practically no work in the Control System HMI design automation field. It should also be noted that the design and configuration tasks of the HMI can significantly increase the cost and complexity of the Control System design project. According to [22], designers spend $10,000 or more to develop each page of HMI graphics for their process operations. Therefore, the actual task is to develop a method of intellectual support for the design of the Control System HMI. 3
The base of the information technology for designing the Control System HMI were suggested by the author in [23] and require further development as part of this work.
The object-oriented approach, which simplifies the implementation of projects in SCADA-tools, is applied for modeling the subject domain "НМІ design". The conceptual model of the subject domain "НМІ design", made in the UML notation, is presented in Fig. 1.
The internal hierarchical structure of the subject domain reveals the aggregate association. The nature of the other relations between classes in the model is set using the named directional associations.
According to Fig. 1, the designer (the class "Designer"), using a set of standard interface elements ("Palette"), is developing the project HMI ("Current Project") step by step. The current HMI project is carried out in the workspace ("WorkSpace"), which includes a set of tools, templates, and methods and means of using them within the project. The final state of the current project (the product of the designer’s labor) is the Control System HMI ("HMI").
The designer fills the workspace with objects that are implemented using various program classes. The class "WorkSpace" is a container for the following classes: MainWindow (the main window of the НМІ development tool); PictureBox (represents a Windows picture box control for displaying an image); StatusBar (represents a Windows status bar control); ToolBar (a toolbar that is a container for a group of commands or controls); TabControl (represents a control that contains multiple items that share the same space on the screen).
Intellectual support of the designer is carried out using certain Expert System (ES). A typical ES consists of the following main components: a inference engine (interpreter); working memory (data base) intended for storage of initial and intermediate data of a current problem; knowledge base; knowledge acquisition component; explanatory component; dialog component. To display them in the conceptual model (Fig. 1), the following classes are introduced: – "Interpreter" for inference engine; – "Knowledge" for knowledge and data base (KB); – "Intelligent Interface" (subsystem "Intelligent Interface") for the remaining components.
As a result of the interaction of the designer with the subsystem "Intelligent Interface" is his intellectual support in the form of recommendations generated by ES.
ES provides intellectual support to the designer by generating advice during the design process. The process of advice inference is as follows. The input of the subsystem "Intelligent Interface" receives information: about the current state of the project (from the "Current Project"); about the typical interface element chosen by the designer (from the "Palette"), as well as the results of the dialogue with the designer (from the "Designer").
This subsystem transfers to the "Interpreter" the results of processing input information in the form of some initial judgments (prerequisites) that describe the current problem design task TSi. The "Interpreter", using the initial data and knowledge from the KB of expert system ("Knowledge"), forms a sequence of rules (advice to the designer), which lead to the solution of the problem design task TSi.
KB of N-th Man-Machine System (MMS) is represented as the tuple [24]: KB( N ) KBC ( N ), KBO( N ) , (1) where KBC(N) – KB about N-th MMS components; KBO(N) – KB about N-th MMS organization, i.e. information about the relations of the MMS components.
The knowledge base of ES consists of a set of local KB. Knowledge about components KBC(ES) and organizations KBO(ES) contained in the KB of Expert System is discussed further. In Fig. 1 presents two local KB "ErgRecuire" (denote it KBEV) and "Alphabet" (KBAL) such as KBEV , KBAL KBO(ES ) .
The first KB "ErgRecuire" contains the following knowledge of the ergonomic support for the HMI designer: psycho-physiological characteristics of the operator; rules for developing IM and HMI; code arrangement rules; rules and recommendations for the design of instrument scales and graphs.
The second KB, "Alphabet", represents knowledge about the coding methods (alphabets) of information about OOC of the technological process (TP). Qualitative and quantitative characteristics of OOC are coding in different ways (alphabets, codes). 17 coding methods are known [25].
Fig. 1 presents the following types of alphabet: "Figure" - coding with abstract geometric figures; "Line" - type of line; "ConvSigns" - conventional symbols; "Area" area of geometric figures; "Orientation" - orientation of the figure or line in space; "LettersNumbers" - letters, punctuation marks and numbers; "Color" - color; "FlickeringFreq" - flicker frequency, "Opacity" - opacity.
Subsystem "Intelligent Interface" received the results of the logical inference from the "Interpreter" and in accordance with them modifies the current HMI project. Thus, we get a solution to the current problem task TSi.
As it is known, GDI+ ("GDI+" in Fig. 1) is the Windows operating system graphic device interface, which enables functions to transfer graphic objects to a monitor. Access to GDI+ functions is implemented through a set of managed classes. The interface of the managed classes associated with images on a screen ("Screen" in Fig. 1) is part of the .NET Framework. 3.1
An analysis of НМІ of modern Control Systems made it possible to reveal their typical structures that determine knowledge about the components of НМІ (i.e. KBC(ES)). The following typical formats for presenting information in the Control System HMI are defined: mnemonic diagrams of three levels (generalized, group and individual); indications (sets of analog and discrete information); current events associated with the change of states of technological systems and processes; alarms; graphs of analog parameters (trends); instructions (for displaying the text of operational instructions); free formats that are used to obtain reference information in the operator's dialogue mode with an intelligent SCADA-system module.
Each information format is practically implemented as a specific HMI zone on the screen. Interface elements that can be located by the designer in the zones were combined into the conceptual class "Palette". The domain model "Palette" in the form of a class diagram of the UML language is shown in Fig. 2. The diagram displays two types of relations - aggregation and generalization, that is, the R2 relation of the type "IS a". The relation R2, as applied to the class diagram, describes the hierarchical structure of classes and the inheritance of their properties and behavior.
Zones of all formats are presented in the form of three classes: "Mnemonic diagram", "Trend" and "Document" in Fig. 2. The purpose of the first two classes is obvious. The "Document" class was introduced to represent all the other formats listed above.
In accordance with the model, for example, the following elements can be placed in the zone of the mnemonic diagram: "Graph"; "ObjektTP" (OOC of technological process); “Control element” (HMI Controls) and “Connecting element” (connectors on a mnemonic diagram).
The “Control element” class is the parent for the following classes: “Direct control” (direct control elements); "Information display elements" and " Information entry element" (information input elements). Each of these classes, in turn, has child classes. Most of them are typical elements of the interface "Button", "TreeView", "ComboBox" and others. HMI may also contain virtual elements that simulate real controls and devices. To describe them, the model has introduced the classes “VirtContrElem” and “VirtDispDevice” respectively.
The KBO(N) organizational knowledge of any MMS can be represented as a complex of morphological structures. The components of these structures are the many goals of the MMS, the conditions, tasks, functions, plans, means of technology and laws of the MMS control [23].
To achieve the “HMI design” goal, the designer must solve a lots of tasks
TS TSi | i 1 Ki , the nomenclature of which is due to known methodological
techniques of IM and HMI design [
KBTS Fni , MTS (TS i ), MTc i , Pmm (KM p , MTc i ) , (2) where Fni Fnij | j 1 K j – set of functions Fnij, that must be implemented to solve the i-th task; MTS(TSi) – morphological-time structure of functions Fnij that shows a planned logical-time sequence of functions necessary for solving the TSi task; MTci – set of technological ways to implement the function Fnij; Pmm(KMp, MTci) – the predicate "The implementation of the function Fnij is possible in a set of technological methods MTci with a combination of conditions KMp".
Windows Forms graphic elements can be divided into three main categories: twodimensional (2-D) vector graphics; drawings; typographic markup.
The managed class interface associated with transfer graphic objects to the monitor in the .NET Framework 4.5 and 4.6 consists of the following namespaces: System.Drawing; System.Drawing.Drawing2D; System.Drawing.Imaging; System.Drawing.Text; System.Drawing.Configuration and System.Drawing.Design.
The parent System.Drawing namespace contains types that support the basic graphical functions of GDI+. The basis of the GDI+ interface is the Graphics class. It directly performs the drawing of straight and curved lines, geometric figures, the output of drawings and text.
The analysis of the functional purpose of the System.Drawing namespace classes, the associated methods of the Graphics class, as well as the types of the alphabet, and the possibility of their use for coding the Windows Forms graphic elements was carried out.
We introduce four sets: – A1 – information transferred to a screen; – A2 – System.Drawing namespace; – A3 – Graphics class methods; – A4 – types of the alphabet.
In order to determine the knowledge required by the "Intelligent Interface" subsystem to implement the transfer and coding functions of HMI elements, we introduce the tetradic relation R3 "To transfer and code a1, use a2, a3, a4" on the A1×A2×A3×A4 Cartesian product.
The relation R3 can be matched to a 4-place predicate. A tuple (a1, a2, a3, a4) belongs to a relation if and only if the predicate of this relation is P(a1, a2, a3, a4)=1.
All elements of the R3 relation are of the same type of tuples. Tuples of the same type make it possible to consider them as analogs of rows (i) in a simple table, that is, in a table where all rows consist of the same number of cells and the same data types are located in the corresponding cells. An example of the R3 relation representation in the form of a table is presented in the Table 1.
Such codes as letters, punctuation, and numbers are represented by one "Аlphanumeric" alphabetic in Table 1. Due to the fact that the computer implementation of the "Color" and "Brightness" codes is associated with the same RGB color model, in Table 1 they are presented in one column.
After formalization and recording of the knowledge given in Table 1 in KB, the expert system will have knowledge about the transfer graphic objects to a screen technology and selection of codes for Windows Forms graphic elements. For example, when solving the problem of transfer raster images, the "Intelligent Interface" subsystem will propose to use the Image class AND/OR Icon class of the System.Drawing space and the corresponding methods of the Graphics class. If designer chooses the Image class, all kinds of codes will be recommended as possible. If he chooses Icon class, the following types of codes will be recommended: "Symbols" AND/OR "Area of figures" AND/OR "Color" AND/OR "Brightness".
The combination of the developed models, technologies and knowledge forms a new information technology of Control System HMI design. 4
The developed information technology has become a theoretical basis to develop an ES, which could deliver a methodology of the design and set of functions as the intellectual tool to support technical designers whose task is to create Control System HMI. The general scheme of the ES demonstrated at Fig. 3.
The “Simulation” subsystem provides interactive extraction of expert knowledge about TP, OOC and their parameters, algorithms of the operator’s activity. The goal of TP may be to decompose into other sub-levels (tasks, procedures, functions and operations). Knowledge about the sub-levels could be formalized in the form semantic graph. The nodes of this graph are the names of the functional units Fi of different scale, and the arcs show the relation RFiFj - "to implement the functional unit Fi (goal, task), it is necessary to perform the functional unit Fj (task, procedure, function, operation)". The root of the tree is the concept of "Technological process". The description of the TP, given by experts, is saved in the tptree.xml file.
The description of TP objects, given by experts, is saved in the objects.xml file. Here is a fragment of this file that describes the "Оperative" state of the Compressor01object of TP. This object has one quantitative parameter “Pressure” and one qualitative parameter “Noise”. The parameter "Pressure" should be in the range from 1,0 to 2,1 Pa in "Оperative" state. The operator must observe (tag "observed") and control (tag "controled") the parameter "Pressure". The “Noise” parameter is only observed and can be “true” or “false”: <objects> <object name="Compressor-01" uuid="{550e8400-e29b-41d4-a716446655440000}"> <state name="Оperative" uuid="s001"> <parameters>
<qnParameter name="Pressure" uuid="p001-p001" union="Pa" controled="true" observed="true"> <minValue>1.0</minValue> <maxValue>2.1</maxValue> </qnParameter> <qnParameter name="Noise" uuid="q001" union="" controled="false" observed="true"> <values> <value> true </value> <value> false </value> </values> </qlParameter> </parameters> </state> </object> </objects>
The use of simulation at the stage prior to the development of the design specification for HMI allows: determine the probabilities of various MMS states; estimate the ability of the operator to process the incoming information necessary for decision making and draw up a list of those situations that he cannot solve due to his psychophysiological limitations. The results obtained will allow the designer to reasonably accomplish the task allocation between the operator and the machine. In the next stages of HMI design, this subsystem, if there are several variants of “paper” HMI prototypes, ensures the selection of the optimal variant. The choice is based on such quantitative indicators of the operator's activity, such as the probability of correct execution and execution time of the control algorithms obtained as a result of a simulation experiment.
The extracted expert knowledge about TP, OOC and their parameters are input to the Control System HMI design. The HMI design process is performed with the participation of the “Coding” subsystem and “Intelligent Support” module of the “Intelligent Interface” subsystem. The scenario of the HMI designer interaction with "Coding" subsystem can be described by the use case diagram shown in Fig. 4. The "Intelligent Interface" subsystem is shown as an external entity.
For each stage of TP a scene (video frame) is built, representing the TP mode. The video frame can be divided into zones with various formats of information presentation and filled with graphic elements. Thus, HMI consists of a set of scenes.
The process of building a scene consists in coding OOC, their states and parameters, as well as developing a scene composition with the allocation of operational space zones using various data presentation formats. The result of the expert's interaction with the intellectual support module is filling the KB with the following knowledge: requirements for the information about the OOC (see position 2 in Fig. 5); about the specifics of the operator’s tasks and environment (see position 3 in Fig. 5). In accordance with the results of interaction, the HMI designer receives the ES recommendations on the choice of alphabet types for coding (see position 1 in Fig. 5). All subsequent steps of building the scene are also supported by recommendations for the designer. They are generated by the intellectual support module.
The graphic and sound editors modules are intended, respectively, for coding OOC and basic interface elements using alphabets coding visual modality (symbols, color, flicker frequency, brightness, etc.) and alphabets of auditory modality (verbal and sound signals) in accordance with ergonomic requirements.
Declarative knowledge is represented in the ES in the form of predicates (j); P(x1, x2), where j is a name of fact (the ordinal number of the fact in the KB); P is a name of predicate; xi, i=1, 2 is a subject constants.
Procedural knowledge is presented in the ES in the form of production rules (k); Ri: Ai → Ki, where k is a name of the production (the ordinal number of the rule in the KB); R is a condition for the applicability of the production core; Ai → Ki is a production core.
After adding an element to the scene, HMI designer can apply coding to it in accordance with the received recommendations. This is done by changing the corresponding properties of the element. An example of the correspondence between the properties of the Button control and the categories of codes (alphabets) that can be applied by using these properties are listed in Table 2.
Fig. 5. An example of a expert survey and the generation of recommendations on the choice of the alphabet Geometric figures, Conventional symbols, Type of line, Number of points or geometric elements, Orientation in space, Alphanumeric
Area of figure Property FlatStyle Font FontHeight ForeColor Height Image Size Text Width
These properties are changed and set using the element control properties panel.
The developed HMI project is saved in the .xml file format. Data from this file is input to the "Intelligent Interface" subsystem for the visualization of HMI. An example of the design results is presented in Fig. 6.
The urgent problem of information and mathematical support development is solved to automate the HMI designer activities.
The method of intellectual support for Control System HMI ergonomic design is proposed for the first time. Information technology that provides a methodology covering both design and pre-design stages of HMI design has been developed. At the pre-design stages of the HMI development, it is planned to carry out simulation modeling of TP and control processes (operator activities). The simulation results will allow more precise specify of the requirements for the operator activities and HMI at the terms of reference creation stage and to increase the validity of the task allocations between operator and machine in the early stages of HMI developing. If there are several “paper” prototypes or real HMI, the simulation will allow justifying the choice of the optimal variant based on quantitative indicators.
The practical significance of the obtained results is that the software realizing the proposed method is developed and can be used to solve practical problems of HMI design.
The experimental results of the Automated Control System operating analysis and modeling for aerated concrete production [26] have shown that the proposed method can be used to solve practical problems of HMI designing, as well as optimizing the structure and methods of TP controlling.
The work was performed as part of the project “Development of methods, models and information technology for managing complex objects under conditions of stochastic uncertainty” (number of 04918) of Zaporizhzhia National Technical University. 17. ANSI/ISA-101.01-2015, Human Machine Interfaces for Process Automation Systems. Julay 2015, 64 p. https://www.isa.org/store/ansi/isa-10101-2015,-human-machine-interfacesfor-process-automation-systems/41921133 (2015) 18. Burns, C.M., Haidukiewicz, J.R.: Ecological interface design. – Boca Raton: CRC Press (2004) 19. Upton, C., Doherty, G.: Extending ecological interface design principles: a manufacturing case study. In: International Journal of Human Computer Studies. 2008. 66 (4), pp. 271– 286 (2008). doi: https://doi.org/10.1016/j.ijhcs.2007.10.007 20. Innovating for people: Handbook of human-centered design methods. Pittsburgh, PA:
LUMA Institute, LLC (2012) 21. Fiset, Jean-Yves: Human-machine Interface Design for Process Control Applications. ISA.
USA. 171 p. (2009) 22. O'Brien, L.: ISA-101 Standard Addresses the HMI Lifecycle.
https://www.arcweb.com/blog/isa-101-standard-addresses-hmi-lifecycle 23. Nazarenko, N.A., Kaminska, Zh.K.: Method of human-machine interface designing of
SCADA-system. In: News SPbGETU "LETI" № 7. 2017, pp. 31–38. (In Russian) (2017) 24. Information-Control Man-Machine Systems: Research, design, test: Handbook / Under the total. ed. Gubinskogo A.I. and Evgrafova V.G. – M.: Mashinostroenie, 1993. – 528 p. (in Russian) (1993) 25. Ergonomics: principles and recommendations. Methodical manual. – M .: VNIITE. – 184 p. (in Russian) (1983) 26. Kaminska, Zh. K., Kulynych, E. M., Serdiuk, S. N.: The Automated Control System operating analysis and modeling for aerated concrete production. In: Radio Electronics, Computer Science, Control. 2018. № 4, pp. 214–225. (In Russian) (2018). doi: 10.15588/1607-3274-2018-4-21