=Paper=
{{Paper
|id=Vol-2488/paper16
|storemode=property
|title=Application Artificial Intelligence for Real-Time Monitoring, Diagnostics, and Correction Human State
|pdfUrl=https://ceur-ws.org/Vol-2488/paper16.pdf
|volume=Vol-2488
|authors=Tetiana Shmelova,Olexandr Sechkok
|dblpUrl=https://dblp.org/rec/conf/iddm/ShmelovaS19
}}
==Application Artificial Intelligence for Real-Time Monitoring, Diagnostics, and Correction Human State==
https://ceur-ws.org/Vol-2488/paper16.pdf
Application Artificial Intelligence for Real-Time
Monitoring, Diagnostics, and Correction Human State
Tetiana Shmelova1[0000-0002-9737-6906], Oleksandr Sechko2[0000-0002-4136-5511]
1
National Aviation University, Komarova av., 1, 03058, Kyiv, Ukraine
shmelova@ukr.net
2
Uzhhorod National University, Narodna Square, 3, 88000, Uzhgorod, Ukraine
aleksander.sechko@gmail.com
Abstract. The authors present the Artificial Intelligence system of monitoring,
diagnostic and according to the correction of the emotional state of patients dur-
ing treatment. Monitoring of the current emotional state of the human, diagnos-
tics of the deformations of emotional experience and determination of the oper-
ator's functional stability will allow timely prevent the development of the situa-
tion towards worsening. Developed Intellectual automated control system
(IACS) to monitor the human condition, Expert System “Estimation of com-
fortable ergonomics for patients”
Keywords: Artificial Intelligence, Expert system, Diagnostics Module, Defor-
mation of Emotional State, Hodograph, Stability
Key Terms: Development, Modelling, Process, Methods of correction, Expert
estimation
1 Introduction
The life of modern people filled with different events, one-way or another involves
changes to the mental and emotional state. The manifestations of mental and emotion-
al states can be both positive and negative. The negative consequence considered
“burnout syndrome”. The nature of the emotional stat e significantly affects the heal-
ing process of patients. It is very important to continuously monitor the emotional
state, timely diagnosis, and deformation of the emotional state of patients during
treatment. With a long process of recovery, patients experience fatigue and overwork,
which can be classified as burnout syndrome. This emotional state is the most danger-
ous because it can lead to economic losses and irreversible consequences in their pro-
fessional activities. So, at the European Conference of the World Health Organization,
held in 2005, it noted that the cost of solving the problems of the mental and emotion-
al state of the working people of the European Union amounts to 3-4% of gross na-
tional income [1]. Burnout syndrome faced by people whose work takes place under
conditions of constant stress and responsibility for life other people, namely, military
personnel, pilots, astronauts, medical professionals, teachers, social workers [2].
The main reason for burnout considered a psychological, mental super fatigue. This
occurs when the requirements (internal and external) for a long time dominated over
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0)
2019 IDDM Workshops.
resources (internal and external) a human equilibrium is disturbed, which inevitably
leads to a syndrome of mental and emotional burnout. The connection of the identi-
fied changes to the nature of the professional activities involving responsibility for the
fate, health, and lives of people is set. These changes interpreted as the result of expo-
sure to prolonged occupational stress.
The psycho-emotional state is the human reaction to the relationship with the envi-
ronment, expressed in terms of the appearance of his feeling comfortable or uncom-
fortable state. Independently recognize the state of burnout syndrome is impossible
because of the conservatism of retardation and personality assessments. Therefore,
scientists of different countries have developed a methodology for assessing the men-
tal and emotional state, and developed tools for the independent qualification of this
state [3; 4].
Now allocate about 100 symptoms, one way or another associated with the syn-
drome of emotional burnout. First of all, it should be noted that the conditions of pro-
fessional activity can sometimes appear and cause chronic fatigue syndrome, which,
incidentally, is often accompanied by a syndrome of emotional burnout. In chronic
fatigue syndrome are typical complaints of patients: progressive fatigue, decreased
performance; poor tolerance of the previously usual loads; muscle weakness; muscle
pain; sleep disorders; headache; forgetfulness; irritability; decrease in mental activity
and ability to concentrate [3].
This publication is first offered to consider human (H) as a control object (CO),
human state monitoring and diagnostics are based on the definition of the patient’s
current state according to the analysis of phase portraits.
The authors' proposal introduces in medicine the methodology for assessing the
mental and emotional state that was used in aviation [5; 6]. Authors have experience
in aviation of operational determination of pilot’s emotional state deviations and deci-
sion making in risk applied the concept of human mental activity, which is based on a
property of consciousness delay or accelerates the flow of subjective time relative to
real-time [7]. The most common means of assessment of the pilot are piloting parame-
ters (deviation of ailerons, rudder direction, etc.) and negotiations on the flight deck,
i.e. radio communications between the pilot and the controller. More available for
investigation are piloting parameters, recorded with modern means. The pace and
range of motion of the pilot during controlling the air vehicle that changes with in-
creasing emotional stress is an indicator of emotional state [9].
The first attempts to organize the monitoring of the crew of civilian liners relate to
the 70 years of the last century. Then in the USA on the recommendations of the
Ames Research Center (ARC) on the body of the pilot intended to place sensors of
objective medical and biological control of the body. Their purpose was to provide
various electrograms, the information of which in real-time on the communication
channels had to be transmitted to the earth, and the earth had to determine what is
really happening to the person, and in critical cases to take appropriate measures. The
attempt failed: the pilots did not want to have sensors and conducting a signaling sys-
tem on their body, firmly stating that the sensors would interfere with the work, so for
researchers, the first place came the problem of developing contactless systems for
assessing the state of the pilot in flight. Automatic recognition of human emotions is
very important in many applications. Emotions can display facial expression, voice,
stroke, pulse, blood pressure, etc. At the moment, it is developing models of machine
learning that can "feel" individual emotions [10].
In addition to monitoring the emotional state of a person, it is proposed to addition-
ally use expert systems that can assist the experts during medical treatment.
The purpose of the publication are
building an Expert system (ES) as Artificial Intelligence (AI) for estimation
of priority of patients “Estimation of comfortable ergonomics for patients” using Ex-
pert Judgment Method (EJM);
the diagnostics of the mental and emotional state of a person; development of
the algorithms of human psycho-emotional diagnosing and monitoring.
2 Expert systems for estimation of emotional state of patients
Expert Systems are software systems developed using different techniques of artificial
intelligence that can act parallel to the "human" experts. The main role is consultative.
Databases of such systems can contain a huge number of data about different diseases,
therapy modalities, corrects methods for care, etc. In the development of "Medical
Expert Systems", the rules and knowledge of human experts are crucial. The teams of
such experts are developing an Expert System considering the changes in medicine
and care methods. The Expert System is one of the varieties of Artificial Intelligence
(AI) [11]. AI in healthcare is the use of algorithms, mathematics methods, and soft-
ware to approximate human cognition in the analysis of complex medical data and for
helping of forecasting of the future emotional state of patients after healthcare. Specif-
ically, AI is the ability for computer algorithms to approximate conclusions; the abil-
ity to gain information, process it and give a well-defined output to the end-user; the
ability to support for medics. These algorithms can recognize patterns in behavior and
create their own logic decision and rational DM. There are many Expert Systems need
for medicine, such as: quantitative estimation of the complexity of the care methods;
quantitative estimation of the complex procedures operators during the working pro-
cess; quantitative estimation of the problem; the significance of the procedures per-
formed by the patients; the importance of individual psychological factors influencing
the Decision Making (DM) in care methods; the importance of social and psychologi-
cal factors influencing the DM in care methods; definition the difficult of procedures
for care methods of persons; comfortable color characteristic of the room; quantitative
estimation of the significance of the corrected methods for improvement of emotion
states, etc.
The main task to develop this system - quantitative estimation of parameter’s sys-
tem using Expert Judgment Method (EJM) [4]. For example, consider the simple ex-
ample of building Expert System “Estimation of comfortable ergonomics for patients.
Part 1 Comfortable color characteristic of the room” - the significance of color char-
acteristics for patient recovery as characteristic of significance by sight. The famous
Luscher Color Test is a psychological test, developed for the use of psychiatrists,
psychologists, physicians and those who are professionally involved with the con-
scious and unconscious characteristics and motivations of others. As is known, emo-
tions (from lat. “emoveo”) are subjective reactions of a person to any external and
internal stimuli through the senses. A person has five basic feelings (hearing, touch,
sight, smell, and taste) that can be used to change a person’s emotional state. Re-
searchers took to calculate and choose the next colors (factors for estimation): pink,
blue, yellow, white, and choosing. In this example the “experts” are the “medics”,
who may choose a need color for patients according to ways of treatment.
Example of estimation using EJM. Firstly, it was necessary to create a matrix of in-
dividual preferences of each expert. We determined the opinion of each expert and
their systems of individual preferences are the following (Table 1).
Table 1. The matrix of individual preferences
Color Pink Blue Yellow White ∑r R
Pink * 0 1 1 2 2
Blue 1 * 1 1 3 1
Yellow 0 0 * 0,5 0.5 3.5
White 0 0 0,5 * 0.5 3.5
The system of preferences of expert No 1: S(R ) R1 R2 R3 ; R4 .
1
The next step was to gather all expert`s opinions and create the matrix of group
preferences. As a result, the matrix of group preferences is the following (Table 2).
Table 2. The matrix of group preferences.
Experts, m=5 Pink Blue Yellow White
1 2 1 3.5 3.5
2 1.5 1.5 3.5 3.5
3 1.5 3.5 1.5 3.5
4 1 2 3 4
5 1.5 1.5 3.5 3.5
Rgr 1.3 1.9 3 3.6
To determine the coordination of experts' opinions, it is necessary to calculate disper-
sion, squared deviation, and coefficient of variation. The results are in Table 3.
Table 3. The matrix of coordination of experts’ opinion
Experts Pink Blue Yellow White
1,2,…n 2 1 3.5 3.5
Rgr 1.3 1.9 3 3.6
Dj 0.12 0.92 0.75 0.05
σj 0.35 0.96 0.86 0.22
υj, % 23.57 50.61 28.87 6.21
Results < 33% > 33% < 33% < 33%
The system of preferences for group of experts:
S(Rgr ) R1 R1 R3 R4
. (1)
If the variation is less than 33% – the opinion of experts is coordinated. If the varia-
tion is more than 33% – the opinion of experts is not coordinated. For "blue", ʋ2 =
39% > 33% and need to obtain Kendal’s coordination coefficient.
As a result, Kendal’s coordination coefficient is equal to 0.71. Our result shows
that the opinions of the experts are coordinated. The significance of the calculations
using χ2-criteria (6,75 > 0,5):
76.5
f2 6.75
1 1
5( 4 1 ) 42
2 12 ( 4 1 ) . (2)
ф 2
t 2
> (3)
The rating correlation coefficient of Spirman and Student's t-criterion. The seventh
task is to compare the opinion of the group of experts and expert No 2 with the help of
the rank correlation coefficient of Spirman (Table 4).
Table 4. The matrix of the correlation coefficient of Spirman
Ranks Pink Blue Yellow White
Ranks of the group, Rgr 1.3 1.9 3 3.6
Ranks of expert No 2 1.5 1.5 3.5 3.5
Our result is rsi = 0.934. So, the coordination of opinions of the group and expert No 2
is high. The significance of the calculations using t-criteria:
42
tcritical 0.934 3.69
1 0.934 2 . (4)
6. Definition of weight coefficients i (Table 5) and graphical presentation of re-
sults (see Fig. 1). Weight coefficients wj of j-factors:
Cj
wj n
C
j 1
j
(5)
n
w 1 j
R 1
where n – is a number of factors; j 1 ; Cj 1 i – are the estimates.
n
Table 5. The matrix of the definition of the significance of color characteristics
Color Rgrj Ci i
Pink 1 1 0.4
Blue 2 0.75 0.3
Yellow 3.5 0.375 0.15
White 3.5 0.375 0.15
Fig. 1. Graphical presentation of weight coefficients
3 Intellectual Automated Control Systems of Monitoring
Human State for Medicine
For operational determination of the pilot’s emotional state deviations and bias in
decision making (DM) in risk applied the concept of human mental activity, which is
based on a property of consciousness delay or accelerates the flow of subjective time
relative to real-time. Deformation of the emotional state defined using a priori model
of human- operator (H-O), based on actual material posterior studies of investigating
aviation accidents received by the International Aviation Committee (IAC) [7]. There
are three types of H-O emotional activity:
spontaneous (optimal) type of activities;
emotional type of activities;
reasonable type of activities.
With the development and improvement of technology in modern Intellectual Au-
tomated Control Systems (IACS) the problem of effectively monitor the human state
can be solved. This research was first offered to consider human as a control object,
human state monitoring and diagnostics is based on comparison patient’s actual and
normal state according to the analysis of phase portraits (see Fig. 2). The analysis and
synthesis of the IACS are carried out in a similar way to poliergatic (man-machine)
system [4; 7; 8], however, as a "control object"(CO) is a person (patient), "control
device"(CD) is an element of control system or a doctor.
The Algorithm of human’s emotional monitoring and diagnostics
1. Definition of the phase portrait of human Hn – diagnosis of the normal con-
dition by characteristics of performance, movement, communication, etc.
2. Introduction into circuit IACS of human parameters and real-time monitoring
of human condition Hn.
3. Analysis of IACS “control device (CD) – human Hn” (current condition):
4. Determination of IACS “CD – human Hn” stability.
5. Determination of area of IACS “CD – human Hn” stability.
6. In case of violation of IACS "CD – human Hn" stability, system correction is
required.
7. Determination of characteristics of links for human condition correction.
8. Synthesis of new corrected system IACS "CD – human Hn+1".
9. Analysis of new system IACS “CD – human Hn+1” (current human condi-
tion), etc.
Fig. 2. The conceptual model of medical IACS
For testing single system approach to the researching of polyergatic systems, it is
advisable all diversity of management systems reduce to several typical systems, in
which main elements are distinguished, and it's functioning should be estimated dur-
ing the investigation of any system. Using dynamic modeling method for solving the
problems of complex technical ergatic systems maintenance can lead to exactly the
same models. The approach lies in the construction of the system of equations, which
describing CO, H-O equations of the automatic control system (ACS), analysis of
control system (CS) for the stability, synthesis of a new reliable system. IACS “CD –
H” could be displayed by following the functional diagram (Fig. 3), which reflects
activity human in case of H state monitoring, i.e. during changes of blood pressure H -
f, for example.
Fig. 3. The conceptual model of medical IACS
During identification with the help of indicators of deviation ( = f – n)
actual blood pressure f from defined pressure n of H, CD (or human (Doctor))
relevant information is analyzing and controlling action is defined until deviation
disappears. Stability of IACS “CD – Human (Hn)” has been obtained with using
criteria: Mikhailov, Nyquist, Hurwitz, area of stability for all coefficients and con-
stants. Determination of IACS "CD – Hn" stability by the Mikhailov criterion. To find
the hodograph by Mikhailov criterion it is necessary to put into a characteristic equa-
tion jw instead of p, received vector M(jw) as the sum of the actual P(w) and imagi-
nary part:
М ( р ) ( Т 2 р 1 )( Т 3 р 1 ) К1К2 К3 Кind КCD( Т1 р 1 ) (6)
М ( jw ) ( Т 2 jw 1 )( Т3 jw 1 ) К1К2 К3 Кind КCD( Т1 jw 1 ) P( w ) jQ( w ) (7)
P( w ) 1 К1 К2 К3 Кind КCD T2T3 w2 (8)
Q( w ) ( T2 Т3 К1К2 К3Кind КCDТ1 )w (9)
Where a real part is P(w) and the imaginary part is Q(w). According to the character-
istics of value P(w) and Q(w) Mikhailov’s hodograph has been built [5]. The type of
hodograph determines the stability of the system. The stability of the system has been
determined by the Nyquist criterion under appropriate deformation of emotional expe-
rience. For example, has been obtained system stability using the Nyquist criterion
with consideration of the dispersions by the operative model in the emotional state.
The indication of diagnostics results of current emotional state pilot in flight using
dynamic panel display of digital data encoding [12; 13].
Synthesis of IACS on the basis of monitoring and diagnostics of human has been
made by building a transition process. Indicators of quality of IACS and reliability of
human have been defined: over-control, oscillations, time of adjusting, etc. With the
help of IACS correction characteristic of the desired system has been obtained. Syn-
thesis and adjustment of the system are planned through a variety of methods to im-
prove human wellbeing and regulation of the human state. The intelligent automated
control system of monitoring and diagnosis of the human condition that is being treat-
ed has been proposed. System IACS has been built with the help of dynamic model-
ing principles, the algorithm of human psycho-emotional diagnosing and monitoring
through IACS system has been provided. IACS subsystems have been formalized in
the form of transmission functions and algorithm of modeling, analysis, and synthesis
by methods of IACS automatic control theory has been developed.
An example of IACS modeling by analysis of the influence of the time constants
and coefficients on the stability and reliability of the system, including and humans
has been represented. Therefore, as a CO it is proposed to consider the human, for
whom applying diagnosis and monitoring of H condition in comparison to its normal
state by the analysis of phase portraits, we can develop methods for adjusting and
improving the human condition.
In cases of large and complex data, methods can be integrated into traditional and
next-generation hybrid systems by processing unsupervised situation data in the deep
landscape models, potentially at high data rates and in near real time, producing a
structured representation of input data with clusters that correspond to common situa-
tion types [14].
4 Conclusion
Controlling mental and emotional state becomes important for various professional
groups of people, especially those related to risk (military, pilots, social workers, and
others). This is due to the invisible appearance of this state, with heavy loss of indi-
vidual and community, as well as the difficulty of self-establishing the state of mental
and emotional causes of burnout. It is advisable to apply the methods of monitoring
the emotional state in medicine. Existing therapeutic approaches based on the patient's
part in the process of emerging from this state. In addition to the psychological impact
on the patient needed: compliance work and rest, avoiding harmful habits, proper
nutrition, clarifying professional and personal goals. Timely detection of the state of
discomfort and constant control of the quality of life is possible when using special
diagnostic equipment. Insufficient level of industrial development in this direction
stimulates the search for new and effective solutions, the basic directions of the crea-
tion of technical solutions presented. The proposed method of monitoring and diag-
nostics the patient's emotional state can have a significant interest in medical institu-
tions (hospitals, motels, hospitals, rehabilitation centers). The new system of monitor-
ing and diagnostics IACS has been built with the help of dynamic modeling princi-
ples, the algorithm of human psycho-emotional monitoring and diagnostics has been
provided. The IACS subsystems have been formalized in the form of transmission
functions; algorithm of modeling, analysis, and synthesis of IACS by automatic con-
trol theory methods has been developed. Correction of emotion state proposed do use
Expert systems, for example, in the article Expert System “Estimation of comfortable
ergonomics for patients. Comfortable color characteristic of the room” presented. An
example of IACS modeling by analysis of the influence of the time constants and
coefficients on the stability and reliability of the system, including humans, has been
represented. It is proposed to consider the human as a control object, to monitor and
to diagnose human state on the base of the phase portraits analysis, which will make it
possible to develop methods for its adjusting and improving. Monitoring of the cur-
rent emotional state of H-O and diagnostics deformations of emotional experience in
the forms of transitions to dangerous types of human activities (reasonable or emo-
tional) in dangerous situations and determining the functional stability of system "CD
- human" will allow timely prevent the development of critical situation towards
worsening.
References
1. Sidorov, P.: Burnout syndrome. Medical newspaper, No. 43. Retrieved from
http://health.mpei.ac.ru/sindrom.htm, last accessed 2019/03/19.
2. Мuhitdinov, M., Jadgarova, Z.: Bases of a single strategy and tactics the use of CFS in the
sphere of eastern philosophies of reconstructive medicine. IN SECOND SCIENTIFIC
MEDICAL CONFERENCE – CONFERENCE RESULTS. Moscow: Center-Region
(2010).
3. Izard, C.: Emotion theory and research: highlights, unanswered questions, and emerging
issues. Annual Review of Psychology, vol. 60, 1–25 (2009).
4. Salem, M., Shmelova, T.: Intelligent Expert Decision Support Systems: Methodologies,
Applications and Challenges, International Publisher of Progressive Information Science
and Technology Research, .215-242, USA, Pennsylvania (2018).
5. Shmelova, T., Kovalyov, Yu., Sechko, O., Shostak, V., Vasyliev, M.: Intellectual automat-
ed control of human state monitoring systems: In Proceedings of the 7TH WORLD
CONGRESS ON AVIATION IN THE XXIST CENTURY. Safety in Aviation and Space
Technologies. Kyiv: National Aviation University (2016).
6. Shmelova T. Kucherov D., Kovalyov Yu., Sechko O., Vasyliev M.: Algorithm of human’s
emotional diagnosing and monitoring: In Proceedings of the ХІІ INTERNATIONAL
CONF. AVIA -2017 : Kyiv: National Aviation University (2017).
7. Leychenko, S., Malishevskiy, A., Mikhalic, N.: Human Factors in Aviation: monograph in
two books. Book 1st. Kirovograd: YMEK (2006).
8. Ivashchenko, N. Automatic regulation. Theory and elements of systems. Moscow: Mechan-
ical Engineering (1973)
9. Golovakha, R. V.: Biotechnical system for monitoring the psychophysiological state of the
operator (dissertation for the degree of candidate of technical sciences). Kharkiv: Kharkiv
National University of Radio Electronics (2004).
10. Jingying, W., Baobin, L., Changye, Z., Shun, L., Tingshao, Z.: Automatic emotion recog-
nition based on non-contact gaits information. In M. Khosrow-Pour (Ed.), Advanced
Methodologies and Technologies in Artificial Intelligence, Computer Simulation, and
Human-Computer Interaction, 54-67. USA, Hershey: IGI Global (2019).
11. IATA https://www.iata.org/publications/Pages/AI-white-paper.aspx last accessed
2018/05/11
12. Shmelova, T., Kharchenko, V., Sikirda, Y., Astafjev, S.: Diagnostics of the emotional state
of the human-operator. Ukrainian certificate of registration of copyright on the product №
43526. Kyiv, State Service of Intellectual Property of Ukraine (2012)
13. Shmelova, T., Sikirda, Y., Zemlyansky, A., Astafjev, S.: Informational-analytical diagnos-
tic complex for the study of the patterns of the Air Navigation System’s human-operator.
Engineering in Agricultural Production, Branch Engineering, Automation: Collection of
Scientific Works of Kirovograd National Technical University, 25(2), 385-392 (2012).
14. Dolgikh S. Spontaneous Concept Learning with Deep Autoencoder In International Jour-
nal of Computational Intelligence Systems, Volume 12, Issue 1, November, 1 – 12, Canada
(2018)