=Paper=
{{Paper
|id=Vol-2035/paper6
|storemode=property
|title=Intelligent System of Adaptive Training Process Based on Neurofeedback
|pdfUrl=https://ceur-ws.org/Vol-2035/paper6.pdf
|volume=Vol-2035
|authors=Vsevolod V. Tetervak,Vladimir S. Vasilyev,Aleksey V. Minin
}}
==Intelligent System of Adaptive Training Process Based on Neurofeedback==
https://ceur-ws.org/Vol-2035/paper6.pdf
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
Intelligent System of Adaptive Training Process Based on
Neurofeedback
Vsevolod V. Tetervak Vladimir S. Vasilyev Alexey V. Minin
Ural federal university Ural federal university Ural federal university
Yekaterinburg, Russia Yekaterinburg, Russia Yekaterinburg, Russia
v.v.tetervak@gmail.com vasilyev.vova@gmail.com alexey-revda@yandex.ru
Abstract
In this paper, some questions regarding design of intellectual systems using
biological feedback for user mental state management are considered. An
example implementation of such system based on the Psychology Experiment
Building Language (PEBL) and EMOTIV EPOC+ neurointerface is shown.
The results of the comparison of the EMOTIV EPOC+ interface and the
Encephalan-EEGR-19/26 mini-encephalograph, which indicate the sufficient
accuracy of the EPOC+ interface for use in neurobiofeedback systems, are
presented.
1 Introduction
Most exercises and tasks are designed for a person with regular parameters. However, each of us has a unique set of skills,
abilities and inclinations. Therefore, for effective exercise performance, it is extremely important to take into account
individual characteristics of a person. Information on these characteristics is contained both in the results of the exercises,
and in the parameters of the physiological activity of the person - the electrical activity of the brain and micro-motion of
the head (the smallest head motions which allow brain to detect body space orientation).
Modern methods of using biofeedback allow to develop useful skills much more effective than conventional training
methods [1]. This paper presents an intelligent system of adaptive training process using biological feedback -
neurofeedback. A study [2] of 2015, in which healthy adults have participated, showed that neurofeedback training
improves the ability to focus and reduces distraction. In a study [3] dated 2016, it was found that elderly people can use
this technique to improve their cognitive abilities that are impaired by aging process. In the similar way, younger and
healthy people can stimulate the workings of their brain.
In a study [3] the influence of music on the mental state of the elderly was examined with usage of Emotiv EPOC
headset. A group of 10 volunteers participated in 10 sessions (2 sessions per week), each 15 minutes long. During the
session, the patient's emotional state (determined by the beta / alpha ratio) directly determined the tempo and rhythm of
playing music track, thereby creating audible feedback. In the study [1] Emotiv EPOC was used to improve the cognitive
abilities of healthy people. A group of volunteers was selected to participate in training game sessions, consisted of tests
implemented in the PEBL programming language.
The TOVA test is one of the long-term function tests (CPT) that examines the attention function and is often used for
the diagnosis of attention deficit hyperactivity disorder (ADHD / ADHD). This test offers both visual and auditory
measures of attention using targets that the subject is instructed to respond to, as well as non-targets, for which subject is
instructed not to respond and refuse from any reaction. TOVA is ubiquitously used as a diagnostic tool to assess the level
of attention and other cognitive parameters of the user. In work [4], novice programmers were instructed, after a small
introductory video, to implement a maze passage program. During the coding session, their emotional state was determined
with the help of Emotiv EPOC. Reinforcing feedback was given if the "involvement" of the novice felt below the
predetermined threshold value.
33
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
When working with biofeedback systems, a large number of parameters should be taken into account. For their
evaluation, and also for choice of the overall behavior management strategy, the use of intelligent systems appears
extremely promising. Intellectual system is a software system capable of solving complex and compound tasks, associated
with a certain problem domain or field of knowledge, information about which are stored in the memory of such intellectual
system. The structure of the intellectual system consists of three main blocks - knowledge base, decision-making
mechanism and intelligent interface. This system is capable of reasoning about effect method by means of biological
feedback.
The purpose of this work is to develop an adaptive user state management system using biofeedback. A key feature of
such system should be not only the usage of EEG data for user’s state assessment and correction, but also the usage of
user's head movement. Also, in order to verify the possibility of using Emotiv EPOC+ headset in such a system, a
comparison of EPOC+ headset and the professional medical encephalograph "Encephalan-EEGR-19/26" mini was
conducted.
2 Methodology
Neurofeedback system
Neurofeedback therapy is widely used in treatments of patients with attention deficit disorder, hyperactivity and learning
difficulties. During the therapy course, some type of feedback is used to teach patient how to perform self-regulation and
reach the overall state of well-being. Feedback can be visual, audial or even tactile. It can be implemented as color change
on the screen, increase / decrease of virtual object’s size or proportions, various sound effects, etc. The main role of
biofeedback is, while providing all necessary information, to help patient to achieve a balanced state by means of self-
regulation and self-control.
The scheme of the adaptive training process using biofeedback is presented below.
Figure 1: General scheme of the adaptive training process using biofeedback.
Thus, the user receives information about his state. For example, in the study [5], user looked at the monitor screen, the
degree of coloration of which depended on the power of the alpha rhythm. By means of neurofeedback, user was able to
control his mental state.
34
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
State correction
There are many ways to adjust user’s state. The main difference between them is the communication method used to inform
the user about his condition at a given time. These methods can be separated in three main categories:
• Audial effects;
• Visual effects;
• Tactile effects.
After communication method is determined, the next step is to select a correction effect and calculate parameters
necessary for effective changing of user’s state. Usually some kind of physical or mental conscious activity is used as a
corrective effect. For example, in the 2005 year study [6], subjects experiencing chronic pain, while watching animation
of the flame on a display, were asked to try to reduce painful sensations by mentally extinguishing the flame and reducing
its size. In the paper [5] the study is described, in which subject looked at the screen with a red circle drawn on it, and the
color saturation of this circle depended on the level of alpha rhythm of the subject, thus encouraging him to maintain the
relaxed state and keep circle as bright as possible. Also, the corrective effect may be focusing on some object or sensation
(i.e. meditation), or performing various respiratory exercises.
To describe the state of the user, system may use different metrics. Below is a list of metrics used in this paper.
METRICS
1. Electroencephalograms (EEG) are used to monitor brain activity in real time.
2. One of the most effective methods for analyzing the motor activity of a person is the use of accelerometer data to
evaluate the accelerations experienced by a person or one of the parts of his body.
3. Indicators of the user's attention allow us to evaluate the state of his cognitive functions directly during the operation
of the system.
Emotiv headset
During the performance of cognitive loads, the most informative indication of patient’s functional state is the electrical
activity of the brain - the EEG. Estimation of EEG data can be performed in a variety of ways, and in this paper we have
concentrated on the evaluation of several power band features.
Assessment of the movement of the head during the execution of tasks allows us to identify the state of the subject and
assess the concentration on the task. Also, the data of a head flickering make it much easier to identify artifacts associated
with movement in the EEG record.
As a device that allows to collect information of both brain electrical activity and head movement, a wireless multi-
channel headset with a high resolution Emotiv EPOC+ was chosen. The characteristics of the headset are shown in Table
1.
Table 1 – EPOC+ headset specification
Number of channels 14
AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6,
Electrode names (international scheme10-20)
F4, F8, AF4
Sampling method Sequential sampling. Single ADC
Sampling rate 128 SPS (2048 Hz internal)
Resolution 14 bits 1 LSB = 0.51μV
0.2 – 43Hz, digital notch filters at 50Hz and
Bandwidth
60Hz
Filtering Built in digital 5th order Sinc filter
Dynamic range 8400μV(pp)
Coupling mode AC coupled
Connectivity Proprietary wireless 2.4GHz band
35
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
Power Internal Lithium Polymer battery 640mAh
Battery life up to 12 hours
Impedance measurement Real-time contact quality using patented system
In addition to EEG sensors, Emotiv EPOC+ is equipped with a three-axis accelerometer, which allows synchronous
recording of EEG and acceleration of the human head. Head movement data can also be used for analysis of the physical
and psychoemotional state of a person.
Motion data
One of the most effective methods for analyzing human motor activity is the use of accelerometer data to evaluate the
accelerations experienced by a person or some part of his body. The three-axis accelerometer provides information on the
magnitude of the active accelerations along three axes, respectively. The signal measured by the accelerometer is a linear
sum of three components:
• Acceleration caused body movement;
• Acceleration caused by gravity;
• Noise inherent to the measuring system.
Thus, to assess the motion of the subject, it is necessary to separate signal components characterizing the motion in
space. According to the study [7], the acceleration frequency caused by human movement is in the range from 0 to 20 Hz.
The gravitational component is in the range from 0 to 0.3 Hz. A component containing instrumental noises is usually in
the range above 20 Hz. To isolate the motion component from the signal, according to [8], a second-order Butterworth
high-pass filter with a cutoff frequency of 0.3 and a low-pass filter with a cutoff frequency of 20 Hz was used.
Assessment of the functional state
PEBL programming language
To assess the state of the user, in addition to the data of physiological activity, a widely accepted test for the variability of
attention, implemented in the programming language PEBL, was used. PEBL is a simple programming language designed
to create and conduct many standard experiments. This free software is licensed under GPL, its compiled executables and
source code available without charge. PEBL is designed to be easily used on multiple computing platforms. Its current
implementation uses SDL as its implementation platform, which is also a cross-platform library that is compiled for Win32,
Linux and Macintosh. PEBL is implemented primarily in C ++ (you do not need to know C ++ to use PEBL), it also uses
flex and bison (GNU versions of lex and yacc).
• Free software for creating psychological experiments;
• Allows you to create your own experiments or use ready-made experiments;
• Allows you to freely share experiments without a license or commitment.
The PEBL language is used to implement various specific tests designed for psychological experiments.
TOVA test
Test of Variables of Attention (TOVA) is such continuous performance test that examines the function of attention and is
frequently used in the diagnosis of attention deficit hyperactivity disorder (ADHD). This test offers both visual and auditory
measures of attention using targets that the subject is instructed to respond to, as well as non-targeted, to which individuals
are instructed to withhold their responses. During the TOVA, the individual responds to the presence of the target and
refrains from responding during the appearance of an untargeted signal. The first half of TOVA consists of a relatively low
ratio of target and non-target presentations (one target for 3.5 non-target), which is a challenge for those with low attention
span; the second half consists of a relatively high ratio of target and non-target representations (3.5 goals per 1 non-target
object), which is difficult for patients with impulsive and hyperactive behavior. Figure 3 shows one of the stages of the
TOVA test. [9]
36
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
Figure 3 – TOVA test stage
The TOVA test is scored by 5 criteria: errors of omission, errors of commission, response time, response time variability
and ADHD score. An omission error occurs when a person does not respond to the target and is considered a measure of
inattention. A commission error occurs when the subject reacts to a non-target object and is considered a measure of
impulsiveness. Response time measures the delay in response, whereas response time variability measures the consistency
of this delay throughout the study, as well as a variable that accounts for 80% of the difference between patients with and
without ADHD. An ADHD score is also provided and offers a comparison of the examinee’s performance to those in the
standardization sample who were diagnosed with ADHD. The ADHD score is comprised of three of the measures taken in
TOVA tests, expressed as z scores: first half response time, total response time variability, and second half d′ score
(ZResponse Time Half 1 + Zd' Half 2 × -1 + ZResponse Time Variability Total). The d′ Measure is derived from the
Theory of Signal Detection and indicates the sensitivity of the response. TOVA provides a summary of the quarters for
each of the variables listed above, which is very useful for tracking potential changes in performance.
Verification of EPOC+ application
To verify the capabilities of the EMOTIV EPOC+ headset, a series of experiments was conducted with a general type of
research. The study was conducted on a group consisting of 7 healthy subjects, with average age about 23 ± 3. Each of the
subjects agreed for the gathered data processing. Stages of research:
• Functional rest (FR) - 5 minutes;
• The first TOVA test (T1) - 3 minutes;
• Hyperventilation load (HL) - 3 minutes;
• The second TOVA test (T2) - 3 minutes;
• Aftereffect (AE) - 5 minutes.
Figure 4 – the order of research
For the first time, the subjects underwent a study using the Emotiv EPOC+ headset, the second time using the
"Encephalan-EEGR-19/26" mini medical encephalograph. As a verification parameter, the ratio of alpha-rhythm to theta-
rhythm and beta-rhythm to theta-rhythm was used.
3 Results
As a result of the work, a conceptual model of intellectual system was developed that realizes adaptive training process
based on neurobiofeedback with Emotiv EPOC+ headset and the test of variability attention TOVA usage.
37
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
Algorithm and system of NBF
Figure 5 – Block scheme of an adaptive training process using biofeedback
The intelligent system consists of two blocks - the data acquisition unit and the biofeedback loop unit. The data acquisition
unit includes the Emotiv EPOC + headset and the data receiving unit. The block of the biofeedback loop contains a system
for determining the user's mental state, a system for selecting the corrective action and a transmission interface for the
control command.
The system based on the algorithm described above, after receiving the data on the human condition, sends this data to
the computer via wireless Bluetooth interface. After this, the primary data filtering is performed using a bandpass filter
with 4Hz - 45Hz limits for EEG data and 0.3Hz - 20Hz for motion data. An important feature of the system is the usage of
accelerometers in mitigation of EEG artifacts associated with the movement of the head. The next step is to highlight the
key features for each of the signals. After this, a structure containing data about all parameters describing the current state
of the user is sent to the queue.
In the next step, the intelligent processing unit receives the user state structure from the queue and the test results
obtained in the TOVA test. Using all the information, the block determines the user's state. If the user is in the target state,
a supporting feedback system is selected that stabilizes the user's state. In the case of a state mismatch, block determines
the parameters of the corrective feedback, which is aimed at transferring the user to the desired state. This feedback loop
continues throughout the entire session.
As a corrective effect, the breathing control technique is used. This technique assesses the state of the user after
performing the first test, then calculates the breathing parameters necessary to transfer the user to the target state.
38
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
Verification
To verify the Emotiv EPOC+ headset, Bland-Altman method was used. Graphs with comparisons of the alpha-rhythm
power ratio to theta rhythm and beta-rhythm power to the theta rhythm are shown below in Figure 6.
Figure 6 - Results of a comparison of the power ratio on the Bland-Altman graph
As seen in the graphs given above, the values of the power ratios are quite close to each other, which makes it possible
to talk about the possibility of using the Emotiv EPOC+ interface with an accuracy on a par with the professional medical
encephalograph "Encephalan-EEGR-19/26" mini.
4 Conclusion
As result of the work, an intelligent system of adaptive training process based on neurobiofeedback with Emotiv EPOC+
neuroheadset usage was developed. A special feature of this implementation is the usage of accelerometer and TOVA test
to assess the level of cognitive functions and to determine the parameters of the corrective effect of biofeedback.
Verification makes the usage of an Emotiv headset in systems of similar type more plausible and advisable. In the future,
the area of interest represents an increase in the number of modalities, an expansion of the range of possible corrective
actions, functional and performance improvements.
39
Proceedings of Information Technologies, Telecommunications and Control Systems (ITTCS) - 2017
References
1. Y. Liu, X. Hou, and O. Sourina, “Fractal dimension based neurofeedback training to improve cognitive abilities,”
presented at the 2015 7th Computer Science and Electronic Engineering Conference, CEEC 2015 - Conference
Proceedings, 2015, pp. 152–156.
2. Y. Liu, X. Hou, O. Sourina, and O. Bazanova, “Individual Theta/Beta Based Algorithm for Neurofeedback Games to
Improve Cognitive Abilities,” in Transactions on Computational Science XXVI, Springer, Berlin, Heidelberg, 2016,
pp. 57–73.
3. R. Ramirez, M. Palencia-Lefler, S. Giraldo, and Z. Vamvakousis, “Musical neurofeedback for treating depression in
elderly people,” Front. Neurosci., vol. 9, no. OCT, 2015.
4. J. Lockwood and S. Bergin, “A neurofeedback system to promote learner engagement,” ArXiv160706232 Cs, Jul.
2016.
5. A. Ossadtchi, T. Shamaeva, E. Okorokova, V. Moiseeva, and M. A. Lebedev, “Neurofeedback learning modifies the
incidence rate of alpha spindles, but not their duration and amplitude,” Sci. Rep., vol. 7, no. 1, 2017.
6. R. C. deCharms et al., “Control over brain activation and pain learned by using real-time functional MRI,” Proc. Natl.
Acad. Sci. U. S. A., vol. 102, no. 51, pp. 18626–18631, Dec. 2005.
7. M. Mathie, “Monitoring and Interpreting Human Movement Patterns Using a Triaxial Accelerometer,” 2003.
8. I. P. Machado, G. Luísa, H. Gamboa, V. Paixão, and R. M. Costa, “Human activity data discovery from triaxial
accelerometer sensor: Non-supervised learning sensitivity to feature extraction parametrization,” Inf. Process. Manag.,
vol. 51, no. 2, pp. 201–214, 2015.
9. D. P. Hurford et al., “Examination of the Effects of Intelligence on the Test of Variables of Attention for Elementary
Students,” J. Atten. Disord., vol. 21, no. 11, pp. 929–937, 2017.
10. V. Borisov, A. Syskov, V. Tetervak, and V. Kublanov, “Mobile Brain - Computer Interface Application for Mental
Status Evaluation.,” in Proceedings - 2017 International Multi-Conference on Engineering, Computer and Information
Sciences SIBIRCON, Novosibirsk, Russia, 2017.
11. A.M. Syskov, V.I. Borisov, and V.S. Kublanov, "Intelligent Multimodal User Interface for Telemedicine Application".
Jubilee 25th Telecommunications Forum TELFOR 2017. p. 1-4.
40