The analysis of human emotion recognition methods and algorithms have been conducted. The new approach for emotion recognition based on the combined set of data from biosignals and visual features is proposed in the paper. In order to solve the task of emotion recognition, it is proposed an architecture of a system that would automatically read data, process them, build models and train them, monitor experiments and provide the user with a Web service based on the most accurate model studied. It was decided to continue the planned research on DEAP dataset since it was the most suitable for the described idea and proper data filling. Since the experiment includes the two totally different sources of data - biosignals and facial videos - the two data pipelines should be designed respectively. After analyzing the prediction distribution for all models, it was found that GRU architectures do best in the field of values, compared to CNN and FNN, which studied the mean. In order to define which filtering coefficients are best suited for this scientific work the automatic search algorithm is suggested. The elaborated algorithm for solving this roadmap is based on Deep reinforcement learning and is successfully demonstrated in solving similar tasks - Neural architecture search. emotion recognition, convolution neural network, dataset, video pipeline, biosignal pipeline, (ORCID: 0000-0003-0133-8591 (V. Yakovyna); 0000-0002-9585-3078 (V. Khavalko); 0000-0002-0563-5748 (A.Boichuk), 0000-0003-
neural network, CNN-based architecture.
Not so long ago the idea for a machine to understand the human thoughts could be comprehended
as total nonsense. Computer-brain interfaces, BrainNet, deep interactive gaming are fields that are
thought to be fictional but at the same time could benefit and move human society to the next level of
development. Digital advertising, marketing, one-on-one interviews are the technologies we are already
using though they could be greatly improved by applying the described idea. After thoroughly
researching the computer-brain interfaces, its versions, the labs and scientists are developing in the
moment of writing this thesis the glimpses of new technologies such as EEG-to-speech, EEG-to-devices
(mental typing) are already on the horizon [
By leveraging a couple of brain-wave detectors and complex algorithms, it’s gradually becoming possible to analyze brain signals and extract reasonable brain patterns. Brain activity, such as neurons and synapses cooperation, can then be recorded by a non-invasive device, so that no surgical intervention is needed. In fact, most of the developed prototypes and mainstream BCIs are non-invasive. Generally, they are contained inside the wearable headbands and earbuds. Regarding the invasive approach, over the last years a specific type of BCI gained attention - a model that utilizes a grid of electrodes implanted directly into the motor cortex and neighboring areas. In this context, motor
2021 Copyright for this paper by its authors. imagery is used as an intuitive and natural strategy to elicit brain activity changes and subsequently to control movements of a robotic arm in real-time.
With the technologies on the horizon that gives the opportunity for much more accurate brain-data extraction the research can be moved to the following stage - emotions and its understanding. The emotions are considered the vital states of the human being and play a dramatic role in its lifecycle, commonly being emphasized in theoretical research as a mechanism of consciousness. The questions about cognition, conscience, philosophy, human nature rise more frequently making the research of emotions a first step of understanding how to answer them.
Besides making a contribution to global topics, the research of emotions can influence people's lives on the baseline level. As it is known emotions affect organisms not only on the psychical level but on the physiological as well. An abundance of positive emotions improves a person's health and work efficiency. On the other hand, negative emotions are one of the main reasons of depression which is the widely spread cause of suicide if being neglected.
For emotion recognition, the emotions should be defined and evaluated quantitatively. The sole definition of initial emotions was first proposed decades ago. However, the precise definition has never been widely acknowledged by psychologists. They tend to assess emotions with two different approaches. One is to split the emotions into separate groups or classes. Another one is to use multidimensional labels. For emotion elicitation, subjects are given a series of emotionally-evocative materials to induce a certain emotion. For the past few years, entertainment stimulations are the most common product. Besides, some new methods called situational stimulation are rising in recent years.
Today, there are more and more categories of signs and data that can be used to teach a machine to recognize a person's emotional state. The following categories should be noted: visual - includes images or videos of the observed person, biometric signals - EMG, EOG, SLC and EEG, textual - contain the semantic context, sound - the intonation of speech. Particular attention should be paid to combined data types, as they significantly increase the area from which the machine learning model learns.
It has been repeatedly confirmed that for the task of recognizing emotions, visual data can be forged,
so they cannot serve as the only category to rely on. On the other hand, most information a person
perceives with his eyes. Thus, visual data is the most familiar to us, which allows us to quickly assess
the emotional state of the interlocutor [
Considering biosignals, and especially EEG, it is assumed that they are almost impossible to falsify, as some of the processes that emit a brain signal are subconscious and different for each individual. Compared to visual signs, a person cannot determine the emotions of another with the help of a biosignal.
After analyzing both types of data, we can conclude that an effective solution is a combination. Based on the visual component, the process of data creation is accelerated, involving the characteristics on which the living organism rests in the system. Biosignals will complement it by filtering out incorrect operation and improving overall accuracy. This system will be considered in this scientific work.
In order to solve the task of emotion recognition, it is necessary to build a system that would automatically read data, process them, build models and train them, monitor experiments and provide the user with a Web service based on the most accurate model studied. The combined set of data from biosignals and visual features will be taken as a basis. The development of the system and the conduct of research will not only lead to the acquisition of a product for the purpose of determining the emotional state, but will also be able to be adapted to related goals, most of which include a computer-brain interface. It is a system and a set of studies that cover the fields of medicine, education, entertainment.
This is especially true for the diagnosis and treatment of diseases of the brain, nervous system patients suffering from paralysis, epilepsy, Alzheimer's symptom.
Depression or anxiety is another diagnosis that falls into the mentioned group. Itcan be explained by the heterogeneous mood that prevents to enjoy most activities. This symptom is also accompanied by other ones, such as irritability, anxiety, inability to conduct problems.
For the last decade, a lot of discussions appeared around cognitive performance improvement enhancers. Coffee or tea is the most commonly mentioned stimulator. However, the debate has gained importance about cognitive enhancement, which includes prescription drugs like Modafinil and Ritalinby both professional workers and students.
The invention of BCIs is also considered an approach to boost the cognitive functions of healthy users. Neurofeedback training (brain activity alteration through operant conditioning), for instance, to improve attention, long and short working memory, critical thinking functions is common among the average healthy user. Another application area is optimized content learning. Although there is a lack of good research-proven data on its effects, the volume is probably small and limited to specific cognitive tasks. Generally speaking, there may be a thin line between non-medical and medical use of neurofeedback.
To address the described challenge the proper dataset is needed, the one with enough data the complex models to be empirically built on. After a broad data exploration for such datasets as DECAF, MAHNOB, etc, it was decided to continue the planned research on DEAP dataset since it was the most suitable for the described idea and proper data filling.
DEAP dataset is publicly available as a state-of-art dataset for visual and biosignal emotion recognition. The dataset was assembled, processed and generated by a R&D team at the Queen Mary University of London. The DEAP dataset consists of multiple physiological signal types and face video records for the evaluation of emotions. 32 channel EEG data were collected from 32 volunteers. The face records and biosignals were recorded by showing 40 preselected music videos which varied in its topic to boost the emotional engagement, each with a duration of 60 seconds (Table 1). The signals were downsampled to 128 Hz and denoised via the bandpass and lowpass frequency filters.
Chanel description
Fp1(EEG) AF3(EEG) F3(EEG) F7(EEG) FC5(EEG) FC1(EEG) C3(EEG) T7(EEG) CP5(EEG) CP1(EEG) P3(EEG) P7(EEG)
Low pass(Hz)
High pass(Hz) 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0
PO3(EEG) O1(EEG) Oz(EEG) Pz(EEG) Fp2(EEG) AF4(EEG) Fz(EEG) F4(EEG) F8(EEG) FC6(EEG) FC2(EEG) Cz(EEG) C4(EEG) T8(EEG) CP6(EEG) CP2(EEG) P4(EEG) P8(EEG) PO4(EEG)
O2(EEG) hEOG (horizontal EOG, hEOG1 - hEOG2)
vEOG (vertical EOG, vEOG1 - vEOG2) zEMG (Zygomaticus Major EMG, zEMG1 - zEMG2)
tEMG (Trapezius EMG, tEMG1 - tEMG2) GSR (values from Twente converted to Geneva format
(Ohm)) Respiration belt Plethysmograph
Temperature 4.2.
0.25 1.0 0.2 0.5 4.5 3.5 0.8 3.0
To make a move toward modeling an AI system the data digestible by backpropagation algorithm is required, since it is vulnerable to unstandardized, missing, sparse values. The data processing pipeline serves as a tool to complete these requirements and secures the correct data format for the training session. Since the experiment includes the two totally different sources of data - biosignals and facial videos - the two data pipelines should be designed respectively.
The video pipeline consists of the following steps: key frame extraction, frame standardization, transition to model’s input format.
The biosignal pipeline, on the other hand, is composed of the bandpass filtering via the empirical definition of the required parameters, outliers handling, standardization.
Both of the mentioned pipelines can also be integrated into the third one - the one that synchronizes the video and signal time-series data frames into a single multi-input model format.
CNN-based signal model - an approach based on stacking all signals vectors on a single frame. It is
possible since the length of the signal vector is constant. For the extraction of this frame, 2D CNNs
were used [
The next modification that was proposed is to add a bottleneck to model architecture in order to preserve kernel forgetting the difference between distinct channels. Bottleneck was identified by CNN with a kernel to wrap the share of channels and a steady step to simulate working with images that CNN specializes in. Another benefit of considering CNN is its parameter size efficiency. Despite being multiple times smaller in complexity compared to feedforward models, CNN produces greated results with the less time needed for the inference computation and training. 4.3.
The main concept was to preserve the long memory values without resetting it anew. This is the
reason GRU was suggested to implement since the GRU layer does not have the forget gates. It also is
trained faster compared to traditional Bidirectional LSTM which gave more computational time to tune
the parameters [
Emotion recognition for images gained the new benchmark horizons over the last decade since the
rise of the convolutional neural network. It is especially robust for the examples that represent the far
extremes of the emotional space as they are more definite and lack the ambiguosity. Speaking about the
ones that are located at the center of 3D or 4D emotional space, they are pruned to be classified as
neutral. Another scenario where the mentioned model type fails is the recognition of concealed
emotions. To deal with such challenges the video-based model is proposed. It will not only derive the
information from the image data but also captures the context over the defined timeframe. Compared
to doing the recognition on images it also helps to make the smoother prediction over time omitting the
accidentally-captured frames that can harm the performance [
The initial goal was to fuse the biosignal channels with video frames, preserving their variance. Also
the video frame theoretically can add more context to the data to learn from. It could help to solve the
obstacles the previous deep learning model faced such as emotion counseling and emotionless
expression. Combining those two methods, the system potentially gains more saturated features to
process and more parameters to be shared between two parallel flows. After completing the parameter
optimization, the video and single flows can be separated and used as pre-trained models. The benefit
of applying such a technique is that the visual model shares the weights with signal one and it produces
the knowledge base that cannot be obtained on the sole visual data alone. The same goes vice-versa for
the signal data as well [
The pipeline for building architecture with such capabilities is: multichannel extraction, video processing, fusion module, multiple deep layers.
In order to obtain the reliable data that fully covers the performance and blind spots of the system flow the two consequent approaches were designed to be tested on.
For the initial AI part testing the following metrics are required to assessed: • • • •
Categorical Accuracy; MAE; MSE;
F-score.
Based on the defined metrics we concluded that the CNN-based architecture has worked best for signals. This is due to the complexity of CNN, which includes fewer parameters than competitors and better matches on datasets with a small number of characteristics (features).
As for the video, the pre-trained model showed better results than the one that started learning from random parameters. This case is refuted by the homogeneity of the frames in the video. On the other hand, the descriptors of the pre-trained model were optimized on 1281167 unique images, which allowed better CNN-Inception extractors convergence to highlight the main characteristics on the basis of which you can make predictions.
The combined model between GRU signals and the pre-trained video model performed best in the Categorical Accuracy metric, but lost to MAE in Signal CNN.
Another approach that was suggested is based on the predicted labels’ distribution analysis. It allows
to detect the situations when the constructed model is not able to fully optimize the parameter for robust
workflow. In the worst-case scenario, the empirically derived model can derive only the mean value of
the whole test dataset. This serves as a major cause of underfitting [
CRNN with pretrained ResNet descriptors - Compared to the CRNN model, which trained using random weights in the beginning, this model focused on 3 points instead of 1. In addition, they are uniform.
The combined model made it possible to study the merged distribution of the previously described models. Similar to the pre-trained CRNN model, the concentration of values is in 2 main classes. As for the GRU for signals, its application gave a smoothing effect to the distribution.
To sum up the results of both of the predefined approaches, the CRNN with pretrained ResNet descriptors showed the most promising insights in understanding of the emotion-space mapping.
The most important predictors are given in Table 2.
After analyzing the prediction distribution for all models, it was found that GRU architectures do best in the field of values, compared to CNN and FNN, which studied the mean. It should be noted that the number of GRU neurons 256 and 128, respectively (Fig. 1). This decision was made due to the inability of the model to optimize the larger layers 512, 256, due to the small number of informative characteristics in the bioset signalset.
CRNN with pre-workouts ResNet101 weights Fig. 2 showed better results compared to the competing model, namely MSE 2.38 and 3.15. This difference is due to insufficient data to study the internal CNN extractors of this architecture. Although 32 1-minute videos were used for each of the 22 people, the difference in information between the processed frames is insignificant, which is a simulation of the upsampling process, which is based on duplicating records to balance classes in a dataset while storing variance.) and standard deviation (std).
Emotion recognition surely is one of the essential tasks we need to solve before declaring the
understanding of the human body and its brain. It plays a major role in the solution that can drastically
increase the level of life for a common person. Biotechnology, BCI, depression handling - the tools that
can be seen on the horizon after making progress in this field [
This paper describes the study of multichannel signals and systems for preparing for the integration with the visual data. Visual emotion recognition was for a while in the market and showed promising results. On the other hand signals differed in their origin and sources. The techniques and methods to extract the positive information from them are also unique, especially the complexity is rising as we dive deeper into the brain. Approaches to the evaluation information flow of the channel to the classification of modules were demonstrated. An effective, efficient and interactive means of model capacity, typical extraction functions, construction of hybrid models were also demonstrated.
The CRNN with pretrained ResNet descriptors demonstrated the most promising results from all the conducted experiments. The downside of this solution is its computation requirements since it is built on two complex neural networks. On the other hand, if one recalls the duration of the data records, it makes clear that the real-time processing is not needed and the best-performing model is fully covering the needed trafic of requests for inferencing. In the worst-case scenario, the performance problem can be transitioned to the infrastructure scaling without changing the state of the deployed model. Since the system is far from being though the process of comparing it to ground truth labels, more modifications are proposed to be designed and experimented. It is also worth mentioning that the ground truth labels cannot be considered 100% accurate since the emotions were assessed by the humans and the emotions itself are the subjectictive not objective measurement. As it was mentioned the signal filtering stage was an essential part of the model's training cycle.
In order to define which filtering coefficients are best suited for this scientific work the automatic search algorithm is suggested. In a conjunction with automated model training cycle, both of those automatic approaches can solve two tasks simultaneously: • Filtering parameters search for biosignals, • Emotion recognition model optimization, • Emotion recognition model hyperparameter tuning.
The type of algorithm for solving this roadmap is based on Deep reinforcement learning and is successfully demonstrated in solving similar tasks - Neural architecture search. Though the proposed method required a tremendous amount of computational power since thousands of signal recognition models are needed to be trained repeatedly. Figure 2: Architecture visualization for GRU model based on signals 7. References Processing of Social Networks (COAPSN-2020), Lviv, Ukraine, May 21, 2020. p. 342352. http://ceur-ws.org/Vol-2616/paper29.pdf [22] Wosiak A., et al. Optimisation of the cooling unit in the system for supervising the condition of large power transformers. Przeglad Elektrotechniczny, 2009, 85.12: 166-169. [23] Lipinski P., Yatsymirskyy M. Efficient 1D and 2D Daubechies wavelet transforms with application to signal processing. In: International Conference on Adaptive and Natural Computing Algorithms.
Springer, Berlin, Heidelberg, 2007. p. 391-398. [24] Stolarek J., Lipiński P. Improving watermark resistance against removal attacks using orthogonal wavelet adaptation. In: International Conference on Current Trends in Theory and Practice of Computer Science. Springer, Berlin, Heidelberg, 2012. p. 588-599. [25] Lipinski P. On domain selection for additive, blind image watermarking. Bulletin of the Polish
Academy of Sciences. Technical Sciences, 2012, 60.2: 317-321. [26] Lipinski P., Yatsymirskyy M. On synthesis of 4-tap and 6-tap reversible wavelet filters. Przegląd
Elektrotechniczny, 2008, 84.12: 284-286. [27] Glonek G., Wojciechowski A. Hybrid orientation based human limbs motion tracking method.
Sensors, 2017, 17.12: 2857. [28] R. G. Alakbarov, “Method for Effective Use of Cloudlet Network Resources,” IJCNIS, vol. 12, no. 5, pp. 46–55, Oct. 2020, doi: 10.5815/ijcnis.2020.05.04. [29] Md. R. Ahmed, T. Islam Robin, and A. Ali Shafin, “Automatic Environmental Sound Recognition (AESR) Using Convolutional Neural Network,” IJMECS, vol. 12, no. 5, pp. 41–54, Oct. 2020, doi: 10.5815/ijmecs.2020.05.04. [30] Opałka S.et al. Multi-Channel Convolutional Neural Networks Architecture Feeding for Effective Eeg Mental Tasks Classification. Sensors, 2018, 18.10: 3451.