Human cognitive state recognition is an important and challenging task. Various registration technologies can be used to collect physiological data that potentially contain relevant information regarding current cognitive state of a human subject. Oculography (eye-tracking) and electroencephalography (EEG) are most popular and well-researched registration technologies. Both technologies have cheap commercial variants that do not require laboratory equipment or involvement of professional physiologist to collect the data. However, it is still problematic and expensive to obtain large-scale datasets of physiological data of such sort. In this work a review and analysis of available open source physiological data is provided. A task of natural reading is considered since work of eyes and human brain during reading are of great interest for cognitive science in combination with machine-learning. A multi-modal approach that involves combining EEG and eye-tracking data in jointly trained artificial neural network is proposed. Intermediate results are presented regarding encoding EEG signals with Variational Auto-Encoder (VAE).
This work1 is aimed at neural network architecture development for assessing the cognitive state of a person while working with a text. Term cognitive states, in this work, does not mean the emotional states of a person while performing a certain work, but which parts of the brain and how they respond to certain stimulus, in this case, to the information contained in the text. The first part of this work is devoted to the overview of feature selection approaches and the study of methods of data preprocessing. In this work, the following data will be used: Electroencephalograms (EEG), eye-tracking and vectorized text sets. An EEG is a collection of electrical signals registered on the brain. Generally speaking, EEG is a method for studying the functional state of the brain. Mathematically, EEG data is represented by an M × N matrix, where M is the number of channels in the device that was used to record signals, and N is the length of the record. Eye-Tracking data is the result of registration process of the patient’s eye movement while reading text, for example. A dataset is a set of coordinates where a person’s eyes were focused at a fixed frequency. The next stage of the work is architecture development. Its development will directly depend on what features will be selected and how they will be grouped in the dataset. Feature selection is an important topic because the area of the research is an intersection of neurophysiology and data science, which leads to certain dificulties with feature engineering. In this research not only with mathematical quality metrics, but also with medical ones will be taken in count. Medical metrics mean that at the stages of data processing and preparation, the correspondence of the selected metrics to reality will be checked, and the features themselves will be extracted in accordance with the observations of experts. For example, EEG data have been studied for a long time, and a lot of patterns, how the brain reacts to certain stimulus, are already known and how these patterns look after certain transformations. Later, according to the obtained data, it will be possible to build hypotheses and conclusions, and most importantly, correctly perform batch sampling during training neural-networks. A significant part of research in related topics is focused only on single type of physiological data, either eye-tracking features or electroencephalography features. This work proposes an approach that combines types of features together with semantic information extracted from text. Future development of this work will include training and evaluation of proposed approach on various datasets, including open-source datasets. Apart from deep neural network architecture the work includes the following steps: data preprocessing, feature extraction and feature selection steps. As a final step, a quality metric of the neural network in cognitive state recognition task will be developed. This metric could be presented as hypothesis which requires further test and designing on experiments.
The rest of this paper is organized as follows: Section 2 discusses related works, Section 3 describes the dataset and data preprocessing issues, the approach is proposed in Section 4, and preliminary results are described in Section 5. 2
A significant amount of work had been dedicated to feature extraction, selection and preprocessing. Usually, to solve classification, regression and all other problems with EEG or eye tracking data word-level characteristics. The following features are usually extracted from raw word-level data – number of fixations, mean fixation duration, gaze duration, number of fixations on word, the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word, total reading time (TRT), the sum of all fixation durations on the current word, first fixation duration (FFD), the duration of the first fixation on the prevailing word, go-past time (GPT), the sum of all fixations on the right of the current word.
According to [
Due to expensive data collection process and necessity of involvement of qualified
physiologists, a very limited amount of data is available for research community.
This work focuses on freely available ZuCo and ZuCo-2 datasets [
In the experiment from ZuCo-2 dataset, the participants had to read 739 sentences that were selected from the Wikipedia dataset. The corpus provides semantic annotations of semantically diferent tasks. Normal Reading task consists of a random set of Wikipedia sentences. Following topics had been selected for the task-specific reading part: political afiliation, education, founder, wife/husband, job title, nationality, employer. The sentences have the same length as ZuCo 1.0, with similar semantic. In Normal reading task, the participants had to read 349 sentences, and 390 sentences in a task-specific reading task. Furthermore, there is also an overlap in the sentences between ZuCo 1.0 and ZuCo 2.0. 100 normal reading and 85 task-specific sentences from ZuCo-2 dataset were already recorded in ZuCo 1.0. This provides an opportunity to compare diferent recording procedures (i.e. session-specific efects) and perform studies on larger number of participants (subject-specific efects).
Eye-tracking device captures eye position and pupil size. Records had been made at a sampling rate of 500 Hz with EyeLink 1000 Plus, SR Research device. The eye tracker was calibrated with a 9-point grid at the beginning of the session and re-validated before each block of sentences.
In ZuCo dataset, the following word-level features are extracted from eye tracking data: 1. X, Y coordinates of the fixation 2. Fixation durations 3. Gaze duration 4. Total reading time 5. Number of fixations on this word 6. Pupil size EEG part of the dataset, contains data from 128-channels of raw data with sampling frequency 500 Hz. After the initial filtering and cleaning, that had been made by dataset maintainers, 23 channels has been removed and 105 has left. In the table below, there is a brief description of both task types in ZuCo-2 dataset.
Normal reading (NR): Normal reading was the first task, participants had to read the sentences naturally, without any specific tasks or instructions. Taskspecific reading (TSR) reading was the second, and the final task In task-specific reading participants had to read sentences with a clearly defined topic. For example, it could be political, economical, scientific related texts or questions on the same topics. Participants were instructed to search for a specific relation in each sentence they read, from the list of topics. Instead of comprehension questions, the participants had to decide for each sentence whether it contains the relation or not, also they were actively annotating each sentence. All sentences within one block involved the same relation type. The blocks started with a test round, which described the relation and was followed by three sample sentences, so that the participants would be familiar with the respective relation type. Event-Related Potentials were calculated for both tasks, based on fixation timestamps.
In a proposed approach event is a fixation on word, so the term
fixationrelated-potential (FRP) is used instead of ERP. In EEG, FRP is usually
calculated using window from -600 ms before fixation, and 1 second after it [
X x(t, k) = s(t) +
N k=1
1 XN n(t, k),
N
k=1
(1)
where N is the length of interval, k is event number, t is time passed after the
k-th event, i.e. length of time interval, s(t) is expected value of the signal, and
n(t, k) is noise. Fixation-related potentials are extracted from training data
because they contain significant information about cognitive human reaction on
each word, and in total the information about each sentence is obtained. In
other words, FRP is an averaged incentives in the EEG signal, related to a
certain action in real life. Therefore, information about human cognitive state
during reading each sentence is contained in FRPs, excluding non-informative
data between sentences and words. After extracting FRPs, alpha (8 - 12 Hz),
beta (12 - 30 Hz), gamma (30 - 45 Hz), theta (4 - 8 Hz), delta (0.5 доби– 4
Hz) frequency bands are extracted. [
Eye Tracking Features. Feature selection and preprocessing approaches are
well studied topics in analysis of physiological data. Usually, to solve
classification, regression and other tasks using EEG or eye tracking modalities word-level
characteristics are utilized. The following features are usually extracted from
raw word-level data (EEG and eye-tracking features for every word in corpus)
[
Among all described above features that could be extracted from eye-tracking
data, gaze features are less researched. In the paper [
Electroencephalography Features. A significant part of proposed work is
planned to be focused on EEG data. Due to possibility of simultaneous
registration of EEG and eye-tracking data, a variety of word-level brain activity signal
feature extraction approaches can be utilized. In work [
EEG data can also be used to fine-tune attention mechanisms in NLP tasks.
In [
The purpose of this work is to propose the approach that could be applied to large scale datasets that consist of heterogeneous sequential data. This is the typical case of a neural networks application when it is problematic to apply popular statistical methods to achieve acceptable result and the size of dataset is counted in hundreds of gigabytes. Recurrent Neural Networks(RNNs) are especially suited for sequential data, such as eye-tracking or EEG data. The most popular type of RNN is LSTM, and its modification Bidirectional LSTM (BiLSTM), is usually used to work with eye-tracking data.
Powerful, but heavy and large architectures, such as BERT or XLM could be
used to make predictions, based on eye-tracking data. Nevertheless, these models
must be fine-tuned before using them, while BiLSTM [
In proposed approach, instead of using word embeddings and EEG features
like in [
In [
As it was said above, embeddings of the EEG time series, eye-tracking features
and work-level features will be used as an input batch to the neural-network.
At present, only EEG input features had been prepared. A popular approach
to generate embeddings is using autoencoders. In current work an approach
that utilizes Variational Autoencoder (VAE) [
KL[Q(Z|X, θ 1), P (X|Z, θ 2)] = EZ∼Q [log(P (X|Z, θ 2)] −
KL[Q(Z|X, θ 1), P (Z)], where X is an input data, Z is hidden states, Q(Z|X, θ 1) and P (X|Z, θ 2) are arguments of encoder and decoder functions consequently. In in a proposed approach LSTM layers were used in both encoder and decoder parts of VAE. Input for VAE is defined as following vector:
X = ((α 0, β 0, γ 0, θ 0, δ 0), . . . , (α 105, β 105, γ 105, θ 105, δ 105))T , where α, β, γ, θ, δ are EEG frequency bands respectively. The main criteria in reconstruction - is to minimize reconstruction loss and make time-series generated by VAE realistic. Diference between original and reconstructed time-series. Even with the minimal batch size, it was impossible to train VAE to reconstruct timeseries of the same smoothness as original. VAE had successfully reconstructed all peaks, and learned the structure of the spectre signal. But because reconstructed time series are not as smooth as the original ones, it was impossible to achieve good R2 score. Scores were computed on full time-series, with 128 channels, while on Fig.2 only first 10 channels are shown. The results are presented in table 2.
To make sure that generated embeddings captures infromation from the original data well-enough, VAE reconstruction original data. On the charts below, EEG frequency bands are presented. Left column is the original data, and reconstructed time-series are on the right column. At present only EEG signal emdeddings method had been chosen and testes. Work with eye-tracking features has started, but still in progress. Future development os this work will include generation of embeddings from eye-tracking fetures sequentially predicting next fixation or its probability. Neural network architecture based on the multi-modal embeddings will be trained on available datasets. Main focus will be made on dimensions of multi-modal embeddings to reduce the padding proportion.
Acknowledgement. This work is supervised by Ivan Shanin, Federal Research Center “Computer Science and Control” of Russian Academy of Sciences.