Eye Tracking in the Study of Cognitive Processes

Abstract

Developed computing and software tools for building a nonlinear dynamic model of the human oculomotor system (OMS) based on input-output experiments using test visual stimuli and innovative Eye-tracking technology. The Volterra model is used for identification in the form of multidimensional transition functions of the 1st, 2nd and 3rd orders, taking into account the inertial and nonlinear properties of the OMS. Eye-tracking software developed by Matlab is being tested on real OMS experimental data.

Keywords

• 1 Eye-tracking technology
• oculo-motor system
• cognitive processes
• psycho-physiological states
• Volterra model
• identification

1. Introduction

The study of human eye movements and the trajectory of their movement reveals the structure of the relationship of the individual with the environment, man with the world. Knowledge of eye movement is of great theoretical and applied importance, expanding the possibilities of studying the specifics of many professions in order to increase the efficiency of the subject of labor [1-4].

The process of acquiring knowledge is a central part of the learning process. Management of this process involves the availability of effective objective indicators to assess the intellectual abilities of the individual. Proposed in the project methods of psychological identification of the individual on the basis of experimental data using innovative Eye-tracking technology and computational means of their processing allow to monitor and diagnose the state of cognitive processes during students' learning activities [5-9].

The aim is to develop software tools for building a non-parametric dynamic model of human OMS taking into account its inertial and nonlinear properties based on experimental input-output data using test visual stimulus and innovative eye tracking technology; introduction of the received information models in diagnostic practice of states of cognitive processes.

The identification process is based on the use of test visual stimuli that are displayed on the computer monitor screen at different distances from the starting position (Fig. 1).

Laptop, visual stimulus, eye tracker

Respondent

Responses of the OMS – eyetracker output signals

The developed software enables support of the following tasks: 1. The relationship study of mental states and cognitive processes in educational activities [1014]. 2. The interaction of mental states and cognitive processes during the educational activities of students, an objective assessment of their cognitive development level, assessment of the effectiveness of training to improve mental processes and for psychological correction of personality [2]. 3. Extension of the individual's creative life due to the early diagnosis of degenerative processes of cognitive functions of the brain. Identification of a gifted personality (building a psychological model of the personality) and evaluation of its abilities. Professional selection (the identification and education of leaders) [9]. 4. The assimilation of scientific knowledge and their respective skills serves as the main goal and the main result of educational activities. The process of mastering knowledge is the central part of the learning process. Managing this process implies the existence of effective objective indicators for assessing an individual’s intellectual abilities [7].

The methods of psychological identification of an individual proposed in the project, based on obtaining experimental data using eye tracking technology and computing means of processing them, allow monitoring and diagnostics of the state of cognitive processes during the educational activities of students.

2. Volterra Model and the Method of the Identification OMS

The basis for creating a mathematical (information) model of the object under study is the results of measurements of its input and output variables, and the solution of the identification problem is associated with obtaining experimental data and processing them taking into account measurement noise.

To describe objects of unknown structure, it is advisable to use the most universal nonlinear nonparametric dynamic models - Volterra models [15-17]. The nonlinear and dynamic properties of the object under study are unambiguously described by a sequence of multidimensional weight functions – Volterra kernels, invariant with respect to the type of input signal.

The input-output ratio for nonlinear dynamic systems (NDS) with an unknown structure (such as a "black box") with one input and one output can be represented by a discrete Volterra polynomial of degree N = 3 in the form [18]:

N 3 m m m y[m]   yˆn[m]   w1[k1]x[m  k1]    w2[k1, k2 ]x[m  k1]x[m  k2 ] 

n1 k1 0 k1 0 k2 0 m m m ( 1 )     w3[k1, k2 , k3]x[m  k1]x[m  k2 ]x[m  k3 ],

k1 0 k2 0 k3 0 where yˆ n [m] is the n-th partial component of the NDS model response; w1[k1], w2[k1,k2], w3[k1,k2,k3] – discrete weight functions of the first, second and third orders; x[k], y[k] – input (stimulus) and output (response) functions of the system being modeled, respectively; k is the time variable.

The block diagram of the Volterra model has the form (Fig. 2).

The task of identification is to select test effects x[m] and develop an algorithm that allows the measured reactions y[m] to identify partial components yn[m], (n=1, 2, 3) and determine on their basis Volterra kernels w1[k1], w2[k1,k2], w3[k1,k2,k3] [18].

3. Eye movement tracking to identify OMS

The information technology of construction of non-parametric dynamic model of human OMS taking into account its inertial and nonlinear properties on the basis of data of experimental researches "input-output" is developed. The OMS Volterra model is used in the form of multidimensional transient functions (MTF) [19, 20].

Methods and tools for the identification of OMS have been developed using technology tracking. The obtained MTF are used to construct the space of diagnostic signs and to carry out the optimal classification of neurophysiological states of personality for research in neuroinformatics and computational neurology. Experimental studies of individuals' OMS were performed using the Tobii TX300 eye tracker (frame rate 300 Hz) and the corresponding software in the Laboratory of Motion Analysis and Ergonomics of Interfaces of the Lublin University of Technology (Lublin, Poland) [18].

Taking into account the specifics of the studied OMS, test step signals are used for identification. If the test signal x[m]=θ[m], where mθ[] is a unit function (Heaviside function), then the partial components of the response y1[m], y2[m], y3[m] will be equal to the transient function of the first order h1[m] and diagonal sections of the transient functions of the second and third orders h2[m, m], h3[m, m, m] respectively [18]:

m y1[m]  h1[m]   w1[m  k1], k1 0

m m y2[m]  h2[m, m]    w2[m  k1, m  k2 ], ( 2 ) k1 0 k2 0

m m m y3[m]  h2[m, m, m]     w3[m  k1, m  k2, m  k3].

k1 0 k2 0 k3 0

Determination of subdiagonal intersections of transient functions is based on the NDS test using L test step signals with given amplitudes ai, i=1,2,…,L (L>=N, N is the degree of the Volterra polynomial). In this case the responses of the NDS are denoted by y1[m], y2[m], …,yL[m]. Reviews of the Volterra model will be view ~yi[m]  ai yˆ1[m]  ai2 yˆ2[m]  ai3yˆ3[m], i  1, L ( 3 ) where yˆ1[m]  hˆ1[m], yˆ2[m]  hˆ2[m, m], yˆ3[m]  hˆ3[m, m, m] – obtained estimates of the partial components of the model – MTF.

To determine the transient functions h1[m], h2[m, m], h3[m, m, m] is used the method of least squares (LSM), which provides the minimum standard error of the deviation of the model responses from the responses of the OMS to the same stimulus:

2

L  N  J N    y j [m]   a nj yˆ n [m]  min ( 4 )

j1 n1 

The minimization of criterion ( 4 ) is reduced to solving a system of normal Gaussian equations, which in vector-matrix form can be written as

AAyˆ  A y , ( 5 ) where

A  a2  a1   aL a N   y1[m]   yˆ1[m]  1N    a2 , y   y2[m], yˆ   yˆ2[m]  .         aLN   yL[m]  yˆ N [m]

The system of Gaussian normal equations ( 5 ) produces good results on the approximation of functions if the number of measurements L is large enough (much greater than the degree of the approximating polynomial N) or the measurement errors are small. Otherwise, the determinant of the system turns out to be close to zero and the system becomes ill-conditioned. In this case, large errors in the parameters estimation of the approximating polynomial are possible.

Tikhonov's method of regularization [21], which is based on a variational method for constructing a regularizing operator, is used to obtain a solution of linear algebraic equations system ( 5 ) that is stable against measurement errors. This method is reduced to finding an approximate solution vector yˆ that minimizes certain smoothing functional. The only vector satisfying the condition of the smoothing functional minimum can be determined from the solution of linear algebraic equations system:

(ΑΑ  αI)yˆ α  Α y , ( 6 ) where Α is the transposed matrix; I is the identity matrix;  is the Tikhonov regularization parameter.

When implementing this algorithm, the regularization parameter  is chosen sufficiently small from the analysis of the available information about the error of the initial data and the calculation error. In the work, the appropriate value of the regularization parameter α is determined by selecti i.e. repeated calculations yˆ , for different values of . The quasi-optimal value of the parameter   0 is selected from the condition || yˆαi1  yˆαi || min , ( 7 ) where i 1  i , 0    1, i  0,1,2,... It should be noted that different ways of determining the regularization parameter can give different results and, as a consequence, different regularized solutions.

4. Computing of transient functions OMS

Information technology of the constructing a nonparametric dynamic model of the human OMS taking into account its inertial and nonlinear properties based on the data of experimental studies input-output was developed. As a basic OMS model – the Volterra model is used in the form of multidimensional transient functions.

Methods and tools for the identification of OMS have been developed using the help of eye tracking technology, and building a features space and optimal classification human states using machine learning [22]. In the Laboratory of Motion Analysis and Interface Ergonomics at the Lublin University of Technology (Lublin, Poland), joint studies of the human OMS were performed to obtain diagnostic information for solving urgent problems in the neuro informatics and the computational neuroscience. Experimental research was carried out using eye tracking technology with the use of the video based Tobii TX300 (300 Hz sampling rate) eye tracker and appropriate software [18].

The following instrumental algorithmic and software tools are developed to achieve the goal of the research: 1. Formation of test signals in the form of bright dots on the computer monitor screen at different distances from the initial position horizontally, vertically and diagonally. 2. Preprocessing (bringing the OMS responses to a common start and rationing to one) and analyzing the data obtained from the eye tracker. 3. Constructing an identification model of OMS in the form of multidimensional transient functions (integral transformations of Volterra kernels).

4. Visualization of data and processing results of experimental research. 4.1.

Experimental research of the OMS

When conducting experimental studies, such actions are carried out: 1. The test subject is placed in front of the computer so that his eyes are at the center of the monitor at a distance of 40-50 cm from him. 2. The subject’s head is fixed in order to prevent its the same experimental conditions. 3. On the subject’s readiness, the Signal Manager of the testsitmuvluissuaplrogram is launched. 4. A red circle appears in the center (or from its edge) – of the screen in the starting position. 5. After a short pause (2-3 sec.), the circle in the starting position disappears and a circle of a different color appears at the point with the specified coordinates – a visual stimulus (test signal), which is displayed in this position for a specified duration (1-2 sec.) – the action makes the eye move in the direction of the visual stimulus. 6. Then this stimulus circle disappears and a red circle appears in the starting position – this makes the eye move in the opposite direction to the starting position, after these actions the experiment ends. 7. Using the eye tracker, the coordinates of the pupil of the eye are determined during its movement (reaction to the visual stimulus) in the period between the starting positions and the coordinate values are stored in the xls-file.

In the studies of each respondent, three experiments were successively implemented for three amplitudes of test signals in the horizontal direction. The distance between the starting position and the test incentives is equal to: 0.33 lx, 0.66 lx, 1.0 lx, where lx is the length of the monitor screen. Coordinates of the starting position (x = 0, y = 0.5 ly), ly – mean the width of the monitor screen. movements during the study and to en

5. Research results

The experiments were organized in order to classify subjects by the state of fatigue. The data for constructing the model – the OMS responses to the same test signals, were obtained using the Tobii Pro TX300 eye tracker at different times of the day: "In the Morning" (before work) and "In the Evening" (after work).

In Fig. 3 and Fig. 4 presents graphs of experimental data at different amplitudes of the test signals "Morning" and "Evening", received from the eye tracker.

The average values of the OMS responses obtained from the eye tracker at various amplitudes of the test signals "In the Morning" and "In the Evening" are shown in Fig. 5.

Graphs of transition functions for the states of the respondent "In the Morning" and "In the Evening" at N = 1 are presented in Fig. 6, at N = 2 - in Fig. 7.

According to averaged data of OMS responses on visual stimuli with a different distance from the start position on the basis of formula ( 5 ) the functions of the OMS were defined when approximation models of degrees N = 3 were used. Graphs of the transient functions estimates for the "In the Morning" and "In the Evening" states of the subject based on model ( 1 ) are shown in Fig. 8.

Received responses with the help of calculations on models at N = 1, N = 2 and N = 3 from various amplitudes of test signals "In the Morning". Graphs these are presented in comparison with similar responses OMS in figures 9-11. Graphs of responses of the model OMS at N = 3 at various amplitudes of the test signals "In the Morning" and “In the Evening” are illu2s.trated in Fig. 1

Deviation of the transient functions

The variability (deviation) of the MTF of different orders n of the approximation model of OMS for the states of the respondent "In the Morning" and "In the Evening" is quantified using the indicator of nεN – normalized standard deviation ( 8 ). The indicators deviation of the MTF of different orders n of the OMS approximation model for respondent states "In the Morning" and "In the Evening" are given in Table 1 and are represented by diagram in Fig. 13.

As can be seen from Fig. 13, the obtained transient function of the 1st order for the "In the Morning" and "In the Evening" are virtually independent of the status of the subject. However, the diagonal cross section of the transient functions of the second and third order change significantly in magnitude and, therefore, can be effectively used as the primary data source when building models of classifiers of psychophysiological conditions of the person using machine learning. 5.2.

Building a classifier of the fatigue

To assess the psycho-physiological state of a person based on the OMS model, studies were carried out: 1. Getting a feature space for building a human state classifier using machine learning. 2. Building classifiers using deterministic and statistical learning methods for pattern recognition based on the data obtained using eye tracking technology.

The discriminant function d(x) is sequentially calculated on the basis of training datasets for object classes A ("In the Morning"), B ("In the Evening").

Gaussian classifier is built for separate the two classes (dichotomy case) a discriminant function of the form is used: d (x)  1 x(S21  S11 )x  (S11m1  S21m2 )x  1 (m1S11m1  m2S21m2  ln | S2 |)  λ max ( 9 ) 2 2 | S1 | where x=(x1,x2,…,xn)' – features vector, n – features space dimensionality, mi – mathematical expectation vector for a features of class i, i=1, 2; Si=M[(x-mi)(x-mi)'] – covariance matrix for class i (M[] – mathematical expectation operation). Si1 – matrix inverse to Si, |Si| – matrix determinant Si, max – classification threshold providing the highest criterion probability of correct recognition training sample objects.

The informativeness of various features was investigated, such as integral of the transient functions (Table 2), the argument and value at maximum derivative of the transient functions (Table 3), the argument and value at minimum derivative of the transient functions (Table 4), the argument and value at the maximum transient response (Table 5). The analysis of the quality of various features combination is carried out on the basis of the criterion probability of correct recognition (PCR, P). The quality of the combination of the selected features from the considered set of features is assessed based on the classification results on the studied data sample.

Formal definition

M x1   h1 (m) m0

M x2   h2 (m, m) m0

M x3   h3 (m, m, m)

m0 Formal definition x4  max h1' (m)

m [0,M ] x5  arg max h1' (m)

m [0,M ] x6  max h2' (m, m)

m [0,M ] x7  arg max h2' (m, m)

m [0,M ] x8  max h3' (m, m, m)

m [0,M ] x9  arg max h3' (m, m, m)

m [0,M ] Formal definition x10  min h1' (m)

m [0,M ] x11  arg min h1' (m)

m [0,M ] x12  min h2' (m, m)

m [0,M ] x13  arg min h2' (m, m)

m [0,M ] x14  min h3' (m, m, m)

m [0,M ] x15  arg min h3' (m, m, m) m [0,M ]

Gaussian classifier of a person’s fatigue state in-dima entswional feature space is provided with the maximum recognition reliability (P = 0.9375) with combinations of the following features: or or or or or  M   x3   h3 (m, m, m) & x12  min h2' (m, m)  ,  m0 m [0,M ]   M   x3   h3 (m, m, m) & x14  min h3' (m, m, m)  ,  m0 m [0,M ]     x8  max h3' (m, m, m) & x6  max h2' (m, m)  ,  m [0,M ] m [0,M ]     x9  arg max h3' (m, m, m) & x6  max h2' (m, m)  ,  m [0,M ] m [0,M ]     x13  arg min h2' (m, m) & x6  max h2' (m, m)  ,  m [0,M ] m [0,M ]  ( 10 ) (11) (12) (13) (14)    x14  min h3' (m, m, m) & x12  min h2' (m, m)  . (15)  m [0,M ] m [0,M ] 

Separately, the PCR features: x9 or x13 – P = 0,625; x3 or x8 – P = 0,6875; x12 – P = 0,75; x6 or x14 – P = 0,8125.

6. Conclusion

Instrumental algorithmic and software tools for building a nonparametric dynamic model of the human oculomotor system taking into account its inertial and nonlinear properties based on the data of experimental studies "input-output" using technology tracking are developed. The Volterra model is used in the form of multidimensional transition functions.

The following tasks are solved: 1. The application of the OMS identification method based on the Volterra model in the form of multidimensional transition functions using test visual stimuli with different distances from the starting position - step functions of different amplitude is substantiated.

2. The information technology of obtaining experimental data for the identification of OMS with the help of test visual stimuli and the use of eye tracking to track the corresponding eye movements has been developed.

3. Developed in the Matlab system software for the identification of OMS based on Volterra polynomials in the form of multidimensional transient functions according to eye tracking.

4. Experimental studies of OMS were performed with the help of eye tracking technology and the transitional functions of the first, second and third orders were determined on the basis of oculographic studies. Studies of local self-government with the help of the obtained transient functions by means of computer modeling confirm the adequacy of the constructed approximation model to the real system.

5. Classifiers of human cognitive states are built on the basis of the studied heuristic features that are resistant to computational errors. The features were calculated on multidimensional transient functions obtained from the integral Volterra models of human OMS within a new approach to diagnosing human conditions.

The analysis of variability of transient functions corresponding to different psychophysiological states of the individual (states of fatigue) is carried out. It is established that the diagonal intersections of the transient functions of the second and third order with respect to the transient functions of the first order for fatigue states change significantly in magnitude. Thus, they can be used as a data source in the formation of spaces of diagnostic features for the construction of classifiers of human psychophysiological states.

7. References

model in sport: A revie of Sport and Exercise

References

[1] Y. Wamain , X. Corveleyn , L. Ott , Y. Coello , “ Does the motor system contribute to the perception of changes in objects visual attributes? The neural dynamics of sensory binding by action , ” Neuropsychologia , 132 , 201190 .7121.doi.org/10.1016/j.neuropsychologia. 2019 .107121

[2] M. Daoudi , M. Coello , P. Desrosiers , L. Ott, “ A New Computational Approach to Identif Human Social Intention in Action,” In the 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018 , Xi'an , China on May 15-19 . ( 2018 ).

[3] F. Quesque , A. Mignon , Y. Coello , “ Cooperative and competitive contexts do not modify th effect of social intention on motor action . Consciousness and Cognition. doi: 10 .1016/j.concog,” 2017 , 06 . 011 .

[4] L. Lanata , Sebastian , F. Di Gruttola , S. Di Modica , E.P. Scilingo and A. Greco1 “Nonlinea Analysis of Eye-Tracking Information for Motor Imagery Assessments . Frontiers in Neuroscience”, 20201 ,3:1431 DOI: 10.3389/fnins. 2019 . 01431 .

[5] N. Gueugneau , L. Crognier , and C. Papaxanthis , “ The influence of eye movements on the temporal features of executed and imagined arm movements , ” Brain Research , 1187 , 2008 , pp. 95 - 102 .

[6] A. Guillot , C. Collet, “ Duration of mentally simulated m:ovAemreenvtiew,” Journal of Motor Behaviour , 2005 , 37 , pp. 10 - 20 .

[7] A. Guillot , C. Collet, “ Construction of the motor imagery integrative and theoretical investigation of motor imagery use ,” International Review Psychology, 1 , 2008 , pp. 32 - 44 .

[8] A. Guillot , C. Collet , V. A. Nguyen , F. Malouin , C. Richards , and J. Doyon , “ Functiona neuroanatomical networks associated with expertise in motor imagery ability ,” Neuroimage, 41 , 2008 , pp. 1471 - 1483 .

[9] A. Guillot , M. Louis , and C. Collet , “ Neurophysiological substrates of motor imagery ability,” In A . Guillot and C. Collet (Eds). The neurophysiological foundations of mental and motor imagery , 2010 , pp. 109 - 124 , Oxford University Press.

[10] M. El Haj , Y. Coello , D. Kapogiannis , K. Gallouj , P. Antoine, “ Negative prospective memory in Alzheimer's Disease: do not perform that action” , Journal of Alzheimer's Disease , 2018 , 61 ( 2 ), pp. 663 - 672 , doi: 10.3233/JAD-170807.