=Paper=
{{Paper
|id=Vol-1492/Paper_19
|storemode=property
|title=Toward Synchronization of EEG and Eye-tracking Data Using an Expert System
|pdfUrl=https://ceur-ws.org/Vol-1492/Paper_19.pdf
|volume=Vol-1492
|dblpUrl=https://dblp.org/rec/conf/csp/JaskulaPS15
}}
==Toward Synchronization of EEG and Eye-tracking Data Using an Expert System==
https://ceur-ws.org/Vol-1492/Paper_19.pdf
Toward Synchronization of EEG and Eye-tracking Data
Using an Expert System
Extended Abstract
Boleslaw Jaskula1 , Krzysztof Pancerz1,2 , and Jaroslaw Szkola1
1
University of Information Technology and Management
Sucharskiego Str. 2, 35-225 Rzeszow, Poland
{bjaskula,jszkola}@wsiz.rzeszow.pl
2
University of Management and Administration
Akademicka Str. 4, 22-400 Zamosc, Poland
kpancerz@wszia.edu.pl
Abstract. In the paper, we present a new approach to solving the problem of
combining data coming from eye-tracking and electroencephalography (EEG).
This is a challenging problem in neurocognitive research. An important issue
added to the process of combining data received from both devices is their syn-
chronization. The created software system solving the problem is based on a spe-
cialized expert system.
Key words: electroencephalography, EEG, eye-tracking, expert system, data
synchronization
1 Synchronization of EEG and Eye-tracking Data
Combining eye-tracking and electroencephalography (EEG) is one of the most fasci-
nating and challenging problems. Scientists are dealing with this problem in order to
close the gap between the psychological mental processes leading to the sensations we
experience in everyday life and their underlying physiological biochemical processes.
They are trying to understand perception, the process by which our brain makes sense
out of the signals coming from our senses. Eye-tracking alone is not sufficient because it
delivers to us only psychophysical data. It provides valuable information about the gaze
location, but does not provide any information about neuronal activity. On the other
hand, EEG, measuring neuronal activity in humans, does not directly provide informa-
tion about the gaze position. For this reason, projects combining EEG and eye-tracking
into one common setup have been started by scientists (see [6]).
Electroencephalography (EEG) is a well-established non-invasive technique for
brain monitoring with high temporal resolution and relatively low cost. As such, EEG
has proven to be a critical monitoring and diagnostic tool in the clinic [11]. EEG is
also a popular research tool among scientists for evaluating somatosensory responses
to stimuli, error detection [4], and sleep or fatigue monitoring [3], [12], among other
uses. Various EEG components in the temporal domain have been used to define dis-
tinct phases of cortical processing in response to stimulus presentation. Eye-tracking
197
is the technology offering the possibility of capturing visual behavior in real-time and
monitoring locations of fixations within images [7]. Recently, eye tracking technology
has become more accurate and user friendly. It has extended to various areas that led to
a wide range of applications, cf. [5], [8].
There are various areas in which combining eye-tracking and EEG is used. First of
all, we can distinguish the following ones:
– visual search which involves finding a target in the midst of distractors (see, for
example, [9], [13]),
– consumer neuroscience which is an emerging interdisciplinary field that combines
psychology, neuroscience, and economics to study how the brain is physiologically
affected by advertising and marketing strategies (see, for example, [1], [2]),
– data collection [10].
There is a need to integrate two methods of gathering data, mentioned earlier, for
neurocognitive research. The main problem arising in combining eye-tracking and EEG
data is their synchronization. Until now, there is a lack of information in the literature
about the fulfilling solution of the problem. In our approach, we propose to use a spe-
cialized expert system.
For the research purposes, we have developed a software system whose task is
to synchronize data received from EyeTracker Tobii T60 and Emotiv EPOC EEG
(Portable version 1.0). The main problem arising in this task is that both devices work
independently and we can only capture the data separately and next save them to files. In
case of Tobii T60, the software is a black-box and we have access only to the recorded
data. Therefore, these data can be analyzed in further stages. In case of EPOC EEG, the
API (Application Programming Interface) is available. It allows us to prepare a dedi-
cated software system that uses the signals obtained from the sensors.
The following assumptions have been made in order to fulfil requirements for the
dedicated software system:
1. The measurement data must be synchronized in time.
2. The devices can be connected to two different machines at the same time (recom-
mended).
3. Measurements should have a high time resolution.
4. An expert system analyzes the received data searching for the selected features
(correlations, patterns, etc.).
To achieve these goals, we have proposed the following solution. Each device must
have a system clock synchronized with the selected time standard. In our research, we
have used a solution based on time synchronization with the atomic clock. The freeware
Atomic Clock Sync v.3.5 has been used. One can also run a local time server to synchro-
nize the two devices. If we run two applications on a single machine, synchronization
is not necessary. However, our experience shows that this is a data-intensive computing
job and there are situations when records are not synchronized. Another advantage of
execution of tasks on different machines is the possibility for running applications un-
der different operating systems. As we have access to the API of EPOC EEG, we are
able to get time resolution between the recorded samples with the accuracy no less than
198
0.1 also with respect to data received from Tobii T60. An expert system analyzes the
synchronized data received from two different devices, searching for selected features
of the signal. An additional advantage of this approach is the ability to export both raw
and processed data in the CSV format, which allows us to perform analyses using other
specialized tools.
Gathering synchronized data is necessary but not sufficient for performing the rea-
soning process. An important problem, with respect to the character of the measured
signals, is their quality as well as noise level. In case of the eye-tracker signal, the pre-
cision is high and the measurement is stable. Meanwhile, in case of the EEG signal,
data are noised and they can include signals from the surroundings. The level of the
EEG signal depends on the region in the brain.
The expert system consists of three main parts:
– a signal filtering part,
– an activity pattern recognition part,
– a reasoning part based on the first order logic.
An important novelty of the proposed approach is the application of information on the
optic nerve gathered from the eye-tracker for removing artefacts. This effect is difficult
to obtain using only the EEG measurement.
References
1. What is ’neuromarketing’? A discussion and agenda for future research. International Journal
of Psychophysiology 63(2), 199–204 (2007)
2. Neuromarketing: the next step in market research? Eureka 1(1), 34–42 (2010)
3. Colrain, I.: Sleep and the brain. Neuropsychological Review 21(1), 1–4 (2011)
4. Davidson, P., Jones, R., Peiris, M.: EEG-based lapse detection with high temporal resolution.
IEEE Transactions on Biomedical Engineering 54(5), 832–839 (2007)
5. Duchowski, A.T.: A breadth-first survey of eye-tracking applications. Behavior Research
Methods, Instruments & Computers 34(4), 455–470 (2002)
6. Görgen, K.: Combining Eyetracking and EEG, Publications of the Institute of Cognitive
Science (PICS), vol. 15-2010. Osnabrück, Germany (2010)
7. Hansen, D., Ji, Q.: In the eye of the beholder: A survey of models for eyes and gaze. EEE
Transactions on Pattern Analysis and Machine Intelligence 32(3), 478–500 (2010)
8. Jacob, R.J.K.: The use of eye movements in human-computer interaction techniques: What
you look at is what you get. ACM Transactions on Information Systems 9(2), 152–169 (1991)
9. Kastner, S., Ungerleider, L.G.: Mechanisms of visual attention in the human cortex. Annual
Review of Neuroscience 23, 315–341 (2000)
10. Khushaba, R.N., Wise, C., Kodagoda, S., Louviere, J., Kahn, B.E., Townsend, C.: Consumer
neuroscience: Assessing the brain response to marketing stimuli using electroencephalogram
(EEG) and eye tracking. Expert Systems with Applications 40(9), 3803–3812 (2013)
11. Lagerlund, T.D., Cascino, G.D., Cicora, K.M., Sharbrough, F.W.: Long-term electroen-
cephalographic monitoring for diagnosis and management of seizures. Mayo Clinic Pro-
ceedings 71(10), 1000–1006 (1996)
12. Landolt, H.: Genetic determination of sleep EEG profiles in healthy humans. Progress in
Brain Research 193, 51–61 (2011)
13. Petersen, S.E., Posner, M.I.: The attention system of the human brain: 20 years after. Annual
Review of Neuroscience 35, 73–89 (2012)