In the paper, we present a new approach to solving the problem of combining data coming from eye-tracking and electroencephalography (EEG). This is a challenging problem in neurocognitive research. An important issue added to the process of combining data received from both devices is their synchronization. The created software system solving the problem is based on a specialized expert system.
is the technology offering the possibility of capturing visual behavior in real-time and
monitoring locations of fixations within images [
There are various areas in which combining eye-tracking and EEG is used. First of
all, we can distinguish the following ones:
– visual search which involves finding a target in the midst of distractors (see, for
example, [
There is a need to integrate two methods of gathering data, mentioned earlier, for neurocognitive research. The main problem arising in combining eye-tracking and EEG data is their synchronization. Until now, there is a lack of information in the literature about the fulfilling solution of the problem. In our approach, we propose to use a specialized expert system.
For the research purposes, we have developed a software system whose task is to synchronize data received from EyeTracker Tobii T60 and Emotiv EPOC EEG (Portable version 1.0). The main problem arising in this task is that both devices work independently and we can only capture the data separately and next save them to files. In case of Tobii T60, the software is a black-box and we have access only to the recorded data. Therefore, these data can be analyzed in further stages. In case of EPOC EEG, the API (Application Programming Interface) is available. It allows us to prepare a dedicated software system that uses the signals obtained from the sensors.
The following assumptions have been made in order to fulfil requirements for the dedicated software system: 1. The measurement data must be synchronized in time. 2. The devices can be connected to two different machines at the same time (recommended). 3. Measurements should have a high time resolution. 4. An expert system analyzes the received data searching for the selected features (correlations, patterns, etc.).
To achieve these goals, we have proposed the following solution. Each device must have a system clock synchronized with the selected time standard. In our research, we have used a solution based on time synchronization with the atomic clock. The freeware Atomic Clock Sync v.3.5 has been used. One can also run a local time server to synchronize the two devices. If we run two applications on a single machine, synchronization is not necessary. However, our experience shows that this is a data-intensive computing job and there are situations when records are not synchronized. Another advantage of execution of tasks on different machines is the possibility for running applications under different operating systems. As we have access to the API of EPOC EEG, we are able to get time resolution between the recorded samples with the accuracy no less than 0.1 also with respect to data received from Tobii T60. An expert system analyzes the synchronized data received from two different devices, searching for selected features of the signal. An additional advantage of this approach is the ability to export both raw and processed data in the CSV format, which allows us to perform analyses using other specialized tools.
Gathering synchronized data is necessary but not sufficient for performing the reasoning process. An important problem, with respect to the character of the measured signals, is their quality as well as noise level. In case of the eye-tracker signal, the precision is high and the measurement is stable. Meanwhile, in case of the EEG signal, data are noised and they can include signals from the surroundings. The level of the EEG signal depends on the region in the brain.
The expert system consists of three main parts: – a signal filtering part, – an activity pattern recognition part, – a reasoning part based on the first order logic.
An important novelty of the proposed approach is the application of information on the optic nerve gathered from the eye-tracker for removing artefacts. This effect is difficult to obtain using only the EEG measurement.