In the near future our brain will be connected to many applications. Recently research has concentrated on using the Brain Computer Interface (BCI) passively to recognize particular users' mental states. In this paper, we explore the possibility to harness electroencephalograph (EEG) signals captured by of-the-shelf EEG low-cost headsets to understand if an exhibition piece is of interest for a visitor. This information can be used to enrich the user profile and consequently to suggest artworks to see during the visit according to a recommendation strategy. The results of the exploratory study show the feasibility of the proposed approach
• Human-centered computing → User models;
Human–computer interaction based on physiological signals is expected to be the next breakthrough in the field of multimedia systems, especially as far as afective computing is concerned. In this view, research presented in this study aims at developing a Brain-Computer Interface (BCI) to understand which exhibition piece the user is interested in during a virtual/real museum visit and use this information to personalize the visit using an appropriate recommendation strategy.
Usually electroencephalography (EEG) devices and BCI provide a
way to measure brain activity and establish a direct communication
between brain and computing systems. Useful applications of EEG
have been developed mainly in the field of assistive technologies,
helping disabled users to control external devices [
Data collected in this experiment allowed learning a model for recognizing in real-time user’s interest and use this information to enrich the user profile and provide recommendations accordingly.
The paper, after a brief section explaining the motivation for pursuing this approach, illustrates how visual engagement is calculated. Then the results of the study are presented. Conclusions and future work directions are discussed in the last section. 2
Many authors have investigated the use of physiological signals to
detect and recognize users’ characteristics during the interaction
[
EEG devices measure brain signals by placing electrodes on
certain locations on the scalp that measure changes in electrical
potential as neurons in the brain’s cerebral cortex are fired [
From the interaction viewpoint, BCI systems can be used in an
active way, by allowing users to control a system by a conscious
mental activity, and in a passive one, by monitoring the user brain
activity to recognize mental states that are used as an input to
the application [
This is the type of approach needed in our system, the indicators
of "engaging" and "interest" as implicit feedback for personalizing
the museum visit. Since in the last few years museums direct their
eforts to provide personalized services both though their websites
and on site ofering personalized guide and descriptions of items,
this information about the user can be used to build a user profile
and then to provide recommendations [
This approach has been used successfully in several projects.
In the FOCUS system BCI is used to monitor engagement while
children are reading [
Results of these research works are promising and, even if the experiments were performed on a small number of users, they show the potentiality of the approach. Therefore we decided to investigate if the same type of information about user’s mental state could be captured using a cheaper headset with only one dry electrode. 3
The visitor experience in a museum is mainly shaped by his behavior based on his interest and engagement in the exhibited items. The cognitive component of interest corresponds to the activation of the pre-frontal cortex of the brain captured using EEG signals. We propose a museum experience that utilizes brain signals acquired by commercially available BCI systems to sense the museum visitors’ engagement in exhibits and provide real- time feedback to the visitor with personalized recommendations.
The recent availability of low-cost commercial and comfortable
EEG headsets makes the use of this technology afordable for a
museum that can then serve a large number of users. In this work,
Neurosky [
The headset is equipped with a single-channel EEG sensor and an electrode that rests on the forehead on the FP1 position according to the international 10-20 system and a second electrode that touches the ear. This sensor is used as ground to filter out the electrical noise. Sensors are capable of detecting raw EEG signals, frequency of diferent brainwaves: Delta (0-3 Hz), Theta (4-7 Hz), Alpha (812 Hz), Beta (12-30 Hz) and Gamma (30-100 Hz), and two mental states (attention and meditation) that are calculated by proprietary algorithms. Neurosky MindWave was chosen due to its afordability, portability, wireless connection capability and the availability of an open source API (Application Programming Interface). Finally, it ofers unencrypted EEG signal.
We are aware that a major limitation in using this headset is the accuracy of the EEG signal, because this headset has only one electrode. However, our challenge is to have as much information as possible, avoiding stressing the user in terms of the discomfort of the device. The size and comfort of the device used may allow for a real-time assessment of users preferences and then for a provision of fine-tuned suggestions and content during the visit. 3.1
Electroencephalography (EEG) is a method to measure the brain’s electrical activity. Usually the EEG signals can be afected by noise due to eye movements, muscle noise, heart signals, and so on. The BCI allows filtering the noise signal while preserving the essential EEG signals. EEG frequencies have been extensively studied and can provide insight into user mood and emotions such as excitement, meditation, pleasure and frustration. EEG measures are also sensitive to cognitive states including engagement and attention.
As far as engagement measurement is concerned, Pope et al.
[
The formula uses the Alpha (α ) band (7-13 Hz) associated with relaxation, the Beta (β ) band (13-30 Hz) associated with attentiveness and focus, and finally the Theta (θ ) band (4-7 Hz) associated with dreaminess and creativity.
Berka et al. [
A BCI-based Assessment of User’s Interests in a Museum Visit 4
In the following sections, we discuss our experimental setup, data collection and analysis and, at the end, our findings. 4.1
Twenty-four participants took part in the study aged from 16 to 60 (mean age of 26.9 with a standard deviation of 10.75). Sixteen of them were male and eight were female. Twenty participants were students and four were part of the teaching staf.
According to the purpose of this study we selected 20 artworks from wikiart.org. The selection was made according to the five main styles present in wikiart classification: Medieval Art, Renaissance Art, Post Renaissance Art, Modern Art, Contemporary Art. For each style, we selected artworks of two diferent genres: painting and sculpture.
Before starting the experiment, participants were given an overview regarding the EEG technology, experiment, and the type of data collected. Consent was signed by all of the subjects. Participants were trained on how to use the headset and the application. Before the experiment they were asked to wait few minutes to stabilize EEG signals. The total time of the experiment was approximately 5 minutes for most of the participants. After completing the experiment, participants were asked to answer a questionnaire about their demographics, health status and other questions to rate the device’s comfort level. The questionnaires showed that all of the subjects did not sufer from any health issues prior to the experiments. The experiment was conducted in a room with controlled lighting in a research lab in our Department. Both the experimental tasks and the EEG recording were controlled with the same computer. 4.2
In order to acquire data to learn a model that can be used to recognize the user’s interest, we implemented an interface to randomly show the selected artworks to the user.
The adopted protocol is the following (Figure 2). At the beginning of the interaction the user is asked to relax for 10 seconds. The data recorded in this time represents the baseline for that user. At the end of this relaxation period, the artwork image, selected randomly, is shown to the user for 10 seconds. During this time, the brain signals are recorded and immediately after an evaluation screen is shown to the user. Through this screen, the user expresses his judgment in terms of "I’m interested" (I), "I’m not interested" (NI) or "Neutral" (N). In order to relax the mind and move on to the next artwork, a neutral screen is shown again for 10 seconds.
Band power data is used to calculate visual engagement. In particular, we use Pope’s formula to calculate a vector of engagement values both for the initial relax time (EngRelax) and for each artwork visualization time (EngImage). Starting from these data we calculate the Euclidean distance between the so acquired two vectors. This distance represents the Visual Engagement Index (VEI) and it is stored in a log file together with the evaluation explicitly expressed by the user. A total of 580 instances were collected. 4.3
To implement a process able to use EEG signals as implicit
feedback, we used the data collected during the experiment to learn
a classification model. To this aim we used the WEKA platform
[
Results, calculated using 10-fold cross-validation, show an average accuracy on three classes of 0.75 and a average F1-measure of 0.672. Analyzing the classification results in more details by looking at the confusion matrix (see Table 1), we noticed that the majority of instances of class N, corresponding to a neutral interest, were misclassified. This result is encouraging and, even if it does not provide a way in detecting nuances in the level of interest of the user, it is able to discriminate between interesting and not interesting items.
Supported by this result we studied the feasibility of using such in real-time by predicting user’s interest while looking at an exhibition piece. To this aim we conducted a very simple experiment. We selected 20 new items from wikiart.org equally distributed along the 5 styles used for the first experiment.
We asked to 10 participants, aged between 19 and 52 y.o, (avg=25.3, std.dev=9.3) to perform a task similar to the one of the first experiment. Each participant had to wear the headset and look at 2 randomly selected artworks according to the protocol described previously. This time, the interest or not towards the artwork was indicated by the system according to the result of the classification of the VEI done using the learned model. Each participant could agree or not with the system by changing the predicted interest. Hit Ratio is a way of calculating how many "hits" a user has in a list of recommended items. A "hit" could be defined as something that the user has clicked on, purchased, or saved/favourite (depending on the context). In this case we consider a "hit" the agreement between the system prediction and the explicit user evaluation. If we consider this as a measure of success of the classifier then the proposed approach showed to be efective since in 70% of cases the user agreed with the system prediction. 5
Detecting visitor’s interest and emotions implicitly from EEG signals using low-cost and commercially available devices is the main aim of our research. In particular we plan to employ this approach in the context of personalized virtual or real museum visit. In this paper, we have presented how the EEG signals, gathered using the Neurosky MindWave device, can be used as a potential source for implicit feedback recognition. To achieve our goal we developed a function to detect visual engagement from EEG signals and then, to test the feasibility of our approach we performed a preliminary controlled experiment aimed at predicting in real-time user’s interest in a piece of art. Results show the feasibility of the proposed approach since, using SVM, we had a good accuracy on three classes (not interesting, neutral and interesting). The implicitly recognized interest can then be used to enrich the user profile and personalize museum visits.
While the work presented here is focused on understanding how to relate observations to predicted ratings, we then hope to develop and implement a prototype that will give us some insight into how implicit feedback can be used efectively in an application environment. For instance the level of engagement can be used not only to suggest what to see during the visit, but also to adapt the description content of a piece of art.
We also plan to conduct experiments using a more accurate commercial device with a major number of electrodes in order to detect not only the interest aroused while looking at an artwork but also to recognize the elicited emotion.