=Paper=
{{Paper
|id=Vol-2091/paper1
|storemode=property
|title=BrainArt: a BCI-based Assessment of User's Interests in a Museum Visit
|pdfUrl=https://ceur-ws.org/Vol-2091/paper1.pdf
|volume=Vol-2091
|authors=Fabio Abbattista,Valeria Carofiglio,Berardina De Carolis
|dblpUrl=https://dblp.org/rec/conf/avi/AbbattistaCC18
}}
==BrainArt: a BCI-based Assessment of User's Interests in a Museum Visit==
https://ceur-ws.org/Vol-2091/paper1.pdf
BrainArt: a BCI-based Assessment of User’s Interests in a
Museum Visit
Fabio Abbattista Valeria Carofiglio Berardina De Carolis
Department of Computer Science, Department of Computer Science, Department of Computer Science,
University of Bari University of Bari University of Bari
Bari, Italy Bari, Italy Bari, Italy
fabio.abbattista@uniba.it valeria.carofiglio@uniba.it berardina.decarolis@uniba.it
ABSTRACT into account that our brain processes constantly information and
In the near future our brain will be connected to many applica- sensory inputs that pervade our daily life, it is easy to imagine ap-
tions. Recently research has concentrated on using the Brain Com- plications that consider the impact of media (images, music, video,
puter Interface (BCI) passively to recognize particular users’ mental movies, etc.) upon the brain. In case of a museum visit, the number
states. In this paper, we explore the possibility to harness electroen- of items that users may look at is huge. In this case, personalized
cephalograph (EEG) signals captured by off-the-shelf EEG low-cost suggestions may be used to tailor the visit to users’ interests and
headsets to understand if an exhibition piece is of interest for a preferences. In recent years, the maturity of methods for the unob-
visitor. This information can be used to enrich the user profile and trusive acquisition of implicit feedback [16] has grown to a level
consequently to suggest artworks to see during the visit according that allows its incorporation in personalized systems. To this aim,
to a recommendation strategy. The results of the exploratory study research has investigated the use of physiological signals to detect
show the feasibility of the proposed approach and recognize users’ interest and engagement. Devices actually
used to get this kind of feedback are mainly heartbeat monitors,
CCS CONCEPTS galvanic skin response sensors and headsets to capture electroen-
cephalogram (EEG) signals. In this paper we investigated on the
• Human-centered computing → User models;
use of a EEG low- cost commercial headset (MindWave - Neurosky)
to capture and recognize the user interest in a artwork by detecting
KEYWORDS
his engagement level during the visualization of a piece of art. The
BCI, Engagement, Museum Visit idea is to apply real-time brainwave signal detection techniques to
ACM Reference Format: get a feedback about which pieces of the exhibition are interesting
Fabio Abbattista, Valeria Carofiglio, and Berardina De Carolis. 2018. BrainArt: for the user while he is looking at that item. To achieve our goal we
a BCI-based Assessment of User’s Interests in a Museum Visit. In Proceed- developed a function to detect visual engagement and then, to test
ings of 2nd Workshop on Advanced Visual Interfaces for Cultural Heritage the feasibility of the approach, we performed a preliminary con-
(AVI-CH 2018). Vol. 2091. CEUR-WS.org, Article 1. http://ceur-ws.org/Vol- trolled experiment aimed at detecting interest of museum visitors
2091/paper1.pdf, 4 pages.
in a item and relating it to the level of visual engagement.
Data collected in this experiment allowed learning a model for
1 INTRODUCTION recognizing in real-time user’s interest and use this information to
Human–computer interaction based on physiological signals is enrich the user profile and provide recommendations accordingly.
expected to be the next breakthrough in the field of multimedia The paper, after a brief section explaining the motivation for
systems, especially as far as affective computing is concerned. In pursuing this approach, illustrates how visual engagement is calcu-
this view, research presented in this study aims at developing a lated. Then the results of the study are presented. Conclusions and
Brain-Computer Interface (BCI) to understand which exhibition future work directions are discussed in the last section.
piece the user is interested in during a virtual/real museum visit and
use this information to personalize the visit using an appropriate
2 MOTIVATIONS AND BACKGROUND
recommendation strategy.
Usually electroencephalography (EEG) devices and BCI provide a Many authors have investigated the use of physiological signals to
way to measure brain activity and establish a direct communication detect and recognize users’ characteristics during the interaction
between brain and computing systems. Useful applications of EEG [15]. Devices actually used to get bio and neuro-feedback have
have been developed mainly in the field of assistive technologies, gained popularity especially in the context of videogames. Due
helping disabled users to control external devices [9, 12]. However, to their ability to capture the engagement of a user beyond his
with the advent of inexpensive and commercial EEG headsets, ap- conscious and controllable behaviors and in a transparent manner,
plications in new domains are being proposed (e.g., for enhancing EEG devices are being used in HCI context [18]. Since non-invasive
the user experience during artistic performances [21], for monitor- commercial electroencephalography (EEG) devices have recently
ing attention levels during learning tasks [2], and so on). Taking become more available on the market it is feasible to think about
their use in domains such us music listening, video watching, etc.
EEG devices measure brain signals by placing electrodes on
AVI-CH 2018, May 29, 2018, Castiglione della Pescaia, Italy
certain locations on the scalp that measure changes in electrical
© 2018 Copyright held by the owner/author(s).
potential as neurons in the brain’s cerebral cortex are fired [13].
1
AVI-CH 2018, May 29, 2018, Castiglione della Pescaia, Italy Fabio Abbattista, Valeria Carofiglio, and Berardina De Carolis
The collected signals are divided into five different frequency bands
that have been proven to provide insight into a person’s cognitive
states such as attention/engagement and relaxation [17].
From the interaction viewpoint, BCI systems can be used in an
active way, by allowing users to control a system by a conscious
mental activity, and in a passive one, by monitoring the user brain
activity to recognize mental states that are used as an input to
the application [18] or to understand the user’s mental state as a
feedback to the received stimulus. In this case the interpretation
of user’s mental state could be used as a source of control to the
automatic system adaptation [6].
This is the type of approach needed in our system, the indicators Figure 1: Neurosky wireless EEG Mindwave device.
of "engaging" and "interest" as implicit feedback for personalizing
the museum visit. Since in the last few years museums direct their
efforts to provide personalized services both though their websites
and on site offering personalized guide and descriptions of items, The headset is equipped with a single-channel EEG sensor and an
this information about the user can be used to build a user profile electrode that rests on the forehead on the FP1 position according to
and then to provide recommendations [10]. An increasing number the international 10-20 system and a second electrode that touches
of museums use personalized museum guides to enhance visitors’ the ear. This sensor is used as ground to filter out the electrical
experiences, attract new visitors, and satisfy the needs of a diverse noise. Sensors are capable of detecting raw EEG signals, frequency
audience [11, 19]. Ardissono et al. [4] provide a detailed survey of different brainwaves: Delta (0-3 Hz), Theta (4-7 Hz), Alpha (8-
of the field of personalized applications in cultural heritage. In 12 Hz), Beta (12-30 Hz) and Gamma (30-100 Hz), and two mental
our approach the BCI can monitor passively the user’s experience states (attention and meditation) that are calculated by proprietary
during the museum visit in real time providing a feedback that can algorithms. Neurosky MindWave was chosen due to its affordability,
be used to personalize the visiting experience. portability, wireless connection capability and the availability of
This approach has been used successfully in several projects. an open source API (Application Programming Interface). Finally,
In the FOCUS system BCI is used to monitor engagement while it offers unencrypted EEG signal.
children are reading [7]. Andujar and Gilbert [3] proposed a proof We are aware that a major limitation in using this headset is
of concept investigating the ability to retain more information by the accuracy of the EEG signal, because this headset has only one
incrementing physiological engagement using the Emotiv EPOC. electrode. However, our challenge is to have as much information
Recently, Yan et al. [22] show how the measurement and analysis of as possible, avoiding stressing the user in terms of the discomfort of
audience engagement from EEG measurement level during a three- the device. The size and comfort of the device used may allow for a
dimensional virtual theatre performance have positive impacts on real-time assessment of users preferences and then for a provision
the user experience. Abdelrahman et al. [1] report their experience of fine-tuned suggestions and content during the visit.
in using EEGfeedback for detecting visual engagement of museum
visitors using Emotiv EPOC.
Results of these research works are promising and, even if the
3.1 Visual Engagement Measurement
experiments were performed on a small number of users, they Electroencephalography (EEG) is a method to measure the brain’s
show the potentiality of the approach. Therefore we decided to electrical activity. Usually the EEG signals can be affected by noise
investigate if the same type of information about user’s mental due to eye movements, muscle noise, heart signals, and so on. The
state could be captured using a cheaper headset with only one dry BCI allows filtering the noise signal while preserving the essential
electrode. EEG signals. EEG frequencies have been extensively studied and
can provide insight into user mood and emotions such as excite-
ment, meditation, pleasure and frustration. EEG measures are also
3 THE PROPOSED APPROACH sensitive to cognitive states including engagement and attention.
The visitor experience in a museum is mainly shaped by his behav- As far as engagement measurement is concerned, Pope et al.
ior based on his interest and engagement in the exhibited items. [14] defined the following formula relying on three of the fre-
The cognitive component of interest corresponds to the activation quency bands which correlate EEG signals with task engagement:
of the pre-frontal cortex of the brain captured using EEG signals. Enдaдement = β/(α + θ ).
We propose a museum experience that utilizes brain signals ac- The formula uses the Alpha (α) band (7-13 Hz) associated with
quired by commercially available BCI systems to sense the museum relaxation, the Beta (β) band (13-30 Hz) associated with attentive-
visitors’ engagement in exhibits and provide real- time feedback to ness and focus, and finally the Theta (θ ) band (4-7 Hz) associated
the visitor with personalized recommendations. with dreaminess and creativity.
The recent availability of low-cost commercial and comfortable Berka et al. [5] has shown that the engagement index reflected a
EEG headsets makes the use of this technology affordable for a person’s process of visual scanning and attention. This formula has
museum that can then serve a large number of users. In this work, been used successfully in several projects with encouraging results
Neurosky [8] wireless EEG Mindwave device is used (Figure 1). [1, 3, 17].
2
A BCI-based Assessment of User’s Interests in a Museum Visit AVI-CH 2018, May 29, 2018, Castiglione della Pescaia, Italy
4 AN EXPLORATORY STUDY
In the following sections, we discuss our experimental setup, data
collection and analysis and, at the end, our findings.
4.1 Participants and Methodology
Twenty-four participants took part in the study aged from 16 to 60
(mean age of 26.9 with a standard deviation of 10.75). Sixteen of
them were male and eight were female. Twenty participants were
students and four were part of the teaching staff.
According to the purpose of this study we selected 20 artworks Figure 2: An example of interaction sequence during the ex-
from wikiart.org. The selection was made according to the five periment.
main styles present in wikiart classification: Medieval Art, Re-
naissance Art, Post Renaissance Art, Modern Art, Contem-
porary Art. For each style, we selected artworks of two different a classification model. To this aim we used the WEKA platform
genres: painting and sculpture. [20]. According to research on the topic we applied the SVM al-
Before starting the experiment, participants were given an overview gorithm and, in particular, we used the SMO (Sequential Minimal
regarding the EEG technology, experiment, and the type of data Optimization) algorithm that handles multiclass data by combining
collected. Consent was signed by all of the subjects. Participants binary SMOs. The three classes of interest were NI= Not Interesting,
were trained on how to use the headset and the application. Before N=Neutral, and I=Interesting.
the experiment they were asked to wait few minutes to stabilize Results, calculated using 10-fold cross-validation, show an aver-
EEG signals. The total time of the experiment was approximately 5 age accuracy on three classes of 0.75 and a average F1-measure of
minutes for most of the participants. After completing the experi- 0.672. Analyzing the classification results in more details by looking
ment, participants were asked to answer a questionnaire about their at the confusion matrix (see Table 1), we noticed that the majority
demographics, health status and other questions to rate the device’s of instances of class N, corresponding to a neutral interest, were
comfort level. The questionnaires showed that all of the subjects misclassified. This result is encouraging and, even if it does not pro-
did not suffer from any health issues prior to the experiments. The vide a way in detecting nuances in the level of interest of the user,
experiment was conducted in a room with controlled lighting in a it is able to discriminate between interesting and not interesting
research lab in our Department. Both the experimental tasks and items.
the EEG recording were controlled with the same computer. Table 1: Confusion Matrix
4.2 Data Acquisition NI N I
In order to acquire data to learn a model that can be used to recog- 198 95 4 NI
nize the user’s interest, we implemented an interface to randomly 95 9 36 N
show the selected artworks to the user. 10 0 228 I
The adopted protocol is the following (Figure 2). At the beginning
of the interaction the user is asked to relax for 10 seconds. The data Supported by this result we studied the feasibility of using such in
recorded in this time represents the baseline for that user. At the real-time by predicting user’s interest while looking at an exhibition
end of this relaxation period, the artwork image, selected randomly, piece. To this aim we conducted a very simple experiment. We
is shown to the user for 10 seconds. During this time, the brain selected 20 new items from wikiart.org equally distributed along
signals are recorded and immediately after an evaluation screen the 5 styles used for the first experiment.
is shown to the user. Through this screen, the user expresses his We asked to 10 participants, aged between 19 and 52 y.o, (avg=25.3,
judgment in terms of "I’m interested" (I), "I’m not interested" (NI) std.dev=9.3) to perform a task similar to the one of the first exper-
or "Neutral" (N). In order to relax the mind and move on to the next iment. Each participant had to wear the headset and look at 2
artwork, a neutral screen is shown again for 10 seconds. randomly selected artworks according to the protocol described
Band power data is used to calculate visual engagement. In par- previously. This time, the interest or not towards the artwork was
ticular, we use Pope’s formula to calculate a vector of engagement indicated by the system according to the result of the classification
values both for the initial relax time (EngRelax) and for each artwork of the VEI done using the learned model. Each participant could
visualization time (EngImage). Starting from these data we calculate agree or not with the system by changing the predicted interest.
the Euclidean distance between the so acquired two vectors. This Hit Ratio is a way of calculating how many "hits" a user has in a
distance represents the Visual Engagement Index (VEI) and it is list of recommended items. A "hit" could be defined as something
stored in a log file together with the evaluation explicitly expressed that the user has clicked on, purchased, or saved/favourite (depend-
by the user. A total of 580 instances were collected. ing on the context). In this case we consider a "hit" the agreement
between the system prediction and the explicit user evaluation. If
4.3 Implicit Feedback Recognition we consider this as a measure of success of the classifier then the
To implement a process able to use EEG signals as implicit feed- proposed approach showed to be effective since in 70% of cases the
back, we used the data collected during the experiment to learn user agreed with the system prediction.
3
AVI-CH 2018, May 29, 2018, Castiglione della Pescaia, Italy Fabio Abbattista, Valeria Carofiglio, and Berardina De Carolis
5 CONCLUSION AND FUTURE WORK [6] Laurent George and Anatole Lécuyer. 2014. Passive Brain–Computer Interfaces.
Springer London, London, 297–308. https://doi.org/10.1007/978-1-4471-6584-2_
DIRECTIONS 13
Detecting visitor’s interest and emotions implicitly from EEG sig- [7] Jin Huang, Chun Yu, Yuntao Wang, Yuhang Zhao, Siqi Liu, Chou Mo, Jie Liu, Lie
Zhang, and Yuanchun Shi. 2014. FOCUS: Enhancing Children’s Engagement in
nals using low-cost and commercially available devices is the main Reading by Using Contextual BCI Training Sessions. In Proceedings of the 32Nd
aim of our research. In particular we plan to employ this approach Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM,
New York, NY, USA, 1905–1908. https://doi.org/10.1145/2556288.2557339
in the context of personalized virtual or real museum visit. In this [8] NeuroSky Inc. 2010. Brain Wave Signal (EEG) of NeuroSky, Inc. http://www.
paper, we have presented how the EEG signals, gathered using the frontiernerds.com/files/neurosky-vs-medical-eeg.pdf. (15 Dec 2010).
Neurosky MindWave device, can be used as a potential source for [9] Ivo Käthner, Jean Daly, Sebastian Halder, J Räderscheidt, Elaine Armstrong, Stefan
Dauwalder, Christoph Hintermüller, Arnau Espinosa, Eloisa Vargiu, Andreas
implicit feedback recognition. To achieve our goal we developed a Pinegger, et al. 2014. A P300 BCI for e-inclusion, cognitive rehabilitation and
function to detect visual engagement from EEG signals and then, smart home control. In Proceedings of the 6th International BCI Conference Graz
2014.
to test the feasibility of our approach we performed a preliminary [10] Tsvi Kuflik, Oliviero Stock, Massimo Zancanaro, Ariel Gorfinkel, Sadek Jbara,
controlled experiment aimed at predicting in real-time user’s inter- Shahar Kats, Julia Sheidin, and Nadav Kashtan. 2011. A Visitor’s Guide in an
est in a piece of art. Results show the feasibility of the proposed Active Museum: Presentations, Communications, and Reflection. J. Comput.
Cult. Herit. 3, 3, Article 11 (Feb. 2011), 25 pages. https://doi.org/10.1145/1921614.
approach since, using SVM, we had a good accuracy on three classes 1921618
(not interesting, neutral and interesting). The implicitly recognized [11] Ioanna Lykourentzou, Xavier Claude, Yannick Naudet, Eric Tobias, Angeliki
interest can then be used to enrich the user profile and personalize Antoniou, George Lepouras, and Costas Vassilakis. 2013. Improving museum
visitors’ Quality of Experience through intelligent recommendations: A visiting
museum visits. style-based approach.. In Intelligent Environments (Workshops). 507–518.
While the work presented here is focused on understanding [12] José del R. Millán, Rüdiger Rupp, Gernot Mueller-Putz, Roderick Murray-Smith,
Claudio Giugliemma, Michael Tangermann, Carmen Vidaurre, Febo Cincotti,
how to relate observations to predicted ratings, we then hope to Andrea Kubler, Robert Leeb, Christa Neuper, Klaus Mueller, and Donatella Mattia.
develop and implement a prototype that will give us some insight 2010. Combining Brain–Computer Interfaces and Assistive Technologies: State-
into how implicit feedback can be used effectively in an application of-the-Art and Challenges. Frontiers in Neuroscience 4 (2010), 161. https://doi.
org/10.3389/fnins.2010.00161
environment. For instance the level of engagement can be used not [13] Ernst Niedermeyer and FH Lopes da Silva. 2005. Electroencephalography: basic
only to suggest what to see during the visit, but also to adapt the principles, clinical applications, and related fields. Lippincott Williams & Wilkins.
description content of a piece of art. [14] Alan T Pope, Edward H Bogart, and Debbie S Bartolome. 1995. Biocybernetic
system evaluates indices of operator engagement in automated task. Biological
We also plan to conduct experiments using a more accurate psychology 40, 1-2 (1995), 187–195. https://doi.org/10.1016/0301-0511(95)05116-3
commercial device with a major number of electrodes in order to [15] George E. Raptis, Christina Katsini, Marios Belk, Christos Fidas, George Samaras,
and Nikolaos Avouris. 2017. Using Eye Gaze Data and Visual Activities to Infer
detect not only the interest aroused while looking at an artwork Human Cognitive Styles: Method and Feasibility Studies. In Proceedings of the
but also to recognize the elicited emotion. 25th Conference on User Modeling, Adaptation and Personalization (UMAP ’17).
ACM, New York, NY, USA, 164–173. https://doi.org/10.1145/3079628.3079690
[16] Björn W. Schuller. 2016. Acquisition of Affect. Springer International Publishing,
REFERENCES Cham, 57–80. https://doi.org/10.1007/978-3-319-31413-6_4
[1] Yomna Abdelrahman, Mariam Hassib, Maria Guinea Marquez, Markus Funk, [17] Daniel Szafir and Bilge Mutlu. 2012. Pay Attention!: Designing Adaptive Agents
and Albrecht Schmidt. 2015. Implicit Engagement Detection for Interactive That Monitor and Improve User Engagement. In Proceedings of the SIGCHI Con-
Museums Using Brain-Computer Interfaces. In Proceedings of the 17th Inter- ference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY,
national Conference on Human-Computer Interaction with Mobile Devices and USA, 11–20. https://doi.org/10.1145/2207676.2207679
Services Adjunct (MobileHCI ’15). ACM, New York, NY, USA, 838–845. https: [18] Desney S. Tan and Anton Nijholt. 2012. Brain-Computer Interfaces: Applying Our
//doi.org/10.1145/2786567.2793709 Minds to Human-Computer Interaction. Springer Publishing Company, Incorpo-
[2] Ray Adams, Richard Comley, and Mahbobeh Ghoreyshi. 2009. The Potential rated.
of the BCI for Accessible and Smart e-Learning. In Universal Access in Human- [19] Ed S. Tan and Katri Oinonen. 2009. Personalising Content Presentation in Mu-
Computer Interaction. Intelligent and Ubiquitous Interaction Environments, Con- seum Exhibitions - A Case Study. In Proceedings of the 2009 15th International
stantine Stephanidis (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 467– Conference on Virtual Systems and Multimedia (VSMM ’09). IEEE Computer Soci-
476. https://doi.org/10.1007/978-3-642-02710-9_51 ety, Washington, DC, USA, 232–238. https://doi.org/10.1109/VSMM.2009.42
[3] Marvin Andujar and Juan E. Gilbert. 2013. Let’s Learn!: Enhancing User’s En- [20] Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. 2016. Data Mining:
gagement Levels Through Passive Brain-computer Interfaces. In CHI ’13 Extended Practical machine learning tools and techniques. Morgan Kaufmann.
Abstracts on Human Factors in Computing Systems (CHI EA ’13). ACM, New York, [21] Shuo Yan, GangYi Ding, Hongsong Li, Ningxiao Sun, Yufeng Wu, Zheng Guan,
NY, USA, 703–708. https://doi.org/10.1145/2468356.2468480 Longfei Zhang, and Tianyu Huang. 2016. Enhancing Audience Engagement
[4] Liliana Ardissono, Tsvi Kuflik, and Daniela Petrelli. 2012. Personalization in in Performing Arts Through an Adaptive Virtual Environment with a Brain-
Cultural Heritage: The Road Travelled and the One Ahead. User Modeling and Computer Interface. In Proceedings of the 21st International Conference on In-
User-Adapted Interaction 22, 1-2 (April 2012), 73–99. https://doi.org/10.1007/ telligent User Interfaces (IUI ’16). ACM, New York, NY, USA, 306–316. https:
s11257-011-9104-x //doi.org/10.1145/2856767.2856768
[5] Chris Berka, Daniel J Levendowski, Michelle N Lumicao, Alan Yau, Gene Davis, [22] Shuo Yan, GangYi Ding, Hongsong Li, Ningxiao Sun, Yufeng Wu, Zheng Guan,
Vladimir T Zivkovic, Richard E Olmstead, Patrice D Tremoulet, and Patrick L Longfei Zhang, and Tianyu Huang. 2016. Enhancing Audience Engagement
Craven. 2007. EEG correlates of task engagement and mental workload in vigi- in Performing Arts Through an Adaptive Virtual Environment with a Brain-
lance, learning, and memory tasks. Aviation, space, and environmental medicine Computer Interface. In Proceedings of the 21st International Conference on In-
78, 5 (2007), B231–B244. telligent User Interfaces (IUI ’16). ACM, New York, NY, USA, 306–316. https:
//doi.org/10.1145/2856767.2856768
4