=Paper=
{{Paper
|id=Vol-2474/shortpaper6
|storemode=property
|title=Shared attention reflected in EEG, electrodermal activity and heart rate
|pdfUrl=https://ceur-ws.org/Vol-2474/shortpaper6.pdf
|volume=Vol-2474
|authors=Anne-Marie Brouwer,Ivo V. Stuldreher,Nattapong Thammasan
|dblpUrl=https://dblp.org/rec/conf/smc/BrouwerST19
}}
==Shared attention reflected in EEG, electrodermal activity and heart rate ==
https://ceur-ws.org/Vol-2474/shortpaper6.pdf
Shared attention reflected in eeg, electrodermal
activity and heart rate
Anne-Marie Brouwer Ivo V. Stuldreher Nattapong Thammasan
Perceptual and Cognitive Systems Perceptual and Cognitive Systems Faculty of Elec. Eng., Math. & CS
TNO TNO University of Twente
Soesterberg, The Netherlands Soesterberg, The Netherlands Enschede, The Netherlands
anne-marie.brouwer@tno.nl ivo.stuldreher@tno.nl n.thammasan@utwente.nl
Abstract— Monitoring directed auditory attention in groups In most research on physiological measures of cognitive
can be helpful in a range of contexts. Concurrent change in and affective processes, measures are extracted after relating
physiological variables across multiple listeners (physiological physiological signals to the time that stimuli of interest occur,
synchrony – PS) may be a suitable marker of attentional focus i.e., stimuli that are expected to elicit the cognitive or
as caused by shared affective or cognitive processes. We here
affective state of interest. In real-life contexts, this is difficult
determine PS for EEG (electroencephalography), EDA
(electrodermal activity) and heart rate in participants who were to do from a practical point of view. In addition, it is often not
instructed to either attend to an audiobook (n = 13) or to clear what the stimulus of interest is. When studying groups,
interspersed auditory events (n = 13) such as emotional sounds, a solution to this is to determine the degree to which
and beeps that attending participants needed to keep track of. physiological measures of multiple people uniformly change
Even though all participants heard the exact same audio track, (physiological synchrony -PS). Highly similar physiological
for both EEG and EDA, PS was higher for participants linked responses, i.e., high PS, would indicate shared attention to an
to participants in their own attentional group than to apparently generally relevant event. [7] showed that moments
participants in the other attentional group. No such effect was of high synchrony in EEG signals between viewers of a
found in heart rate. For a single individual, EEG PS allowed
popular television series co-occurred with interesting events,
attribution to the correct attentional group in 85% of the cases,
for EDA this was 81%. Hearing is not the same as attending - and predicted the expressions of interest and attention to the
our results are promising for monitoring group affective and television series, as measured by viewership. [8] showed that
cognitive processes and how an individual relates to that. synchrony in EEG signals between students in a classroom
predicted class engagement and classroom dynamics, a
Keywords— attention, affective, cognitive, EEG, EDA, ECG, relationship that may be driven by shared attention in a group.
heart rate, skin conductance, auditory, group [9] presented participants with the same auditory or
audiovisual stimulus, but instructed them to either attend to
I. INTRODUCTION
this stimulus, or to perform an unrelated mental arithmetic
We are interested in tools that enable continuous task throughout the duration of the stimulus. They showed
monitoring of cognitive or affective processes, without that EEG PS differed between these conditions. There is also
requiring conscious action of the monitored individuals. a body of literature on synchrony in peripheral physiological
Information about attention in a group of individuals, or how measures such as heart rate and EDA (reviewed by [10]).
attention in a certain individual relates to attention in other Rather than as indicators of shared directed attention (and
individuals, may be useful to study and support children in an hence, shared affective and cognitive processes), these have
educational setting who suffer from attentional problems, or been more generally interpreted as indicators of some form
helpful to evaluate and design effective educational material. of connectedness between people. Up to date, PS literature on
Continuous and implicit measures of attention may be neural and peripheral physiological signals have remained
extracted from physiological signals. For instance, in a series separate.
of similar stimuli, a deviant that automatically draws In the current study we compare PS in neural and
attention generates a P3 peak in electroencephalography peripheral physiological variables to determine differential
(EEG) [1]. Not only bottom-up, but also top-down, ‘self- attentional focus of individuals who are all presented with the
determined’ attention to events elicits attention-related same stimulus, and are all attending to it, be it to different
evoked potentials in EEG [2] [3]. Emotional stimuli have stimulus aspects. Reminiscent to a classroom setting where
been shown to affect physiological measures, such as students hear the teacher talk as well as hearing other auditory
electrodermal activity (EDA) and heart rate [4] [5] and potentially interesting events, we present our participants
cognitive working memory tasks induce changes in a range with the same auditory stimulus, consisting of an audiobook,
of physiological measures as well [6]. While the interspersed with short stimuli. Participants are instructed to
physiological responses elicited by emotional stimuli and attend to either the audiobook narrative, or to the short
mental tasks do not reflect (only) attention, these processes stimuli. We hypothesize that EEG, EDA and heart rate
are expected to be associated with attention, and hence are of recordings of participants are more strongly synchronized
interest when one is interested in monitoring attention. with those of participants in the same attentional condition
compared to the other attentional condition. To the best of our
knowledge, this is the first study that examines synchrony in
This work was supported by The Netherlands Organization for multiple neural and peripheral physiological measures, and
Scientific Research (NWA Startimpuls 400.17.602). the extent to which these measures distinguish between
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
groups of individuals with a different auditory attentional At the end of the audiobook, the instruction was presented
focus. to sing a song aloud after the subsequent auditory countdown
reached 0. This instruction had to be followed by the short-
II. METHODS stimuli attending group and was expected to induce stress
A. Participants [12].
We recorded from 27 participants (aged between 18 and Finally, participants filled out a questionnaire in which
48) with no self-reported problems in hearing or attention. they were asked to report as many emotional sounds as they
Participants were recruited from the participant pool of TNO could remember, to estimate the average number of high and
(the research institute where the study was conducted). Prior low beeps in a sequence, and questions about the content of
to the experiment all participants signed an informed consent the narrative.
form and after the experiment they received a small monetary
D. Analysis
award for their time and travelling costs. Data of one
participant was discarded due to failed physiological Data processing was done using MATLAB 2018b
recordings. The study was approved by the TNO Institutional software (Mathworks, Natick, MA, USA).
Review Board (TCPE) and the TU Delft Human Research EDA was downsampled to 64 Hz. The phasic component
Ethics Committee. of the signal was extracted using Continuous Decomposition
B. Materials analysis [14] as implemented in the Ledalab toolbox for
Matlab.
EEG, EDA and ECG (electrocardiogram) were recorded
using an ActiveTwo system (BioSemi, Amsterdam, ECG measurements were processed to acquire the inter-
Netherlands) at 1024 Hz. EEG was recorded with 32 active beat interval (IBI – the inverse of heart rate). After
Ag-AgCl electrodes, placed on the scalp according to the 10- downsampling to 256 Hz, ECG was high-pass filtered at 0.5
20 system, together with a common mode sense (CMS) active Hz. Peaks were detected from a squared version of the
electrode and a driven right leg (DRL) passive electrode for reconstructed frequency-localized version of the ECG
referencing. Electrode impedance threshold was set at 20 waveform using wavelets [15]. The IBI semi-time series was
kOhm. For EDA, two passive gelled Nihon Kohden transformed into a timeseries. This was done by interpolating
electrodes were placed on the ventral side of the distal consecutive IBIs and then resampling at 2 Hz.
phalanges of the middle and index finger. For ECG, two
active gelled Ag-AgCl electrodes were placed at the right EEG was processed offline with EEGLAB v14.1.2 for
clavicle and lowest floating left rib. EDA and heart rate were MATLAB [16]. EEG was first downsampled to 256 Hz, high-
also recorded using wearable systems. These data will be pass filtered at 1 Hz and notch filtered at 50 Hz, using the
discussed elsewhere. standard FIR-filter implement in EEGLAB function
pop_eegfiltnew. Channels were re-referenced to the average
C. Stimuli and Design channel values. Logistic infomax independent component
Each participant listened to the same audio file, composed analysis (ICA, [17]) was performed on more strongly filtered
of a 66 min audiobook (a Dutch thriller ‘Zure koekjes’, data to classify artifactual independent components, i.e.,
written by Corine Hartman) interspersed with other auditory components not reflecting sources of neural activity, but
stimuli. Intervals between these short stimuli varied between ocular or muscle-related artifacts. These components were
35 and 55 seconds. Half of the participants were asked to removed from the data. Samples whose squared amplitude
focus on the narrative of the audiobook and ignore all other magnitude exceeded the mean-squared amplitude of that
stimuli or instructions; and half of the participants were asked channel by more than four standard deviations were marked
to focus on the other stimuli and perform accompanying as missing data (’NaN’).
tasks, and ignore the narrative. The auditory stimuli were Similarity of EEG between participants in the time-
emotional sounds, beeps, and the instruction to sing a song. domain was assessed using correlated component analysis
The order of sounds and beeps was randomly determined. (CorrCA) [18]. CorrCA is similar to the more familiar
Emotional sounds were taken from the IADS principal component analysis, except that projections of
(International Affective Digitized Sounds – [11]). The IADS CorrCA capture maximal correlations between data sets
is a collection of acoustic stimuli that have been normatively instead of maximal variance within a set of data. Rather than
rated for emotion. Examples of stimuli are the sound of a treating EEG channels separately, this analysis results in
crying baby or a cheering sports crowd. We selected 12 correlated components. ISC (inter-subject correlation) is
neutral sounds (IADS number 246, 262, 373, 376, 382, 627, determined by the sum of correlations of the first three of
698, 700, 708, 720, 723, 728), 12 pleasant sounds (110, 200, these components. See [9] for a detailed description of the
201, 202, 311, 352, 353, 365, 366, 367, 415, 717) and 12 procedure that was followed. To discriminate between
unpleasant sounds (115, 255, 260, 276, 277, 278, 279, 285, attentional task conditions, correlated component vectors
286, 290, 292, 422). Sound duration was 6 seconds. were extracted from both the narrative and short-stimuli
group. Data from each subject was then projected on these
Beeps were presented in blocks of 30 seconds, with every component vectors. Correlations between each participant
two seconds a 100ms high (1kHz) or low (250Hz) pitched with all other members of the narrative and short-stimuli
beep. Short-stimuli attending participants needed to group were computed. The average correlation between a
separately count the number of high and low tones [12]. This participant and all participants in the narrative and in the
task was practiced with them beforehand. In total, 27 blocks short-stimuli groups are from now on referred to as ISC-
of sounds were presented.
narrative and ISC-short-stim. To avoid training biases in the
component extraction step, data from the to-be tested subject
were excluded in this step.
Similarity of EDA (phasic component) and IBI between
participants in the time-domain was assessed using a moving
window approach, introduced by [19]. Pearson correlations
were calculated over successive, running 15s windows at 1s
increments. The overall correlation between two responses
was computed as the natural logarithm of the sum of all
positive correlations divided by the sum of the absolute
values of all negative correlations. As for EEG, ISC-narrative
and ISC-short-stim were determined for each participant by
determining his or her ISC with each of the members of the
narrative group as well as with the short-stimuli group.
Wilcoxon rank sum tests were performed to test for
differences in performance with respect to the questions
about the auditory stimuli between the two attentional groups.
Paired sample t-tests were conducted to test whether ISC-
narrative and ISC-short-stim were significantly different
within each attentional group for EEG, EDA and IBI.
III. RESULTS
Participants in the narrative group answered more
questions about the narrative correctly than participants in the
short-stimuli group (Z=2.68, p=.007), whereas participants in
the short-stimuli group could name more emotional sounds
(Z=2.68, p=.007) and were closer to the actual average
number of high and low beeps (Z=2.82, p=.005) than the
narrative group. This indicated that participants followed the
attentional instruction.
Fig. 1 shows the inter-subject correlation (ISC) averaged
across participants of the narrative group (left bars) and the
short-stimuli group (right bars) when paired with participants
of the narrative group (dark bars) or short-stimuli group (light
bars). Data of individual participants are plotted on top of the
bars. For EEG (Fig. 1A) ISC is higher for most participants
when paired to participants of their own attentional group
compared to participants from the other group. This is so both
for participants in the narrative group (t12 = 3.57, p = 0.004)
as well as the short-stimuli group (t12= -3.57, p = 0.004). For
EDA (Fig. 1B), the same pattern of results is observed, but it
only reaches significance for the short-stimuli group (t12= -
3.932, p = 0.002; narrative group: t12= 0.96, p = 0.357). For Fig. 1. Inter-subject correlations for narrative-attending participants (NA)
IBI (Fig. 1C), the trend is again the same but no significant and short-stimuli attending participants (SSA) when related to participants
effects were observed (narrative group: t12 = 0.85, p = 0.413; of each of the two groups, for EEG (A), EDA (B) and IBI (C). Connected
dots display subject-to-group correlations of each of the individual
short stimuli: t12 = -1.37, p=0.196). participants, where blue lines indicate individuals for which ISC-NA > ISC-
When assuming for each participant that she or he follows SSA and red, dotted lines indicate individuals for which ISC-SSA > ISC-
NA. Paired sample t-tests revealed that within-group correlations were
the attentional instruction as indicated by the group with higher than between-group correlations in EEG and EDA (**p < 0.01).
whom she or he shows the highest averaged synchrony,
classification accuracies are significantly higher than chance
for EEG and EDA. For EEG, classification accuracy is 85%
both for participants from the narrative and from the short-
stimuli group. For EDA, classification accuracy is 77% for
participants from the short-stimuli group and 85% for the
narrative group. For IBI, classification accuracy is not higher
than chance in both groups. Chance level was determined by
using surrogate data with randomized group labels.
Significance levels were determined using 10000 renditions
of randomized group labels. An overview of the classification
data is presented in Fig. 2.
heart rate accelerations and the orienting system with
decelerations [22]. The type of response to a certain
emotional stimulus can differ between individuals and
occasions.
In future analysis, we will examine patterns of synchrony
in the different modalities as a function of stimulus type.
Events relevant for mental tasks (counting the beeps) may be
strongly associated to synchrony in EEG, whereas emotional
stimuli (IADS and the instruction to sing a song) may be
strongly associated to synchrony in EDA. Patterns of
multimodal synchrony might even allow us to identify the
type of shared mental activity and therewith the instigator of
shared attention. Combining synchrony measures from
different modalities may support detection of (certain)
relevant events, although it is still unclear how this can be
done best [23]. It is also of interest to relate PS to behavioral
Fig. 2. Classification accuracy of inferring attentional group from ISC in or cognitive performance on tasks related to the to be attended
EEG, EDA and IBI where each participant was designated to be in the stimuli. An individual’s (moment of) low PS may be
attentional group for which he or she they showed the highest ISC. Data are
presented seperately for narrative-attending participants (NA) and short- predictive of later poor performance. Finally, we want to
stimuli attending participants (SSA). Theoretical chance level is 0.5. mention that in the current study, we purposely examined
Statistical chance level is indicated by the dashed line. interpersonal PS in a situation with very limited eye- or body
movements, and no interpersonal communication. This was
IV. DISCUSSION done in order to avoid possible confounds of physiological
We showed that PS in both EEG and EDA is indicative of measures with movements [24] and with the view of studying
shared attention: EEG and EDA signals of participants are PS in the context of attention, apart from interpersonal
more strongly synchronized with those of participants in the interaction. However, it would be of interest to bring this
same attentional condition compared to the other. For IBI, we view together with the large literature on interpersonal
did not find this. synchrony in behavioral measures such as gestures and
speech during social interaction [25].
In our setting, all participants attended to the auditory
stimulus. While participants in the short-stimuli condition ACKNOWLEDGMENT
were instructed to ignore the narrative, it was probably hard The authors thank Ana Borovac for help with recording
to do this at times without concurrent short-stimuli. In
the participants.
contrast to e.g. the study by [9], our participants did not have
another task at these times and their attention was likely REFERENCES
directed to the auditory environment, since they expected an [1] Polich, J. (2007) Updating P300: An Integrative Theory of P3a and
auditory stimulus that was relevant for them. We therefore P3b. Clin. Neurophysiol. 118(10), 2128–2148.
expect that the difference between the groups and therefore, [2] Donchin, E., Spencer, K. M., and Wijesinghe, R. (2000). The mental
our PS effects, will be strongest when only considering the prosthesis: assessing the speed of a P300-based brain-computer
times during which concurrent short-stimuli are played. In interface. IEEE Trans. Neural Syst. Rehabil. Eng. 8, 174–179.
future analysis we will first recover occurrence of generally [3] Hill, N. J., & Schölkopf, B. (2012). An online brain–computer interface
high PS (which will also indicate which events generated based on shifting attention to concurrent streams of auditory stimuli.
Journal of Neural Engineering, 9(2), 026011.
generally high shared attention) and then focus analysis at
[4] Boucsein, W. (1992). Electrodermal activity. New York, NY: Plenum
those times, to examine whether this will result in even Press.
clearer effects, perhaps also showing an effect for IBI. [5] Kreibig, S. D. (2010). Autonomic nervous system activity in emotion:
a review. Biol. Psychol. 84, 394–421.
EEG PS performed relatively well, as might have been
expected based on previous PS literature and the more direct [6] Hogervorst, M. A., Brouwer, A.-M., & van Erp, J.B.F. (2014).
Combining and comparing EEG, peripheral physiology and eye-related
link with attention. However, EDA did well too, which, even measures for the assessment of mental workload. Front. in Neurosci.,
though wearable EEG systems are available, is convenient 8, 322.
from a user perspective. The finding that IBI performed worst [7] Dmochowski, J. P., Bezdek, M. A., Abelson, B. P., Johnson, J. S.,
may not be unexpected given the fact that the relation Schumacher, E. H., & Parra, L. C. (2014). Audience preferences are
between heart rate and mental state seems less predicted by temporal reliability of neural processing. Nature
communications, 5, 4567.
straightforward than EDA. Whereas EDA has consistently
[8] Dikker, S., Wan, L., Davidesco, I., Kaggen, L., Oostrik, M.,
been found to be positively related to arousal [20], the McClintock, J., Rowland, J., Michalareas, G., van Bavel, J. J., Ding,
relation between emotional stimuli and heart rate has been M. & Poeppel, D. (2017). Brain-to-brain synchrony tracks real-world
found to be more complex. Both positive (e.g. [13]) and dynamic group interactions in the classroom. Curr. Biol., 27(9), 1375-
negative (e.g. [21]) relations between heart rate and arousal 1380.
been reported. The reason for this is probably that arousal can [9] Ki, J. J., Kelly, S. P., & Parra, L. C. (2016). Attention strongly
be associated with the body being prepared for action, cf. the modulates reliability of neural responses to naturalistic narrative
stimuli. Journal of Neuroscience, 36(10), 3092-3101.
defense reflex, or with a concentrated, focused state, cf. the
[10] Palumbo, R. V., Marraccini, M. E., Weyandt, L. L., Wilder-Smith, O.,
orienting reflex, where the defense system is associated with McGee, H. A., Liu, S., & Goodwin, M. S. (2017). Interpersonal
autonomic physiology: A systematic review of the literature. [19] Marci, C. D. (2006). A biologically based measure of emotional
Personality and Social Psychology Review, 21(2), 99-141. engagement: Context matters. Journal of Advertising Research, 46(4):
[11] Bradley, M. M. and Lang, P. J. (2007). The International Affective 381–387.
Digitized Sounds (IADS-2): Affective ratings of sounds and instruction [20] Andreassi, J. L. (2007). Psychophysiology: Human Behavior and
manual. University of Florida, Gainesville, FL, Tech. Rep. B-3. Physiological Response (5th ed.). LLC, New York: Psychology Press,
[12] De Dieuleveult, A. L., Brouwer, A. M., Siemonsma, P. C., Van Erp, J. Taylor & Francis Group.
B., & Brenner, E. (2018). Aging and sensitivity to illusory target [21] Brouwer, A. M., Hogervorst, M. A., Reuderink, B., van der Werf, Y.,
motion with or without secondary tasks. Multisensory Research, 31(3- & van Erp, J. B. F. (2015). Physiological signals distinguish between
4), 227-249. reading emotional and non-emotional sections in a novel. Brain-
[13] Brouwer, A. M., & Hogervorst, M. A. (2014). A new paradigm to Computer Interfaces 2(2–3), 76–89.
induce mental stress: the Sing-a-Song Stress Test (SSST). Frontiers in [22] Graham, F.K., & Clifton, R.K. (1966). Heart-rate change as a
neuroscience, 8, 224. component of the orienting response. Psychol. Bull. 65(5), 305.
[14] Benedek, M. & Kaernbach, C. (2010). A continuous measure of phasic [23] Stuldreher, I. V., de Winter, J. C. F., Thammasan, N., Brouwer, A.-M.
electrodermal activity. Journal of Neuroscience Methods, 190(1): 80– (in press). Analytic Approaches for the Combination of Autonomic and
91. Neural Activity in the Assessment of Physiological Synchrony.
[15] The Mathworks, Inc., ‘R wave detection in the ECG’, 2015. [Online] Proceedings 2019 IEEE International Conference on Systems, Man and
Available: https://nl.mathworks.com/help/wavelet/ug/r-wave- Cybernetics (SMC).
detection-in-the-ecg.html. [Accessed: 19- Mar- 2019]. [24] Brouwer, A.-M., Zander, T. O., van Erp, J. B. F., Korteling, J. E. &
[16] Delorme, A. and Makeig, S. (2004). Eeglab: an open source toolbox Bronkhorst, A. W. (2015) Using neurophysiological signals that reflect
for analysis of single-trial eeg dynamics including independent cognitive or affective state: six recommendations to avoid common
component analysis. Journal of Neuroscience Methods, 134(1):9–21. pitfalls. Frontiers in Neuroscience, 9, 136.
[17] Bell, A. J. & Sejnowski, T. J. (1995). An information-maximization [25] Delaherche E., Chetouani M., Mahdhaoui A., Saint-Georges C., Viaux
approach to blind separation and blind deconvolution. Neural S. & Cohen D. (2012). Interpersonal Synchrony: A Survey of
Computation, 7(6):1129–1159. Evaluation Methods across Disciplines, IEEE Transactions on
Affective Computing, 3(3), 349-365.
[18] Dmochowski, J. P., Sajda, P., Dias, J., & Parra, L. C. (2012). Correlated
components of ongoing eeg point to emotionally laden attention–a
possible marker of engagement? Frontiers in Human Neuroscience,
6:112.