=Paper=
{{Paper
|id=Vol-1663/bmaw2016_paper_7
|storemode=property
|title=A Probabilistic Approach for Detection and Analysis of Cognitive Flow
|pdfUrl=https://ceur-ws.org/Vol-1663/bmaw2016_paper_7.pdf
|volume=Vol-1663
|authors=Debatri Chatterjee,Aniruddha Sinha,Meghamala Sinha,Sanjoy Kumar Saha
|dblpUrl=https://dblp.org/rec/conf/uai/ChatterjeeSSS16
}}
==A Probabilistic Approach for Detection and Analysis of Cognitive Flow==
https://ceur-ws.org/Vol-1663/bmaw2016_paper_7.pdf
A Probabilistic Approach for Detection and Analysis of Cognitive Flow
Debatri Chatterjee, Aniruddha Sinha, Meghamala Sinha Sanjoy Kumar Saha
TCS Research, Tata Consultancy Services Ltd. Computer Science & Engineering,
Kolkata, India Jadavpur University,
Email: {debatri.chatterjee, aniruddha.s}@tcs.com, Kolkata, India
meghamala.sinha@gmail.com Email: sks_ju@yahoo.co.in
Abstract to the balance between the challenge of any task and the
skill of the person executing the task. If the task difficulty
is low compared to the held skill level of an individual, the
A performer may undergo a task with varying person tends to be in the bored state (Csíkszentmihályi,
difficulty level. It is important to know the mental Mihály, 1990). Alternatively, if the task difficulty is high,
state in order to maintain the optimum level of and the skill level is low, the person is supposed to be in
performance. The mental state of an individual the anxiety state. However, if the skill and task difficulty
varies according to their IQ levels, task level matches, the person enter Flow states i.e. a state of
difficulties or other psychological or focused concentration with a sense of enjoyment
environmental reasons. We have tried to measure (Csíkszentmihályi, Mihály, 1997), (Csíkszentmihályi,
the cognitive state of individuals, while they are Mihály, 1999). In our case, the skill level of an individual
performing tasks of various complexity levels, is directly related to his or her IQ level and is treated as the
using physiological responses like brain prior knowledge of the individual. On the other hand, the
activation, heart rate variability and galvanic skin challenge of the task is synonymous to the task difficulty
response. In this paper we have proposed a level. Hence the flow-boredom state can be represented as
Bayesian network based model to shown in Figure 1.
probabilistically evaluate the cognitive state of an
individual from the difficulty levels of the tasks,
IQ level of the individual and observations made
using the physiological sensing. Twenty subjects
with various IQ levels are asked to play a
modified Tower of London (TOL) game having
three complexity levels: low, medium and high.
The sensor data collected have been used to train
the Bayesian model for generating the conditional
probability distribution for the desired cognitive
state. Results show that it can be used as a tool to
determine the current cognitive state of any
individual, provided we know their IQ score. In
case of any contradiction between the desired Figure 1: Flow state for IQ level and Task Difficulty
cognitive state (obtained from prior knowledge) For a better learning experience, it is necessary for students
and the observed cognitive state (obtained during to remain in the flow state throughout the session. It is a
testing), the personal insights of a performer is challenging task to create an environment which is
analyzed. enjoyable to a student and attracts complete attention as
well as keep them motivated. To achieve this it is important
1. INTRODUCTION to detect whether the student is in their flow state based on
which we can induce a tailored learning environment.
Cognitive flow is defined as a state of mind which is Present state of the art literature suggests using standard
achieved while a person is performing a task with complete flow state questionnaires for measuring flow state. This
concentration and engagement. This state is closely related method is indirect and might be biased. A more reliable
BMAW 2016 - Page 44 of 59
approach would be to estimate the flow state based on Bassi, M., 2000). The three conditions (Csikszentmihályi,
physiological changes in an individual (Sinha, Aniruddha, M.; Abuhamdeh, S. & Nakamura, J, 2005) that must be
et al, 2015). satisfied to achieve flow are: (a) the task must have a clear
In this paper, our main purpose is to detect the cognitive goal and rate of progress, (b) the task should accompany a
flow with the help of physiological signals and continuous feedback process to help the person maintain
probabilistic reasoning models. In order to do this, a the flow state and (c) the challenge level of the task and the
modified Tower of London game (Schnirman, Geoffrey skill level of the person must be balanced. In the fields of
M., Marilyn C. Welsh, and Paul D. Retzlaff, 1998) is education, detecting the flow state of mind is important to
implemented for various difficulty levels in PEBL: The provide appropriate learning environment for students.
Psychology Experiment Building Language (Mueller, Various researches have been conducted for Flow
Shane T., and Brian J. Piper, 2014). measurement in different domains like piano playing (de
Our work mainly consists of following phases – Manzano, Örjan, et al, 2010), video-game (Kramer, Daniel,
2007), online games (Hsu, Chin-Lung, and Hsi-Peng Lu,
(i) Extraction and analysis of data from various 2004), social networking sites (Mauri, Maurizio, et al,
physiological sensors like Electro-encephalogram (EEG), 2011), e-commerce business (Koufaris, Marios, 2002) etc.
Heart-rate variability (HRV) and Galvanic Skin response
(GSR) while a participant is performing a standard Most of these approaches use indirect methods for
psychometric test of various difficulty levels. measuring flow (Csikszentmihalyi, Mihaly, and Isabella
Selega Csikszentmihalyi, 1992), (Novak, Thomas P., and
(ii) Creating a Bayesian Network (BN) framework with Donna L. Hoffman, 1997), (Nakamura, J. and
the objective to correlate the nodes based on sensor data as Csíkszentmihályi, 2009). The EEG is recently being used
mentioned in previous step with the state nodes by using extensively in the fields of education with the help of Brain
data obtained in step (i). Computer Interface (BCI) technology (van Schaik, Paul,
(iii) Validation of BN based model for test data Stewart Martin, and Michael Vallance, 2012). A greater
left temporal alpha activity is noticed (Kramer, Daniel,
We have used the Bayesian Network based model because 2007) indicating the flow state of the performer in
it provides a theoretically efficient and consistent comparison to the right temporal lobe. The mid beta and
mechanism for processing imprecise and uncertain theta activity also have a distinctive effect on performance
information. Although there has been a growing interest whereas no significant effect was found in the delta
among researchers to use BN in the field of education and waveforms. Higher alpha activity coupled with lower beta
student knowledge evaluation (Chrysafiadi, K. and Virvou, activity to characterize the flow state (Mauri, Maurizio, et
M, 2013), very little work has been done to model EEG and al, 2011). Researchers are attempting to measure boredom,
other physiological sensor data. We have created a model anxiety etc. from EEG signals (Chanel Guillaume et. al,
for our problem domain by representing a high level 2008 and Berta Riccardo et.al, 2013). Recently low cost
probability distribution over a set of random variable devices are being used for analysing the effect of various
denoting the different states which have direct elementary cognitive tasks (Chatterjee, Debatri et al.,
dependencies among themselves. Conditional probability 2015). Some of these works also suggested using other
tables for each node are updated based on the sensor data. physiological responses like GSR and heart rate for
The paper is organized as follows-. In section 2, we have assessing the flow state (Chaouachi, Maher and Claude
given a brief summary of existing approaches adopted for Frasson, 2010). The main problem of using multichannel
analysing flow state and related works. In section 3, we physiological sensors is that, we have to find out an
have described the design of the game, experimental setup, appropriate mechanism for fusing the results obtained from
methodologies for the analysis of sensor data and multiple sensors.
construction of Bayesian network. Section 4, presents the Bayesian network (Pearl, Judea, 1986) is becoming an
analysis of results that we achieved from the experiments. increasingly popular technique to model uncertain and
Finally in section 5, we have concluded and have provided complex domains. Unlike classical statistical models, BN
the future prospect of this work. allow the introduction of prior knowledge into models.
This prevents extraneous data to be considered which
2. RELATED WORK might alter desired results. BN uses the concept of
conditional probability which is proven to be very useful in
The ‘Flow state’ has been described (Csikszentmihályi, applications to the real world problem domain, where
Mihály; Harper & Row, 2015) as an experience achieved probability of occurrence of an event is conditionally
by a person while performing a task and is dependent on dependent on the probability of occurrence of a previous
the personality and ability of the person. A Flow state can event.
only be achieved when a person is engaged in an active
task. Passive tasks such as watching television, taking a Bayesian Network modelling has been used in the areas of
bath etc. do not induce flow experience (Csikszentmihályi, medicine (Koufaris, Marios, 2002), document
M., Larson, R., & Prescott, S, 1977), (DelleFave, A., & classification, information retrieval (Luis M. de Campos,
BMAW 2016 - Page 45 of 59
Juan M. Fernández-Luna and Juan F. Huete, 2004), image where, N is the total number of moves, Ndisk is number of
processing, decision support system (F.J. Díez, J. Mira, E. disks in the game and Nspace and is the available number of
Iturralde and S. Zubillaga, 1997), gaming, bioinformatics empty space.
(Neapolitan, Richard, 2009), gene analysis (Friedman, N.;
Screen shots for three different levels of games are shown
Linial, M.; Nachman, I.; Pe'er, D, 2000) etc.
in Figure 2. The minimum number of moves per disk for
In the present work, BN, a desired state is derived based on the low difficulty game is chosen as two and the total
the static information namely IQ of the individual and number of different disk is two, the complexity for this
difficulty of the task. The estimated state is derived based level of game is 3/(2+4)=0.5. Similarly, the complexities
on the sensor observation while the task is performed. The for the medium difficulty game is 4/(3+3)=0.67 and high
desired and the estimated state may not match as the prior difficulty game is 8/(4+2)=1.33 according to (1). These
characterization of individual in an exact manner is not calculations show that the complexity for each task
possible. In that case, the model can help to decide the task increases with respect to increase in difficulty level. The
level to maintain the flow state of the individual. number of moves and number of disks for various
complexity levels are chosen based on user feedback
collected from 20 participants who also participated in the
data collection.
3. METHODOLOGY
The low difficulty level session consist of 10 set of games
This section explains the game designed and the with 30 sec duration each. Similarly the medium session
methodology adopted for measurement and analysis of consist of 6 set of games each having 30 sec duration. The
physiological signals and creation of BN model based on difficult session consist of 4 set of games with 30 sec
the findings. duration each. For lower complexity game it is usually
finished by most of the participants before 30 seconds,
3.1 GAME DESIGN hence the number of games for various complexity levels
are varied so that the completion time for easy, medium
Tower of London (TOL) is a classical puzzle based game and hard sessions are comparable. Finally, it is found that
used by psychologists for assessment of executive for all the participants the minimum time to complete
functions and planning capabilities of an individual. We among all types (low, medium and high) of tasks is 90
modified the standard TOL-R (Schnirmanetal, 1998) game seconds, hence the corresponding sensor data are
for three levels of difficulties (Difficult, Moderate and considered for further processing.
Easy) in PEBL. For a game session, a target configuration
is shown at the top of the screen as shown in fig. 2. The
goal is to move a pile of disks given at the bottom so that
the given assembly matches the target configuration shown
on the top of the screen. Participants can only move one
disk at a time, and cannot move a disk onto a pile that has
no more room (indicated by the size of the grey rectangle).
Participant has to click on the pile they want to move a disk
off, and it will move up above the piles. Next, they click on
another pile, and the disk will move down to that pile.
There is a time limit to finish each game. Participants are
(a) (b)
instructed to finish each game within the allotted time. If
the participant fails to finish the game within the session
related time, the session ends and a new game starts.
Information like start time of the game, duration, number
of moves, total number of success etc. per session are
stored.
The complexity of each game is determined with the help
of the number of different coloured disks, the minimum
number of moves needed to finish the game and the number
of available empty space among the stacks for the disk to
move around. Hence, the game complexity Gcom is defined
(c)
as,
N Figure 2: Screenshot of the three Tower of London games
Gcom (1)
N disk N space (a) Low (b) Medium and (c) High
BMAW 2016 - Page 46 of 59
3.2 EXPERIMENTAL SETUP 3.2.3 Participant Feedback Form
The standard Game-flow indicator (GFI) feedback form
3.2.1 Participants has been used to assess if a participant experienced
We have selected a group of 20 participants of varying IQ, boredom and flow experiences during the experiments and
from our research lab. The IQ scores are found to vary the findings are used as the reference. This questionnaire
between 80 and 131. The average age of the participants based feedback form described by (Bakker, A. B, 2005)
are 22-30 years. They are all right handed male engineers and (Bakker, A. B., 2008) is ideal to measure level of
belonging to similar socio-economic background. They engagement while playing a game. For doing this, the
had normal or corrected to normal vision. The selection is overall scores for both flow and boredom questionnaires
made to reduce the participant related bias so that the are calculated assuming 1= strongly disagree, 2= disagree,
variation is only in their IQ levels. 3 = undecided, 4 = agree and 5 = strongly agree. Different
questions are asked regarding the experience of
3.2.2 Data Collection participants while playing the game.
An in house python based data capture tool is used for the The participants are asked to fill up three feedback forms,
data collection. The capture tool allows participants to play one for each of the three games, immediately after the end
the game on a standard 17 inch computer screen placed at of each session. This feedback is necessary as they provide
a viewing distance of approximately 25 inch and a ground truth for cognitive flow along with the game data.
simultaneously allows to collect the EEG, GSR and We have used the feedback forms to analyze the
Photoplethysmogram (PPG) signals. Participants are asked contradictions which we have seen during the Bayesian
to play the game while wearing a single lead EEG device Network analysis as explained in section 3.4.
from Neurosky1. It is a dry sensor with a lead placed in FP1
position and the grounding is done with a clip fixed to left 3.3 SENSOR DATA ANALYSIS
earlobe. We have used a GSR device from eSense 2 to
record the variations in skin conductance level. All the 3.3.1 Skill-challenge analysis using EEG signals
participants are right handed and wore the GSR sensors on
In the present work, we have experimented with various
the middle and index fingers of the left hand. The right
frequency band energies described by (W. Klimesch,
hand is kept completely free so that they can play the game
1999), (H. Sijuan, 2010) and time domain Hjorth
comfortably using mouse. The oxygen saturation level and
parameters as described by (Gudmundsson, Runarsson,
the pulse rate are measured by the pulse oximeter from
Sigurdsson, Eiriksdottir, Johnsen, 2007), (V. Carmen, et al.
Contec3, through the left ring finger. The devices used are
2009) as shown in (2) and (3).
shown in Figure 3. The participants are asked to play three
session of the game (High, Low and Medium) with a F {E G , E T , E D , E lE , E mE , H a , H m , H c } (2)
resting period of 5 min between each game. For half of the The first five features are the energies in various frequency
participants the order of the game followed is high, bands namely, delta ( E G as 0.5 - 4 Hz), theta ( E T as 4 – 7.5
medium, low and for remaining half low, medium, high Hz), alpha ( E D as 7.5 – 12.5 Hz), low-beta ( E lE as 12.5 –
sequence is followed. During the game, scores, number of 16 Hz) and mid-beta ( E mE as 16 – 20 Hz) respectively. The
moves, duration etc. are also recorded for further analysis. energies in each band are extracted using Welch’s power
spectral density as defined by (Welch, Peter, 1967). The
last three features in (2) are the Hjorth parameters namely
Activity ( H a ), Mobility ( H m ) and Complexity ( H c )
respectively as given in (3).
dx(t ) dx(t )
(a) (b) Ha( ) H m( )
Ha var( x(t )) H m dt Hc dt (3)
a m
H ( x(t )) H ( x(t ))
Here x(t) indicates the time domain signal in a window of
(c) duration 1 sec and dx(t ) is the first order derivative of the
dt
signal.
Figure 3: Data collection devices: (a) Neurosky EEG
device (b) GSR device from eSense (c) Pulse oxymeter
from Contec 3.3.2 Analysis of GSR signal
The galvanic skin response (GSR) is the electro-dermal
response where the skin conductance changes with the
1 3
http://neurosky.com http://www.coopermedical.com/overnight-pulse-ox/cms-50d-plus-
2
https://www.mindfield.de/en/biofeedback/products/esense/esense-skin- recording-fingertip-pulse-oximeter.html
response
BMAW 2016 - Page 47 of 59
amount of secretion from the sweat glands in presence of 3.4 BAYESIAN NETWORK CONSTRUCTION
stressful, likeable events. Therefore GSR can be used as a
good predictor of concentration, mental workload etc. in Our problem statements for creating the Bayesian Network
flow study (Nourbakhsh, Nargess, et al, 2012). During flow are as follows -
experience, subjects should experience a higher x A number of random participants are performing a
concentration and focus on the task. This can be measured task in a controlled environment.
using the GSR. x The task can be of three difficulty levels: easy,
moderate or hard.
The GSR device consists of two electrodes which applies a
x The participants are categorized based on their
constant voltage to the skin. The current which then flows
intelligence levels i.e. the IQ scores.
through the skin can be detected. The GSR signal consists
of two components: phasic, which is the fast varying x The participants can be in either of three states: flow,
component and tonic, which is the slow component. Both boredom or anxiety.
contain important information associated with specific x Their performance and feedback are recorded to
physiological aspect of the mental states. The tonic determine their mental state and experience.
component (Tc) is calculated only by taking the inverse x During data capture, three physiological sensors
transform of first few Fourier coefficients (4) and the namely EEG sensor, HRV sensor and GSR sensor are
phasic component (Pc) is calculated by inversing the higher used.
order coefficient of the Fourier coefficients (5).The eSense
GSR sensor has 5Hz sampling frequency hence we have Now a Bayesian Network (BN) framework is created with
used first 4 components (0<=k<=3) of the Fourier the objective to diagnose and investigate the different
coefficients which corresponds to 0.5Hz. The IFFT in (4) relationships between the sensors nodes and the state nodes
and (5) corresponds to Inverse Fast Fourier Transform.The as shown in Figure 4.
tonic component (Tc) of the GSR signal is computed for
windows of duration 1 second. Next to investigate the
distribution Tc over the task interval, we compute the mean
and the kurtosis as given in (6) using N successive
windows for each task (low, medium, high). These
parameters provide the statistical information on the way
the skin conductance changes for an individual over time,
for a given task.
N 1 2S
j( ) nk
Tc IFFT (¦ x(n).e N
) , k 0,1,2,3 (4)
n 0
N 1 2S
j( ) nk
Pc IFFT (¦ x(n).e N
) , k 4,5,...N 1 (5)
n 0
Tcmean mean(Tc (n)) , Tckurt kurtosis(Tc (n)) , 1 d n d N (6)
Figure 4: Proposed Bayesian Network
3.3.3 Calculation of HRV from PPG signals The BN consists of three types of nodes:
Stress level can be evaluated using Heart rate variability (J. Evidence nodes: Task Difficulty (TD), Intelligence
Taelman, S. Vandeput, A. Spaepen and S. Van Huffel, Quotient (IQ)
2009), (McDuff, Daniel, Sarah Gontarek, and Rosalind
Picard, 2014). When the task challenge is low compared to Sensor nodes: GSR, GSR, HRV
the subjects’ skill level, then the heart rate variability is State nodes: Emotional State (O), Desired State (D), Final
high compared to the flow state where task challenge Outcome (F)
matches the skill. We have used the SPO2 device wearable
on the ring finger for sensing the Photoplethysmogram Through in the Evidence nodes we can input the static
(PPG) signal. We have calculated three time domain HRV knowledge or evidence which we have in our problem
parameters namely 1) rMSSD (root mean square of domain. In our case, we know beforehand, the IQ level of
successive differences between adjacent NN intervals), 2) the participants and also the difficulty levels of task they
SDSD (Standard deviation of successive differences are given. The Evidence nodes serve as parents to the
between adjacent NN intervals), 3) SDNN (Successive Sensor nodes, since the sensor readings are a direct causal
difference between NN intervals) as explained in (McDuff, effect of the two conditions (IQ & TD). Based on the
2014). Out of these SDNN is found to give a better results obtained from sensors, we can predict the current
indication of the stress level for our experiment. cognitive state of the participants. The derived state i.e. the
Emotional state (O) from the sensor reading is compared
with the Desired state (achieved from ground truth during
training) and finally a conclusion (F) is drawn based on
BMAW 2016 - Page 48 of 59
these two states. We have used a separate desired state in Figure 6(a) and Figure 6(b). The line representing the
order to compare it to the results derived from the signals medium difficulty game for all the subjects is found to
about what emotional condition the subject is currently in. overlap over the other two; the separation between the
medium games across participants is not consistent. Hence
We have created the Bayesian Network in SamIam4, a
for better representation, we have only plotted the GSR for
comprehensive tool for modelling and reasoning with
high and low difficulty games.
Bayesian networks, developed in Java by the Automated
Reasoning Group of Professor Adnan Darwiche, UCLA.
At the beginning of the experiment, the participants are
categorized based on their IQ levels measured by a
standardized IQ5 test prior to playing the game. This IQ test
is a free online test from Brainmetrix.
The IQ level and the task difficulty level serve as evidence
nodes in the BN. Depending on the state of these two
nodes, the sensor readings vary. Hence all the sensor nodes
have them as immediate parents. The readings of all three
sensors are used to analysis the emotion state of the
participants and are compared to the ground truth i.e. the
desired state. If two state matches, then we can conclude
the actual state via the final outcome and that the Bayesian (a)
Network is expected to give the correct outcome. If not,
then there is a case of contradiction caused by one of more
sensors giving faulty state. In case of a contradiction
between observed state and desired state the BN model can
be used to resolve this conflict by reasoning between the
various nodes using probabilistic queries.
4. RESULTS AND DISCUSSIONS
4.1 EEG SIGNAL ANALYSIS
Various features of the EEG signals namely alpha, beta, (b)
theta, delta, attention, meditation, Hjorth etc. for all the
experiments for all the participants are extracted using (2) Figure 5: Separation between Low, Medium, High
and (3). Among all features the Activity measure of Hjorth difficulty task for the Hjorth calculation of EEG signals
parameter is found to be indicative of variations of brain for (a) 20 subjects and (b) 16 subjects
signals with difficulty level. The raw EEG signal is used Out of 20 participants 8 participants (6, 7, 8, 9, 10, 11, 12
for calculating activity for windows of duration 1 sec and and 20) played the game in high-medium-low sequence
the overall mean is taken for each session of the game of and the remaining participants played in low-medium-high
all the participants. The results for different difficulty sequence. It is evident from the plots that kurtosis performs
levels (Low, Medium and High) are combined separately better than mean of the tonic power in separating the low
for all the 20 participants and compared as shown in Figure and high difficulty tasks.
5(a). We have found good amount of separation between
three different tasks for 16 out of 20 participants. The
comparison for these selected participants are shown in
Figure 5(b).
4.1 GSR SIGNAL ANALYSIS
The GSR data is subdivided into a number of windows of
duration 1 sec. Next we calculate both tonic and phasic
power using (4) and (5).The phasic power does not show
sufficient separation between different task levels whereas
the tonic gives good separation. The mean and kurtosis of
the tonic power is calculated using (6). The plot of mean
and kurtosis of tonic component for GSR are shown in (a)
4 5
http://reasoning.cs.ucla.edu/samiam/ http://www.brainmetrix.com/free-iq-test/
BMAW 2016 - Page 49 of 59
occurrence over all possible conditions. The scores of IQ
less than 89 are treated as Low IQ, scores between 90 and
109 are treated as Medium IQ and scores with 110 is treated
as High IQ. In the present study we got 6 participants
having high IQ, 9 participants having medium IQ and 5
participants having low IQ.
We have used the data for 18 out of 20 participants to train
the BN. The sensor data for all the training participants (18
subjects) are classified into three levels (Low, Medium,
High) based on their observations. An example is shown
in Table 1 where the levels of the EEG sensor data are
shown for 5 training participants with High IQ, playing
three levels of games. It can be seen that for the easy game
(TD = low), all of them have low EEG feature values, for
(b) medium game (TD = medium), one of them has high EEG
feature and similarly for difficult game (TD = high). The
Figure 6: Separation between the (a) Mean and (b) probability of occurrence of each level is calculated and
Kurtosis of the Tonic (µS) component of GSR during the updated in the respective Conditional Probability Table
Low and High difficulty games. (CPT) of the BN. After the training the final CPT for the
EEG node is shown in Table 2. The same rule is followed
4.2 PPG SIGNAL ANALYSIS for the rest of the sensor nodes as well.
The PPG signal analysis does not give a clear, consistent Table 1: The levels for EEG Sensor data (Mean of Hjorth -
separation between difficulty levels of the task for all the Activity) for 5 Participants with High IQ and playing three
participants. We have plotted the successive difference levels of game (TD). Here L-Low, M-Medium, H-High.
between NN intervals (SDNN) in Figure 7. For 10 out of
TD EEG level (Mean of Hjorth - Activity)
20 subjects there is a separation between the two SDNN
values out of which only 5 have the value for the low game L L L L L L
lower than the high game. Hence any substantial
information cannot be derived from this result of PPG M M H M M M
analysis. H H M H H H
Table 2: Conditional probabilities of the EEG node is
shown for the pair (TD, IQ)
EEG (L,H) (M,H) (H,H) (L,M) (M,M) (H,M)
L 0.98 0.02 0.02 0.98 0.01 0.01
M 0.01 0.79 0.19 0.01 0.98 0.01
H 0.01 0.19 0.79 0.01 0.01 0.98
EEG (L,L) (M,L) (H,L)
Figure 7: Separation of the HRV(SDNN) in msec for all L 0.5 0.25 0.25
the 20 subjects between the high and low difficulty game M 0.25 0.5 0.25
4.3 BAYESIAN NETWORK ANALYSIS H 0.25 0.25 0.5
The Bayesian network framework, given in Figure 4, has The data for the remaining 2 participants (having high IQ
been trained based on the results obtained from and low IQ) are used to validate whether the BN nodes are
physiological sensor data collected during the experiments. providing the correct results. Different combinations of
The conditional probability tables associated with each evidences for these two subjects are checked in the BN
node have been updated according to the occurrence of (Figure 8 through Figure 11) and the reasons for
their respective parent node. contradictions are explained. The states of the BN are
Given a participant of a particular IQ level and solving a shown as rectangular blocks containing the percentage
game of a particular difficulty level, we have collected each equivalent of the probability values of the random
of the sensor data and have calculated their probability of variables.
BMAW 2016 - Page 50 of 59
Case 1: A participant of high IQ is playing a high difficulty state without being too anxious and matches with the
level game Emotional State of the BN. However a contradiction is
shown in the Final Outcome indicating a deviation of the
behaviour of the participant from the expected one.
Figure 8: Bayesian Network in Query mode for the
Figure 9: Bayesian Network in Query mode for the
evidence Task Difficulty= High and IQ=Expert/High evidence Task Difficulty=Low and IQ=Expert/High
In this case we can see that the Emotional State (O)
matches with the Desired State (D) hence the final outcome
is Flow state. Also all the sensors are providing the
expected result, except the EEG feature value. The subject
used in this experiment has high IQ. From the
questionnaire based survey we have found that the
participants has reported to be sometimes in anxious state
during the high difficulty game. This could be the reason
for the EEG feature value to be very high for him. We have
found from participant’s feedback that he was mostly in the
flow state. Hence the output of the Bayesian model seems
to be correlating with the participant’s feedback.
Case 2: A participant of high IQ is playing a low difficulty
level task
In this case we can see that if a participant with high IQ is
playing a low difficulty task, the sensor nodes are
providing the desired states. This is shown in Figure 9. The
EEG feature value and GSR states are in low state as
expected, as the participant is expected to be in a relaxed Figure 10: Bayesian Network in Query mode for the
state. However, the HRV indicates a low state as opposed evidence Task Difficulty= High and IQ=Poor/Low
to the expected high state leading to the contradiction. If a Case 4: A participant of low IQ is playing a low difficulty
highly intelligent participant is playing a very easy task level task
with low concentration (GSR=Low) then he / she might be
in a restless state which can lead to anxiety. Hence the true In this case we can see that all the states are providing the
mental state cannot be correctly determined. expected result except the EEG feature value as shown in
Figure 11. The participants used in this experiment has low
Case 3: A participant of low IQ is playing a high difficulty IQ. We have found from participants’ feedback that he was
level task in the relaxed state and did not find the game too difficult.
In this case we can see that all the states are providing the This could be the reason for the EEG feature value to be
expected result except the EEG feature value as shown in low but with very low percentage. It can also be observed
Figure 10. The participant in this experiment has low IQ. from all the four cases that if the contradiction option is
According to participant’s feedback, he was in the flow ignored in the final outcome state, then the next highest
probability belongs to the state same as the desired state.
BMAW 2016 - Page 51 of 59
This shows that while the Bayesian Network provides the Acknowledgements
correct prediction of the actual mental state, given the
sensor data and static knowledge, it also hints at the We are thankful to the participants who have provided their
possibility of a faulty sensor data based on the probability time and inputs for the data collection for the experiments.
of the contradiction.
References
Bakker, A. B. (2005). “Flow among music teachers and
their students: The crossover of peak experiences”. Journal
of Vocational Behavior, 66, 26–44. Retrieved from http://
dspace.library.uu.nl/handle/1874/10711.
Bakker, A. B. (2008). “The work-related flow inventory:
Construction and initial validation of the WOLF”. Journal
of Vocational Behavior, 72(3), 400–414.
Berta, Riccardo, et al. (2013). "Electroencephalogram and
physiological signal analysis for assessing flow in
games." Computational Intelligence and AI in Games,
IEEE Transactions on 5.2: 164-175.
V. Carmen, et al. (2009). “Time Domain Parameters as a
feature for EEG-based Brain-Computer Interfaces” in
Neural Networks 22: 1313-1319
Chanel, Guillaume, et al. (2008). "Boredom, engagement
and anxiety as indicators for adaptation to difficulty in
games." Proceedings of the 12th international conference
Figure 11: Bayesian Network in Query mode for the
on Entertainment and media in the ubiquitous era. ACM.
evidence Task Difficulty= Low and IQ=Poor/Low
Chaouachi, Maher, and Claude Frasson. (2010).
5. CONCLUSION AND FUTURE WORK "Exploring the relationship between learner EEG mental
engagement and affect." Intelligent tutoring systems.
In this work we have measured the cognitive state of a Springer Berlin Heidelberg.
participant performing task of varying difficulty level from
brain signals and certain physiological parameters like Chatterjee, Debatri, et al. (2015). "Analyzing elementary
heart rate variability and galvanic skin response. Results cognitive tasks with Bloom's taxonomy using low cost
show that EEG (Hjorth-Activity) and GSR signals can be commercial EEG device." Tenth International Conference
successfully used to distinguish the performance for all the on Intelligent Sensors, Sensor Networks and Information
participants in three different level tasks. On the other Processing (ISSNIP). IEEE.
hand, the analysis of the HRV data is not providing Chrysafiadi, K. and Virvou, M. (2013). “Student modelling
consistent information. These results can be fused together approaches: A literature review for the last decade.” Expert
and analyzed along with the performance and feedback Systems with Applications, 40(11): 4715-4722.
data to get further insights of the mental state for the
participants. Csíkszentmihályi, Mihály, (1990). “Flow: The Psychology
of Optimal Experience”, New York: Harper and Row,
From our proposed Bayesian Network we need to ISBN 0-06-092043-2.
determine whether there is a contradiction between
observed and desired state. In case of contradiction we Csikszentmihalyi, Mihaly. (1997). "Finding flow.".
need to find out whether there are any sensor(s) nodes Csikszentmihalyi, M. (1999). “If we are so rich, why aren't
giving faulty data. In future a large number of participants we happy?” American Psychologist. Vol 54(10), 821-827.
are to be considered for further analysis. Moreover, the
workflow of the Bayesian Network should be dynamic for Csikszentmihályi, Mihály; Harper & Row. (2015).
handling a large number of data. Several knowledge under "FLOW: The Psychology of Optimal Experience"
uncertainty can be determined in future by using such Retrieved 2 April 2015.
probabilistic graphical models. In the area of education, Csikszentmihalyi, Mihaly, and Isabella Selega
models can be designed for generating a sequence of Csikszentmihalyi, eds. (1992). “Optimal experience:
questions which are intermixed in difficulty level to Psychological studies of flow in consciousness.”
analyze the variation in cognitive states of an individual Cambridge University Press.
during the assessment process.
Csikszentmihályi, M., Larson, R., & Prescott, S. (1977).
“The ecology of adolescent activity and experience.”
Journal of Youth and Adolescence, 6, 281-294.
BMAW 2016 - Page 52 of 59
Csikszentmihályi, M.; Abuhamdeh, S. & Nakamura, J. test battery." Journal of neuroscience methods 222: 250-
(2005), "Flow", in Elliot, A., Handbook of Competence 259.
and Motivation, New York: The Guilford Press, pp. 598–
Nakamura, J. and Csíkszentmihályi, M. (2009). “Flow
698.
Theory and Research.” In Snyder, C. R. and Lopez, S. J.
deManzano, Örjan, et al. (2010). "The psychophysiology eds. Oxford handbook of Positive Psychology.Oxford
of flow during piano playing." Emotion 10.3: 301. University Press, Oxford, 195-206.
DelleFave, A., & Bassi, M. (2000). “The quality of Neapolitan, Richard (2009). Probabilistic Methods for
experience in adolescents’ daily lives: Developmental Bioinformatics. Burlington, MA: Morgan Kaufmann.
perspectives.” Genetic, Social, and General Psychology p. 406. ISBN 9780123704764.
Monographs, 126, 347-367.
Nourbakhsh, Nargess, et al. (2012). "Using galvanic skin
F.J. Díez, J. Mira, E. Iturralde and S. Zubillaga (1997). response for cognitive load measurement in arithmetic and
"DIAVAL, a Bayesian expert system for reading tasks." Proceedings of the 24th Australian
echocardiography". Artificial Intelligence in Medicine Computer-Human Interaction Conference. ACM.
(Elsevier) 10 (1): 59–73.
Novak, Thomas P., and Donna L. Hoffman. (1997).
Friedman, N.; Linial, M.; Nachman, I.; Pe'er, D. (2000). "Measuring the flow experience among web users."
"Using Bayesian Networks to Analyze Expression Data". Interval Research Corporation 31.
Journal of Computational Biology 7 (3–4): 601–620.
Pearl, Judea. (1986). "Fusion, propagation, and structuring
S. Gudmundsson, P. Runarsson, T. Sigurdsson, S. in belief networks." Artificial intelligence 29.3: 241-288.
Eiriksdottir, K. Johnsen, (2007). “Reliability of
Pearl, Judea, and Stuart Russell. (1998). “Bayesian
quantitative EEG features”, in Clinical Neurophysiology
networks.” Computer Science Department, University of
118: 21622171, August 2007.
California.
Hsu, Chin-Lung, and Hsi-Peng Lu. (2004). "Why do
Schnirman, Geoffrey M., Marilyn C. Welsh, and Paul D.
people play on-line games? An extended TAM with social
Retzlaff. (1998). "Development of the Towerof London-
influences and flow experience." Information &
revised." Assessment 5.4: 355-360.
Management 41.7: 853-868.
Schnirman, G.M., Welsh, M.C., Retzlaff, P.D. (1998).
W. Klimesch, (1999). “EEG alpha and Theta oscillations
“Development of the Tower of London-Revised.”
reflect cognitive and memory performance: a review and
Assessment 5(4), 355 –360.
analysis” in Brain Research Reviews 29.
H. Sijuan, (2010). “Feature extraction and classification of
Koufaris, Marios. (2002). "Applying the technology
EEG for imagery movement based on mu/beta rhythms”,
acceptance model and flow theory to online consumer
3rd International Conference on Biomedical Engineering
behavior." Information systems research 13.2: 205-223
and Informatics (BMEI).
Kramer, Daniel. (2007). "Predictions of performance by
Sinha, Aniruddha, et al. (2015) "Dynamic assessment of
EEG and skin conductance." Indiana Undergraduate
learners' mental state for an improved learning
Journal of Cognitive Science 2: 3-13.
experience." Frontiers in Education Conference (FIE),
Luis M. de Campos, Juan M. Fernández-Luna and Juan F. 2015. 32614. IEEE.
Huete (2004). "Bayesian networks and information
J. Taelman, S. Vandeput, A. Spaepen and S. Van Huffel,
retrieval: an introduction to the special issue". Information
(2009). "Influence of Mental Stress on Heart Rate and
Processing & Management (Elsevier) 40 (5): 727–733.
Heart Rate Variability", 4th European Conference of the
Mauri, Maurizio, et al. (2011). "Why is Facebook so International Federation for Medical and Biological
successful? Psychophysiological measures describe a core Engineering, vol. 22, pp. 1366-1369.
flow state while using Facebook." Cyberpsychology,
Van Schaik, Paul, Stewart Martin, and Michael Vallance.
Behavior, and Social Networking 14.12: 723-731
(2012). "Measuring flow experience in an immersive
McDuff, Daniel, Sarah Gontarek, and Rosalind Picard. virtual environment for collaborative learning." Journal of
(2014). "Remote measurement of cognitive stress via heart Computer Assisted Learning 28.4: 350-365.
rate variability." Engineering in Medicine and Biology
Welch, Peter. (1967). "The use of fast Fourier transform for
Society (EMBC), 36th Annual International Conference of
the estimation of power spectra: a method based on time
the IEEE.
averaging over short, modified periodograms." IEEE
Mueller, Shane T., and Brian J. Piper. (2014). "The Transactions on audio and electroacoustics: 70-73.
psychology experiment building language (pebl) and pebl
BMAW 2016 - Page 53 of 59