A Probabilistic Approach for Detection and Analysis of Cognitive Flow Debatri Chatterjee, Aniruddha Sinha, Meghamala Sinha Sanjoy Kumar Saha TCS Research, Tata Consultancy Services Ltd. Computer Science & Engineering, Kolkata, India Jadavpur University, Email: {debatri.chatterjee, aniruddha.s}@tcs.com, Kolkata, India meghamala.sinha@gmail.com Email: sks_ju@yahoo.co.in Abstract to the balance between the challenge of any task and the skill of the person executing the task. If the task difficulty is low compared to the held skill level of an individual, the A performer may undergo a task with varying person tends to be in the bored state (Csíkszentmihályi, difficulty level. It is important to know the mental Mihály, 1990). Alternatively, if the task difficulty is high, state in order to maintain the optimum level of and the skill level is low, the person is supposed to be in performance. The mental state of an individual the anxiety state. However, if the skill and task difficulty varies according to their IQ levels, task level matches, the person enter Flow states i.e. a state of difficulties or other psychological or focused concentration with a sense of enjoyment environmental reasons. We have tried to measure (Csíkszentmihályi, Mihály, 1997), (Csíkszentmihályi, the cognitive state of individuals, while they are Mihály, 1999). In our case, the skill level of an individual performing tasks of various complexity levels, is directly related to his or her IQ level and is treated as the using physiological responses like brain prior knowledge of the individual. On the other hand, the activation, heart rate variability and galvanic skin challenge of the task is synonymous to the task difficulty response. In this paper we have proposed a level. Hence the flow-boredom state can be represented as Bayesian network based model to shown in Figure 1. probabilistically evaluate the cognitive state of an individual from the difficulty levels of the tasks, IQ level of the individual and observations made using the physiological sensing. Twenty subjects with various IQ levels are asked to play a modified Tower of London (TOL) game having three complexity levels: low, medium and high. The sensor data collected have been used to train the Bayesian model for generating the conditional probability distribution for the desired cognitive state. Results show that it can be used as a tool to determine the current cognitive state of any individual, provided we know their IQ score. In case of any contradiction between the desired Figure 1: Flow state for IQ level and Task Difficulty cognitive state (obtained from prior knowledge) For a better learning experience, it is necessary for students and the observed cognitive state (obtained during to remain in the flow state throughout the session. It is a testing), the personal insights of a performer is challenging task to create an environment which is analyzed. enjoyable to a student and attracts complete attention as well as keep them motivated. To achieve this it is important 1. INTRODUCTION to detect whether the student is in their flow state based on which we can induce a tailored learning environment. Cognitive flow is defined as a state of mind which is Present state of the art literature suggests using standard achieved while a person is performing a task with complete flow state questionnaires for measuring flow state. This concentration and engagement. This state is closely related method is indirect and might be biased. A more reliable BMAW 2016 - Page 44 of 59 approach would be to estimate the flow state based on Bassi, M., 2000). The three conditions (Csikszentmihályi, physiological changes in an individual (Sinha, Aniruddha, M.; Abuhamdeh, S. & Nakamura, J, 2005) that must be et al, 2015). satisfied to achieve flow are: (a) the task must have a clear In this paper, our main purpose is to detect the cognitive goal and rate of progress, (b) the task should accompany a flow with the help of physiological signals and continuous feedback process to help the person maintain probabilistic reasoning models. In order to do this, a the flow state and (c) the challenge level of the task and the modified Tower of London game (Schnirman, Geoffrey skill level of the person must be balanced. In the fields of M., Marilyn C. Welsh, and Paul D. Retzlaff, 1998) is education, detecting the flow state of mind is important to implemented for various difficulty levels in PEBL: The provide appropriate learning environment for students. Psychology Experiment Building Language (Mueller, Various researches have been conducted for Flow Shane T., and Brian J. Piper, 2014). measurement in different domains like piano playing (de Our work mainly consists of following phases – Manzano, Örjan, et al, 2010), video-game (Kramer, Daniel, 2007), online games (Hsu, Chin-Lung, and Hsi-Peng Lu, (i) Extraction and analysis of data from various 2004), social networking sites (Mauri, Maurizio, et al, physiological sensors like Electro-encephalogram (EEG), 2011), e-commerce business (Koufaris, Marios, 2002) etc. Heart-rate variability (HRV) and Galvanic Skin response (GSR) while a participant is performing a standard Most of these approaches use indirect methods for psychometric test of various difficulty levels. measuring flow (Csikszentmihalyi, Mihaly, and Isabella Selega Csikszentmihalyi, 1992), (Novak, Thomas P., and (ii) Creating a Bayesian Network (BN) framework with Donna L. Hoffman, 1997), (Nakamura, J. and the objective to correlate the nodes based on sensor data as Csíkszentmihályi, 2009). The EEG is recently being used mentioned in previous step with the state nodes by using extensively in the fields of education with the help of Brain data obtained in step (i). Computer Interface (BCI) technology (van Schaik, Paul, (iii) Validation of BN based model for test data Stewart Martin, and Michael Vallance, 2012). A greater left temporal alpha activity is noticed (Kramer, Daniel, We have used the Bayesian Network based model because 2007) indicating the flow state of the performer in it provides a theoretically efficient and consistent comparison to the right temporal lobe. The mid beta and mechanism for processing imprecise and uncertain theta activity also have a distinctive effect on performance information. Although there has been a growing interest whereas no significant effect was found in the delta among researchers to use BN in the field of education and waveforms. Higher alpha activity coupled with lower beta student knowledge evaluation (Chrysafiadi, K. and Virvou, activity to characterize the flow state (Mauri, Maurizio, et M, 2013), very little work has been done to model EEG and al, 2011). Researchers are attempting to measure boredom, other physiological sensor data. We have created a model anxiety etc. from EEG signals (Chanel Guillaume et. al, for our problem domain by representing a high level 2008 and Berta Riccardo et.al, 2013). Recently low cost probability distribution over a set of random variable devices are being used for analysing the effect of various denoting the different states which have direct elementary cognitive tasks (Chatterjee, Debatri et al., dependencies among themselves. Conditional probability 2015). Some of these works also suggested using other tables for each node are updated based on the sensor data. physiological responses like GSR and heart rate for The paper is organized as follows-. In section 2, we have assessing the flow state (Chaouachi, Maher and Claude given a brief summary of existing approaches adopted for Frasson, 2010). The main problem of using multichannel analysing flow state and related works. In section 3, we physiological sensors is that, we have to find out an have described the design of the game, experimental setup, appropriate mechanism for fusing the results obtained from methodologies for the analysis of sensor data and multiple sensors. construction of Bayesian network. Section 4, presents the Bayesian network (Pearl, Judea, 1986) is becoming an analysis of results that we achieved from the experiments. increasingly popular technique to model uncertain and Finally in section 5, we have concluded and have provided complex domains. Unlike classical statistical models, BN the future prospect of this work. allow the introduction of prior knowledge into models. This prevents extraneous data to be considered which 2. RELATED WORK might alter desired results. BN uses the concept of conditional probability which is proven to be very useful in The ‘Flow state’ has been described (Csikszentmihályi, applications to the real world problem domain, where Mihály; Harper & Row, 2015) as an experience achieved probability of occurrence of an event is conditionally by a person while performing a task and is dependent on dependent on the probability of occurrence of a previous the personality and ability of the person. A Flow state can event. only be achieved when a person is engaged in an active task. Passive tasks such as watching television, taking a Bayesian Network modelling has been used in the areas of bath etc. do not induce flow experience (Csikszentmihályi, medicine (Koufaris, Marios, 2002), document M., Larson, R., & Prescott, S, 1977), (DelleFave, A., & classification, information retrieval (Luis M. de Campos, BMAW 2016 - Page 45 of 59 Juan M. Fernández-Luna and Juan F. Huete, 2004), image where, N is the total number of moves, Ndisk is number of processing, decision support system (F.J. Díez, J. Mira, E. disks in the game and Nspace and is the available number of Iturralde and S. Zubillaga, 1997), gaming, bioinformatics empty space. (Neapolitan, Richard, 2009), gene analysis (Friedman, N.; Screen shots for three different levels of games are shown Linial, M.; Nachman, I.; Pe'er, D, 2000) etc. in Figure 2. The minimum number of moves per disk for In the present work, BN, a desired state is derived based on the low difficulty game is chosen as two and the total the static information namely IQ of the individual and number of different disk is two, the complexity for this difficulty of the task. The estimated state is derived based level of game is 3/(2+4)=0.5. Similarly, the complexities on the sensor observation while the task is performed. The for the medium difficulty game is 4/(3+3)=0.67 and high desired and the estimated state may not match as the prior difficulty game is 8/(4+2)=1.33 according to (1). These characterization of individual in an exact manner is not calculations show that the complexity for each task possible. In that case, the model can help to decide the task increases with respect to increase in difficulty level. The level to maintain the flow state of the individual. number of moves and number of disks for various complexity levels are chosen based on user feedback collected from 20 participants who also participated in the data collection. 3. METHODOLOGY The low difficulty level session consist of 10 set of games This section explains the game designed and the with 30 sec duration each. Similarly the medium session methodology adopted for measurement and analysis of consist of 6 set of games each having 30 sec duration. The physiological signals and creation of BN model based on difficult session consist of 4 set of games with 30 sec the findings. duration each. For lower complexity game it is usually finished by most of the participants before 30 seconds, 3.1 GAME DESIGN hence the number of games for various complexity levels are varied so that the completion time for easy, medium Tower of London (TOL) is a classical puzzle based game and hard sessions are comparable. Finally, it is found that used by psychologists for assessment of executive for all the participants the minimum time to complete functions and planning capabilities of an individual. We among all types (low, medium and high) of tasks is 90 modified the standard TOL-R (Schnirmanetal, 1998) game seconds, hence the corresponding sensor data are for three levels of difficulties (Difficult, Moderate and considered for further processing. Easy) in PEBL. For a game session, a target configuration is shown at the top of the screen as shown in fig. 2. The goal is to move a pile of disks given at the bottom so that the given assembly matches the target configuration shown on the top of the screen. Participants can only move one disk at a time, and cannot move a disk onto a pile that has no more room (indicated by the size of the grey rectangle). Participant has to click on the pile they want to move a disk off, and it will move up above the piles. Next, they click on another pile, and the disk will move down to that pile. There is a time limit to finish each game. Participants are (a) (b) instructed to finish each game within the allotted time. If the participant fails to finish the game within the session related time, the session ends and a new game starts. Information like start time of the game, duration, number of moves, total number of success etc. per session are stored. The complexity of each game is determined with the help of the number of different coloured disks, the minimum number of moves needed to finish the game and the number of available empty space among the stacks for the disk to move around. Hence, the game complexity Gcom is defined (c) as, N Figure 2: Screenshot of the three Tower of London games Gcom (1) N disk  N space (a) Low (b) Medium and (c) High BMAW 2016 - Page 46 of 59 3.2 EXPERIMENTAL SETUP 3.2.3 Participant Feedback Form The standard Game-flow indicator (GFI) feedback form 3.2.1 Participants has been used to assess if a participant experienced We have selected a group of 20 participants of varying IQ, boredom and flow experiences during the experiments and from our research lab. The IQ scores are found to vary the findings are used as the reference. This questionnaire between 80 and 131. The average age of the participants based feedback form described by (Bakker, A. B, 2005) are 22-30 years. They are all right handed male engineers and (Bakker, A. B., 2008) is ideal to measure level of belonging to similar socio-economic background. They engagement while playing a game. For doing this, the had normal or corrected to normal vision. The selection is overall scores for both flow and boredom questionnaires made to reduce the participant related bias so that the are calculated assuming 1= strongly disagree, 2= disagree, variation is only in their IQ levels. 3 = undecided, 4 = agree and 5 = strongly agree. Different questions are asked regarding the experience of 3.2.2 Data Collection participants while playing the game. An in house python based data capture tool is used for the The participants are asked to fill up three feedback forms, data collection. The capture tool allows participants to play one for each of the three games, immediately after the end the game on a standard 17 inch computer screen placed at of each session. This feedback is necessary as they provide a viewing distance of approximately 25 inch and a ground truth for cognitive flow along with the game data. simultaneously allows to collect the EEG, GSR and We have used the feedback forms to analyze the Photoplethysmogram (PPG) signals. Participants are asked contradictions which we have seen during the Bayesian to play the game while wearing a single lead EEG device Network analysis as explained in section 3.4. from Neurosky1. It is a dry sensor with a lead placed in FP1 position and the grounding is done with a clip fixed to left 3.3 SENSOR DATA ANALYSIS earlobe. We have used a GSR device from eSense 2 to record the variations in skin conductance level. All the 3.3.1 Skill-challenge analysis using EEG signals participants are right handed and wore the GSR sensors on In the present work, we have experimented with various the middle and index fingers of the left hand. The right frequency band energies described by (W. Klimesch, hand is kept completely free so that they can play the game 1999), (H. Sijuan, 2010) and time domain Hjorth comfortably using mouse. The oxygen saturation level and parameters as described by (Gudmundsson, Runarsson, the pulse rate are measured by the pulse oximeter from Sigurdsson, Eiriksdottir, Johnsen, 2007), (V. Carmen, et al. Contec3, through the left ring finger. The devices used are 2009) as shown in (2) and (3). shown in Figure 3. The participants are asked to play three session of the game (High, Low and Medium) with a F {E G , E T , E D , E lE , E mE , H a , H m , H c } (2) resting period of 5 min between each game. For half of the The first five features are the energies in various frequency participants the order of the game followed is high, bands namely, delta ( E G as 0.5 - 4 Hz), theta ( E T as 4 – 7.5 medium, low and for remaining half low, medium, high Hz), alpha ( E D as 7.5 – 12.5 Hz), low-beta ( E lE as 12.5 – sequence is followed. During the game, scores, number of 16 Hz) and mid-beta ( E mE as 16 – 20 Hz) respectively. The moves, duration etc. are also recorded for further analysis. energies in each band are extracted using Welch’s power spectral density as defined by (Welch, Peter, 1967). The last three features in (2) are the Hjorth parameters namely Activity ( H a ), Mobility ( H m ) and Complexity ( H c ) respectively as given in (3). dx(t ) dx(t ) (a) (b) Ha( ) H m( ) Ha var( x(t )) H m dt Hc dt (3) a m H ( x(t )) H ( x(t )) Here x(t) indicates the time domain signal in a window of (c) duration 1 sec and dx(t ) is the first order derivative of the dt signal. Figure 3: Data collection devices: (a) Neurosky EEG device (b) GSR device from eSense (c) Pulse oxymeter from Contec 3.3.2 Analysis of GSR signal The galvanic skin response (GSR) is the electro-dermal response where the skin conductance changes with the 1 3 http://neurosky.com http://www.coopermedical.com/overnight-pulse-ox/cms-50d-plus- 2 https://www.mindfield.de/en/biofeedback/products/esense/esense-skin- recording-fingertip-pulse-oximeter.html response BMAW 2016 - Page 47 of 59 amount of secretion from the sweat glands in presence of 3.4 BAYESIAN NETWORK CONSTRUCTION stressful, likeable events. Therefore GSR can be used as a good predictor of concentration, mental workload etc. in Our problem statements for creating the Bayesian Network flow study (Nourbakhsh, Nargess, et al, 2012). During flow are as follows - experience, subjects should experience a higher x A number of random participants are performing a concentration and focus on the task. This can be measured task in a controlled environment. using the GSR. x The task can be of three difficulty levels: easy, moderate or hard. The GSR device consists of two electrodes which applies a x The participants are categorized based on their constant voltage to the skin. The current which then flows intelligence levels i.e. the IQ scores. through the skin can be detected. The GSR signal consists of two components: phasic, which is the fast varying x The participants can be in either of three states: flow, component and tonic, which is the slow component. Both boredom or anxiety. contain important information associated with specific x Their performance and feedback are recorded to physiological aspect of the mental states. The tonic determine their mental state and experience. component (Tc) is calculated only by taking the inverse x During data capture, three physiological sensors transform of first few Fourier coefficients (4) and the namely EEG sensor, HRV sensor and GSR sensor are phasic component (Pc) is calculated by inversing the higher used. order coefficient of the Fourier coefficients (5).The eSense GSR sensor has 5Hz sampling frequency hence we have Now a Bayesian Network (BN) framework is created with used first 4 components (0<=k<=3) of the Fourier the objective to diagnose and investigate the different coefficients which corresponds to 0.5Hz. The IFFT in (4) relationships between the sensors nodes and the state nodes and (5) corresponds to Inverse Fast Fourier Transform.The as shown in Figure 4. tonic component (Tc) of the GSR signal is computed for windows of duration 1 second. Next to investigate the distribution Tc over the task interval, we compute the mean and the kurtosis as given in (6) using N successive windows for each task (low, medium, high). These parameters provide the statistical information on the way the skin conductance changes for an individual over time, for a given task. N 1 2S  j( ) nk Tc IFFT (¦ x(n).e N ) , k 0,1,2,3 (4) n 0 N 1 2S  j( ) nk Pc IFFT (¦ x(n).e N ) , k 4,5,...N  1 (5) n 0 Tcmean mean(Tc (n)) , Tckurt kurtosis(Tc (n)) , 1 d n d N (6) Figure 4: Proposed Bayesian Network 3.3.3 Calculation of HRV from PPG signals The BN consists of three types of nodes: Stress level can be evaluated using Heart rate variability (J. Evidence nodes: Task Difficulty (TD), Intelligence Taelman, S. Vandeput, A. Spaepen and S. Van Huffel, Quotient (IQ) 2009), (McDuff, Daniel, Sarah Gontarek, and Rosalind Picard, 2014). When the task challenge is low compared to Sensor nodes: GSR, GSR, HRV the subjects’ skill level, then the heart rate variability is State nodes: Emotional State (O), Desired State (D), Final high compared to the flow state where task challenge Outcome (F) matches the skill. We have used the SPO2 device wearable on the ring finger for sensing the Photoplethysmogram Through in the Evidence nodes we can input the static (PPG) signal. We have calculated three time domain HRV knowledge or evidence which we have in our problem parameters namely 1) rMSSD (root mean square of domain. In our case, we know beforehand, the IQ level of successive differences between adjacent NN intervals), 2) the participants and also the difficulty levels of task they SDSD (Standard deviation of successive differences are given. The Evidence nodes serve as parents to the between adjacent NN intervals), 3) SDNN (Successive Sensor nodes, since the sensor readings are a direct causal difference between NN intervals) as explained in (McDuff, effect of the two conditions (IQ & TD). Based on the 2014). Out of these SDNN is found to give a better results obtained from sensors, we can predict the current indication of the stress level for our experiment. cognitive state of the participants. The derived state i.e. the Emotional state (O) from the sensor reading is compared with the Desired state (achieved from ground truth during training) and finally a conclusion (F) is drawn based on BMAW 2016 - Page 48 of 59 these two states. We have used a separate desired state in Figure 6(a) and Figure 6(b). The line representing the order to compare it to the results derived from the signals medium difficulty game for all the subjects is found to about what emotional condition the subject is currently in. overlap over the other two; the separation between the medium games across participants is not consistent. Hence We have created the Bayesian Network in SamIam4, a for better representation, we have only plotted the GSR for comprehensive tool for modelling and reasoning with high and low difficulty games. Bayesian networks, developed in Java by the Automated Reasoning Group of Professor Adnan Darwiche, UCLA. At the beginning of the experiment, the participants are categorized based on their IQ levels measured by a standardized IQ5 test prior to playing the game. This IQ test is a free online test from Brainmetrix. The IQ level and the task difficulty level serve as evidence nodes in the BN. Depending on the state of these two nodes, the sensor readings vary. Hence all the sensor nodes have them as immediate parents. The readings of all three sensors are used to analysis the emotion state of the participants and are compared to the ground truth i.e. the desired state. If two state matches, then we can conclude the actual state via the final outcome and that the Bayesian (a) Network is expected to give the correct outcome. If not, then there is a case of contradiction caused by one of more sensors giving faulty state. In case of a contradiction between observed state and desired state the BN model can be used to resolve this conflict by reasoning between the various nodes using probabilistic queries. 4. RESULTS AND DISCUSSIONS 4.1 EEG SIGNAL ANALYSIS Various features of the EEG signals namely alpha, beta, (b) theta, delta, attention, meditation, Hjorth etc. for all the experiments for all the participants are extracted using (2) Figure 5: Separation between Low, Medium, High and (3). Among all features the Activity measure of Hjorth difficulty task for the Hjorth calculation of EEG signals parameter is found to be indicative of variations of brain for (a) 20 subjects and (b) 16 subjects signals with difficulty level. The raw EEG signal is used Out of 20 participants 8 participants (6, 7, 8, 9, 10, 11, 12 for calculating activity for windows of duration 1 sec and and 20) played the game in high-medium-low sequence the overall mean is taken for each session of the game of and the remaining participants played in low-medium-high all the participants. The results for different difficulty sequence. It is evident from the plots that kurtosis performs levels (Low, Medium and High) are combined separately better than mean of the tonic power in separating the low for all the 20 participants and compared as shown in Figure and high difficulty tasks. 5(a). We have found good amount of separation between three different tasks for 16 out of 20 participants. The comparison for these selected participants are shown in Figure 5(b). 4.1 GSR SIGNAL ANALYSIS The GSR data is subdivided into a number of windows of duration 1 sec. Next we calculate both tonic and phasic power using (4) and (5).The phasic power does not show sufficient separation between different task levels whereas the tonic gives good separation. The mean and kurtosis of the tonic power is calculated using (6). The plot of mean and kurtosis of tonic component for GSR are shown in (a) 4 5 http://reasoning.cs.ucla.edu/samiam/ http://www.brainmetrix.com/free-iq-test/ BMAW 2016 - Page 49 of 59 occurrence over all possible conditions. The scores of IQ less than 89 are treated as Low IQ, scores between 90 and 109 are treated as Medium IQ and scores with 110 is treated as High IQ. In the present study we got 6 participants having high IQ, 9 participants having medium IQ and 5 participants having low IQ. We have used the data for 18 out of 20 participants to train the BN. The sensor data for all the training participants (18 subjects) are classified into three levels (Low, Medium, High) based on their observations. An example is shown in Table 1 where the levels of the EEG sensor data are shown for 5 training participants with High IQ, playing three levels of games. It can be seen that for the easy game (TD = low), all of them have low EEG feature values, for (b) medium game (TD = medium), one of them has high EEG feature and similarly for difficult game (TD = high). The Figure 6: Separation between the (a) Mean and (b) probability of occurrence of each level is calculated and Kurtosis of the Tonic (µS) component of GSR during the updated in the respective Conditional Probability Table Low and High difficulty games. (CPT) of the BN. After the training the final CPT for the EEG node is shown in Table 2. The same rule is followed 4.2 PPG SIGNAL ANALYSIS for the rest of the sensor nodes as well. The PPG signal analysis does not give a clear, consistent Table 1: The levels for EEG Sensor data (Mean of Hjorth - separation between difficulty levels of the task for all the Activity) for 5 Participants with High IQ and playing three participants. We have plotted the successive difference levels of game (TD). Here L-Low, M-Medium, H-High. between NN intervals (SDNN) in Figure 7. For 10 out of TD EEG level (Mean of Hjorth - Activity) 20 subjects there is a separation between the two SDNN values out of which only 5 have the value for the low game L L L L L L lower than the high game. Hence any substantial information cannot be derived from this result of PPG M M H M M M analysis. H H M H H H Table 2: Conditional probabilities of the EEG node is shown for the pair (TD, IQ) EEG (L,H) (M,H) (H,H) (L,M) (M,M) (H,M) L 0.98 0.02 0.02 0.98 0.01 0.01 M 0.01 0.79 0.19 0.01 0.98 0.01 H 0.01 0.19 0.79 0.01 0.01 0.98 EEG (L,L) (M,L) (H,L) Figure 7: Separation of the HRV(SDNN) in msec for all L 0.5 0.25 0.25 the 20 subjects between the high and low difficulty game M 0.25 0.5 0.25 4.3 BAYESIAN NETWORK ANALYSIS H 0.25 0.25 0.5 The Bayesian network framework, given in Figure 4, has The data for the remaining 2 participants (having high IQ been trained based on the results obtained from and low IQ) are used to validate whether the BN nodes are physiological sensor data collected during the experiments. providing the correct results. Different combinations of The conditional probability tables associated with each evidences for these two subjects are checked in the BN node have been updated according to the occurrence of (Figure 8 through Figure 11) and the reasons for their respective parent node. contradictions are explained. The states of the BN are Given a participant of a particular IQ level and solving a shown as rectangular blocks containing the percentage game of a particular difficulty level, we have collected each equivalent of the probability values of the random of the sensor data and have calculated their probability of variables. BMAW 2016 - Page 50 of 59 Case 1: A participant of high IQ is playing a high difficulty state without being too anxious and matches with the level game Emotional State of the BN. However a contradiction is shown in the Final Outcome indicating a deviation of the behaviour of the participant from the expected one. Figure 8: Bayesian Network in Query mode for the Figure 9: Bayesian Network in Query mode for the evidence Task Difficulty= High and IQ=Expert/High evidence Task Difficulty=Low and IQ=Expert/High In this case we can see that the Emotional State (O) matches with the Desired State (D) hence the final outcome is Flow state. Also all the sensors are providing the expected result, except the EEG feature value. The subject used in this experiment has high IQ. From the questionnaire based survey we have found that the participants has reported to be sometimes in anxious state during the high difficulty game. This could be the reason for the EEG feature value to be very high for him. We have found from participant’s feedback that he was mostly in the flow state. Hence the output of the Bayesian model seems to be correlating with the participant’s feedback. Case 2: A participant of high IQ is playing a low difficulty level task In this case we can see that if a participant with high IQ is playing a low difficulty task, the sensor nodes are providing the desired states. This is shown in Figure 9. The EEG feature value and GSR states are in low state as expected, as the participant is expected to be in a relaxed Figure 10: Bayesian Network in Query mode for the state. However, the HRV indicates a low state as opposed evidence Task Difficulty= High and IQ=Poor/Low to the expected high state leading to the contradiction. If a Case 4: A participant of low IQ is playing a low difficulty highly intelligent participant is playing a very easy task level task with low concentration (GSR=Low) then he / she might be in a restless state which can lead to anxiety. Hence the true In this case we can see that all the states are providing the mental state cannot be correctly determined. expected result except the EEG feature value as shown in Figure 11. The participants used in this experiment has low Case 3: A participant of low IQ is playing a high difficulty IQ. We have found from participants’ feedback that he was level task in the relaxed state and did not find the game too difficult. In this case we can see that all the states are providing the This could be the reason for the EEG feature value to be expected result except the EEG feature value as shown in low but with very low percentage. It can also be observed Figure 10. The participant in this experiment has low IQ. from all the four cases that if the contradiction option is According to participant’s feedback, he was in the flow ignored in the final outcome state, then the next highest probability belongs to the state same as the desired state. BMAW 2016 - Page 51 of 59 This shows that while the Bayesian Network provides the Acknowledgements correct prediction of the actual mental state, given the sensor data and static knowledge, it also hints at the We are thankful to the participants who have provided their possibility of a faulty sensor data based on the probability time and inputs for the data collection for the experiments. of the contradiction. References Bakker, A. B. (2005). “Flow among music teachers and their students: The crossover of peak experiences”. Journal of Vocational Behavior, 66, 26–44. Retrieved from http:// dspace.library.uu.nl/handle/1874/10711. Bakker, A. B. (2008). “The work-related flow inventory: Construction and initial validation of the WOLF”. Journal of Vocational Behavior, 72(3), 400–414. Berta, Riccardo, et al. (2013). "Electroencephalogram and physiological signal analysis for assessing flow in games." Computational Intelligence and AI in Games, IEEE Transactions on 5.2: 164-175. V. Carmen, et al. (2009). “Time Domain Parameters as a feature for EEG-based Brain-Computer Interfaces” in Neural Networks 22: 1313-1319 Chanel, Guillaume, et al. (2008). "Boredom, engagement and anxiety as indicators for adaptation to difficulty in games." Proceedings of the 12th international conference Figure 11: Bayesian Network in Query mode for the on Entertainment and media in the ubiquitous era. ACM. evidence Task Difficulty= Low and IQ=Poor/Low Chaouachi, Maher, and Claude Frasson. (2010). 5. CONCLUSION AND FUTURE WORK "Exploring the relationship between learner EEG mental engagement and affect." Intelligent tutoring systems. In this work we have measured the cognitive state of a Springer Berlin Heidelberg. participant performing task of varying difficulty level from brain signals and certain physiological parameters like Chatterjee, Debatri, et al. (2015). "Analyzing elementary heart rate variability and galvanic skin response. Results cognitive tasks with Bloom's taxonomy using low cost show that EEG (Hjorth-Activity) and GSR signals can be commercial EEG device." Tenth International Conference successfully used to distinguish the performance for all the on Intelligent Sensors, Sensor Networks and Information participants in three different level tasks. On the other Processing (ISSNIP). IEEE. hand, the analysis of the HRV data is not providing Chrysafiadi, K. and Virvou, M. (2013). “Student modelling consistent information. These results can be fused together approaches: A literature review for the last decade.” Expert and analyzed along with the performance and feedback Systems with Applications, 40(11): 4715-4722. data to get further insights of the mental state for the participants. Csíkszentmihályi, Mihály, (1990). “Flow: The Psychology of Optimal Experience”, New York: Harper and Row, From our proposed Bayesian Network we need to ISBN 0-06-092043-2. determine whether there is a contradiction between observed and desired state. In case of contradiction we Csikszentmihalyi, Mihaly. (1997). "Finding flow.". need to find out whether there are any sensor(s) nodes Csikszentmihalyi, M. (1999). “If we are so rich, why aren't giving faulty data. In future a large number of participants we happy?” American Psychologist. Vol 54(10), 821-827. are to be considered for further analysis. Moreover, the workflow of the Bayesian Network should be dynamic for Csikszentmihályi, Mihály; Harper & Row. (2015). handling a large number of data. Several knowledge under "FLOW: The Psychology of Optimal Experience" uncertainty can be determined in future by using such Retrieved 2 April 2015. probabilistic graphical models. In the area of education, Csikszentmihalyi, Mihaly, and Isabella Selega models can be designed for generating a sequence of Csikszentmihalyi, eds. (1992). “Optimal experience: questions which are intermixed in difficulty level to Psychological studies of flow in consciousness.” analyze the variation in cognitive states of an individual Cambridge University Press. during the assessment process. Csikszentmihályi, M., Larson, R., & Prescott, S. (1977). “The ecology of adolescent activity and experience.” Journal of Youth and Adolescence, 6, 281-294. BMAW 2016 - Page 52 of 59 Csikszentmihályi, M.; Abuhamdeh, S. & Nakamura, J. test battery." Journal of neuroscience methods 222: 250- (2005), "Flow", in Elliot, A., Handbook of Competence 259. and Motivation, New York: The Guilford Press, pp. 598– Nakamura, J. and Csíkszentmihályi, M. (2009). “Flow 698. Theory and Research.” In Snyder, C. R. and Lopez, S. J. deManzano, Örjan, et al. (2010). "The psychophysiology eds. Oxford handbook of Positive Psychology.Oxford of flow during piano playing." Emotion 10.3: 301. University Press, Oxford, 195-206. DelleFave, A., & Bassi, M. (2000). “The quality of Neapolitan, Richard (2009). Probabilistic Methods for experience in adolescents’ daily lives: Developmental Bioinformatics. Burlington, MA: Morgan Kaufmann. perspectives.” Genetic, Social, and General Psychology p. 406. ISBN 9780123704764. Monographs, 126, 347-367. Nourbakhsh, Nargess, et al. (2012). "Using galvanic skin F.J. Díez, J. Mira, E. Iturralde and S. Zubillaga (1997). response for cognitive load measurement in arithmetic and "DIAVAL, a Bayesian expert system for reading tasks." Proceedings of the 24th Australian echocardiography". Artificial Intelligence in Medicine Computer-Human Interaction Conference. ACM. (Elsevier) 10 (1): 59–73. Novak, Thomas P., and Donna L. Hoffman. (1997). Friedman, N.; Linial, M.; Nachman, I.; Pe'er, D. (2000). "Measuring the flow experience among web users." "Using Bayesian Networks to Analyze Expression Data". Interval Research Corporation 31. Journal of Computational Biology 7 (3–4): 601–620. Pearl, Judea. (1986). "Fusion, propagation, and structuring S. Gudmundsson, P. Runarsson, T. Sigurdsson, S. in belief networks." Artificial intelligence 29.3: 241-288. Eiriksdottir, K. Johnsen, (2007). “Reliability of Pearl, Judea, and Stuart Russell. (1998). “Bayesian quantitative EEG features”, in Clinical Neurophysiology networks.” Computer Science Department, University of 118: 21622171, August 2007. California. Hsu, Chin-Lung, and Hsi-Peng Lu. (2004). "Why do Schnirman, Geoffrey M., Marilyn C. Welsh, and Paul D. people play on-line games? An extended TAM with social Retzlaff. (1998). "Development of the Towerof London- influences and flow experience." Information & revised." Assessment 5.4: 355-360. Management 41.7: 853-868. Schnirman, G.M., Welsh, M.C., Retzlaff, P.D. (1998). W. Klimesch, (1999). “EEG alpha and Theta oscillations “Development of the Tower of London-Revised.” reflect cognitive and memory performance: a review and Assessment 5(4), 355 –360. analysis” in Brain Research Reviews 29. H. Sijuan, (2010). “Feature extraction and classification of Koufaris, Marios. (2002). "Applying the technology EEG for imagery movement based on mu/beta rhythms”, acceptance model and flow theory to online consumer 3rd International Conference on Biomedical Engineering behavior." Information systems research 13.2: 205-223 and Informatics (BMEI). Kramer, Daniel. (2007). "Predictions of performance by Sinha, Aniruddha, et al. (2015) "Dynamic assessment of EEG and skin conductance." Indiana Undergraduate learners' mental state for an improved learning Journal of Cognitive Science 2: 3-13. experience." Frontiers in Education Conference (FIE), Luis M. de Campos, Juan M. Fernández-Luna and Juan F. 2015. 32614. IEEE. Huete (2004). "Bayesian networks and information J. Taelman, S. Vandeput, A. Spaepen and S. Van Huffel, retrieval: an introduction to the special issue". Information (2009). "Influence of Mental Stress on Heart Rate and Processing & Management (Elsevier) 40 (5): 727–733. Heart Rate Variability", 4th European Conference of the Mauri, Maurizio, et al. (2011). "Why is Facebook so International Federation for Medical and Biological successful? Psychophysiological measures describe a core Engineering, vol. 22, pp. 1366-1369. flow state while using Facebook." Cyberpsychology, Van Schaik, Paul, Stewart Martin, and Michael Vallance. Behavior, and Social Networking 14.12: 723-731 (2012). "Measuring flow experience in an immersive McDuff, Daniel, Sarah Gontarek, and Rosalind Picard. virtual environment for collaborative learning." Journal of (2014). "Remote measurement of cognitive stress via heart Computer Assisted Learning 28.4: 350-365. rate variability." Engineering in Medicine and Biology Welch, Peter. (1967). "The use of fast Fourier transform for Society (EMBC), 36th Annual International Conference of the estimation of power spectra: a method based on time the IEEE. averaging over short, modified periodograms." IEEE Mueller, Shane T., and Brian J. Piper. (2014). "The Transactions on audio and electroacoustics: 70-73. psychology experiment building language (pebl) and pebl BMAW 2016 - Page 53 of 59