Learning Analytics for Sensor-Based Adaptive Learning (LISA) 6 is a research project aimed at improving learner support through the use of sensor data. Specifically, the research to bring together user-centric learning analytics, analysis of sensor data indicating the emotional state of a learner and adaptive feedback as a learning support is being progressed to provide solutions for sensor-based adaptive learning. Furthermore, the developed solutions will be integrated into learning environments and products.
In the pilot study presented in this paper, we aimed at exploring emotion detection using sensors and tried to interpret the data attained by participants. We first discuss the role of emotions in a learning context, classify emotions in four academic emotions, collect Haeseon Yun et al. physiological data using sensors and interpret empirical data to relate with emotions in a learning context.
Emotion plays a navigating role in cognitive knowledge attainment and it affects the outcomes of learners' behaviors and even future choice on actions [IF10]. For successful learning, providing an environment where learners feel emotionally supported is important so that both emotional and cognitive learning can take place [GS08]. One way to provide an emotionally supportive environment is to provide awareness of his or her own emotional state to a learner so that the information can encourage him or her to modify his or her emotional state [Bu06]. For instance, if a learner is in a negative emotional state such as frustration or boredom, knowing his or her current state can help a learner to make a decision to break out from the state.
The research in autonomic response in emotion examines various physiological signals such as heart rate, heart rate variability, respiratory rate, electro-dermal activity to detect positive (happiness, contentment, joy, peacefulness, and calmness) and negative (anger, disgust, fear, sadness, surprise, fear, depression, boredom, embarrassment) emotions [Kr10] [CM15] [Fa09]. However, applying outcomes of the studies in general emotion may not be applicable in a learning context since the emotional states interested in learning context are different from general emotions [Wo09]. For instance, intrinsic motivation, state of flow [Cs90] are the emotional states that are positively related to successful learning yet, classifying these states based on the previous studies of emotion is not simple. Considering emotion in a learning context, Pekrun and colleagues [Pe02] focused on academic emotion which is consisted of the positive emotions which are enjoyment, hope, pride and relief and the negative emotions such as anger, anxiety, shame, hopelessness and boredom.
As vast research in sensors to detect physiological changes relation to emotion has been conducted in the field of autonomic response and the emotional state in a learning context is investigated in education and educational psychology, the commonly used dimensions of emotions [Wo09] were selected. Specifically, four emotions (joyful, satisfied, angry, and bored) which are related to academic emotions were used for this study as follows:
Positive valence and high arousal: excited and joyful Valence refers to a range from happiness to unhappiness whereas arousal pertains to a range from excitement to calmness. For instance, when a person is being surprised by a nice gift, he or she can be excited (high arousal) and happy (positive valence) whereas when one is being scared by someone, one can still be excited (high arousal) but angry or unhappy (negative valence).
To detect emotions in an experiment setting, emotional stimuli are applied to learners and self-reports on the emotional stimuli are used to relate the emotional stimuli with specific emotions. For instance, Lang and colleagues [LBC08] presented a set of emotional pictures as emotional stimuli and asked participants to indicate their levels of emotions.
Additionally, hardware sensors are also adopted to associate physiological changes during the presence of emotional stimulus with specific emotion. For example, a camera was used to detect head position and movement to relate to emotion [Wo09] and various researchers advocated the feasibility of motivation detection using hardware sensors including camera and other physiological sensors such as EEG, EMG, electro-dermal activity sensors and heart rate sensor
Specifically, heart rate variability is shown to tell when the given task induces stress [YH08] and skin conductance and electro-dermal activity describes changes in emotion [Kr10] [Mc12].
Even though there are many attempts from previous mentioned studies using sensors to detect emotion, relating a specific emotion with sensors data in a learning context is still in its explorative state. Therefore, in this study we have conducted a pilot study to investigate further into sensor data in relation to emotion, specifically academic emotion.
As wearable sensors can measure physiological changes which then can elaborate emotional state, we have replicated [LBC08] experiment in addition to using physiological measurement of EDA and ECG. The purpose of the pilot study was to investigate physiological signals based on the founded experiment and to further make improvements for the future study when integrating sensors in an emotional stimuli experiment setting.
In this pilot study, we used 58 IAPS (International Affective Picture System) pictures and each participant wore ECG and EDA sensors. The total number of six subjects participated in this pilot study. The participants were selected based on their willingness and availability. Participants of the studies were introduced with the purpose of the study as to relate physiological studies with emotional pictures. Physiological signals were measured using pre-gelled electrodes (ECG on left part of their chest or collarbone and EDA on two fingers) and the signals were verified visually using real-time online visualization tool before conducting the experiment. Similar to the original study, all participants were explained to look at the picture and rate the Self-Assessment Manikin (SAM) [LBC08] which includes three dimensions of emotions. To clarify three dimensions, example slides Haeseon Yun et al.
were provided as figure 1. When a participant understood what to do during this experiment, he or she began the experiment and it took around 30 minutes.
The experiment was consisted of 58 sessions. Each session was consisted of a prompt ("Get ready for the next slide"), one of 58 emotional pictures and a SAM rating scale for self-report. The participant was prompted to prepare for the next picture for a duration of five seconds. Then the picture was displayed for six seconds followed by the screen with the 9-point scale SAM ratings. The rating process was accomplished by clicking on radio buttons associated with the three SAM scales for valence, arousal and dominance. A countdown timer was located on the bottom of the rating page to inform the remaining time to encourage a participant to make his or her selection.
Three physiological signals (ECG, EDA and HRV) along with self-report response value were collected from the six participants. The analysis of the collected data took two parallel approaches. First, participants' self-report response on presented pictures were classified into 4 dimensions of emotion (joyful, satisfied, angry and bored) and compared with the value from the original study we have replicated. We were able to attain all six participant's response on their perceptions on presented pictures.
To classify pictures into four emotions, self-report values that the subjects made were taken into consideration. Out of 9 scale, when the self-report value for a given picture was higher than 6, it is considered high and when the value was lower than 4, it was considered low. When the value falls between 4 to 6, it was marked with multiple emotions. For instance, when a picture was marked as 8 in arousal and 2 in valence, the picture was classified as angry and when the picture was perceived as 4 in arousal and 8 in valence, it was marked as satisfied/joyful based on the dimension described in section 2. The overall results show that there were some similar evaluations of the pictures especially for pictures classified in angry and joyful dimensions between the original group and our experiment subjects. For example, 3100 (burned victim) were claimed to stimulate anger by 5 out of 6 participants. Picture 5450 (liftoff) was perceived as joyful picture by 4 out of 6 participants which corresponds to the normative study result. However, for satisfied and bored, only one participant indicated bored in one picture (3230: dying man) which was coincided with the study that we have replicated. Even though similarity in evaluation of pictures in satisfied or bored dimensions were observed, the relation between our subjects and the group from the original study was more obvious for the pictures with high arousal (angry or joyful).
The physiological data (HRV and EDA) successfully retrieved in this pilot study resulted in 3 data set out of 6 subject data due to technical problems. Therefore, the analysis of the HRV and EDA will focus on 3 data set. For the EDA measured during the experiment, we could find the individual difference as shown in the figure 3. Three recorded signals showed that individual started at the different point (9-13 micro Siemens) and their range of changes was different by person. This difference could be due to the random order of picture presented or individual difference in perceiving emotional pictures or this may be due to the individual difference in EDA detection. The skin conductance is known to widely different between subjects [Br16]. Overall, we have observed some high peaks in EDA value when angry or joyful pictures were shown to the participants which was also shown in the previous study [Mc01]. For instance, subject A rated the picture 3100 (burned victim) as the picture that induced angry emotion and the signal had slight peak from 5 to 5.2 micro Siemens. 8620 (Circus Horse) was presented afterward and it was marked as joyful and the EDA signal shows upward trend to 5.5 micro Siemens. When 2070 (baby) was shown to the subject A, the subject had a dynamic change in EDA (6.4 to 7 micro Siemens) and this picture was reported joyful. EDA data of subject B also showed highest peak of 7250 (b-day cake) with 11.2 micro Siemens and the picture was perceived as joyful. From subject C, low value in EDA was also recorded when 2280 (Neut. Boy), 5480 (Fireworks), 7510 (skyscraper), 2780 (actor makeup) and 1930 (shark) were presented and these pictures were considered low or undefined in arousal level (satisfied or bored) by the subject. Similar to subject A and B, EDA value of subject C was high when positive emotion was induced. For instance, highest EDA was observed when 1920 (dolphins) was presented and furthermore, when 5910 (fireworks), 2070 (baby), 7330 (ice cream), 5020(flowers) and 7410 (M&M) were presented, the subject C had high EDA signal and the pictures were marked as joyful and satisfied.
The heart rate variability (HRV) recorded for this experiment was analyzed to check if the irregularity in heart beat is observed. According to [HH79], the HRV is suppressed (regular heart beat) when the task is demanding, which can be also translated as stressful.
The HRV signal data collected by the subjects had a different range. For instance, participant A and 3's HRV was from 520 to 850 millisecond whereas participant B's range was from 550 to 1100 millisecond as shown in the figure 2. The participant A showed highest HRV value when marking 1710 (3 puppies), 5950 (lightning) and 2070 (baby) and these pictures were marked as all joyful. The lowest HRV was marked during 7010 (basket) and 7380 (roach pizza) which were defined as angry by the participant. Participant B also had high peak, close to 1100 millisecond, when viewing 1710 (3 puppies) and the picture was marked as joyful picture by the subject. The lowest peak was observed when 2130 (angry woman) was marked neutral but somewhat aroused, which tends toward angry. Participant C showed high peaks during 7400 (candy), 7100 (fire hydrant) and 7510 (skyscraper) and low peaks during 7030 (iron), 7380 (roach pizza), 7010 (basket). The high HRV value seems to correlate with positive emotion with rather low arousal which can be defined as joyful and low HRV explains bored or angry. Distinguishing between anger and boredom from the data was not clear.
Awareness of one's cognitive and emotional state (self-awareness) is the first step to improve learning process (self-control) and the lack of self-awareness attributes to learning deficiencies [Zi02]. As emotion plays an important role in learning, our pilot study strived to integrate objective means to detect emotions using physiological sensors and relate to previous findings to further investigate the ways to provide awareness back to learners. Specifically, the paper focused on measuring two physiological signals (EDA and HRV) during emotional picture experiment and aimed to take a step closer to detection and interpretation of physiological data in emotion. Clear statement between physiological data and emotions could not be made due to the small sample size, yet the measured signals such as changes in EDA data were observed when joyful and satisfactory pictures were presented. Furthermore, from HRV values, we observed a distinction between positive and negative emotions, even though distinguishing within positive emotions (joyful or satisfied) and negative emotions (angry or bored) was not possible at this stage. The limitations of the study, namely the small sample size and some technical problems, resulted in a small data set for viable data analysis. Further improvement in hardware and software for accurate detection should be realized along with an increase in sample size as A Pilot Study of Emotion Detection using Sensors in a Learning Context a next step.
In addition to emotion detection and interpretation using sensors, as a next step in LISA project, we are investigating the ideal way to provide awareness of one's academic emotion and recommendation. As the primary findings, based on the users' input (focus group and workshop), heuristic evaluation on the prototypes and consideration of suitable pedagogical approach, we have suggested further steps in [Yu17]. Furthermore, the pedagogical approach, a learning companion, to design a device to provide awareness to learners in a friendly and intuitive way was explained in the study.
A learning companion investigated in [Yu17] in connection with this pilot study implies the need for a further systematic consideration in four aspects: 1) user experience, 2) pedagogical/ instructional support, 3) technical realization and 4) data privacy. The user experience should focus on how to design a learning companion to be trusted and respected by a user [JL16]. The pedagogical/instructional support will focus on emotional support which plays a guiding role in knowledge attainment [IF10], creativity and problem solving [Ah13]. The technical realization should consider feasibility of the concept considering the finance, hardware and software development and data privacy will follow a general rule of thumb to entrust all data control over to the user without transferring to any other medium or cloud without user's knowledge.
Research conducted in the LISA project requires a comprehensive investigation on not only the detection and interpretation of sensor data for emotion but also the means to provide information back to learners in a meaningful, intuitive way. Our current findings in this paper focused on the former part of the overall project and presented an initial work in emotional detection and interpretation using sensor device. The findings showed promising results as a work in progress as the physiological sensor data were able to distinguish between negative and positive emotions even with the discussed limitations.
This work has been funded by the BMBF project LISA (16SV7534K).