Towards an Emo-aware Education Through Physiological Emotion Detection

Abstract

During the di erent educative processes in the high school or university, a student feels di erent emotions. For instance, physiological stress has mostly experimented during exam periods, when there is academic overload, in new topics or learning too focused on memorization. Additionally, the stress has been associated with chronic diseases (e.g., heart diseases, faults in the immune system, anxiety or headaches). In line with these notions, in this paper, we introduce the idea of measuring emotions in order to empower the educative processes by providing relevant emotional information of all stakeholders involved in the educational task. This kind of information is highly useful for analyzing new educational methodologies or for evaluating the current educational approaches (educational institutions), and for students because it will allow a possible optimization in their teaching-learning process. Regarding the results, we present a preliminary experiment to evaluate the emotion detector, which obtained an accuracy of 79.17%.

Keywords

• educational approach
• emo-aware architecture
• real-time emotion detection
• physiological stress

-

The stress and its in uence on the life of the humans have been resumed at present with great force, driven by the new theoretical conceptions assumed, its recognition as a disease or its association with multiple alterations of the normal functioning of the organism. Despite its insertion in the eld of medical, social and educational sciences, a general consensus among experts on the de nition of the term stress has not been achieved. This situation has generated a conceptual, theoretical and methodological diversity re ected in a wide range of research collected in many studies. Academic stress is a systemic process, of an adaptive and essentially psychological nature, which occurs when the student is subjected, in school contexts, to a series of demands that, under the assessment of the student, are considered stressors; when these stressors cause a systematic imbalance (stressful situation) that manifests itself in a series of symptoms (indicators of imbalance); and when this imbalance forces the student to carry out coping actions to restore the systemic equilibrium [ 22 ].

According to Arias-Gund n and Vizoso-Gomez [ 3 ], the main factors that generate stress in people are: poverty, constant changes in the employment situation and social, pollution and competition among co-workers and classes. Several studies agree that entering university or school represents a set of highly stressful situations, due to a lack of adaptation to the new environment. This kind of stress can be classi ed as academic stress, it is expressed for example, during exam periods [ 5 ] [ 16 ] [ 23 ], when there is academic overload [ 5 ] [ 16 ] at the beginning of the in the courses [ 23 ], a teaching and learning focused on memorization [ 16 ], when there is a lack of time [ 23 ], the demands of some subjects [ 16 ], during the interventions in public [ 23 ], at the moment that there are methodological deciencies of the teaching sta [ 23 ] and when unsatisfactory results are obtained [ 23 ]. For an educational institution, it is important to know the main academic stressors in its students, given that stress has been associated with chronic diseases [ 20 ], heart diseases [ 20 ], faults in the immune system [ 20 ] , anxiety [ 16 ] [ 21 ], headaches [ 16 ], anger [ 21 ], metabolic and hormonal disorders, depression [ 16 ] [ 20 ] [ 2 ] , sadness [ 16 ] [ 21 ]; irritability [ 22 ] [ 2 ], decrease in self-esteem, insomnia [ 22 ] [ 2 ], even with asthma [ 22 ], memory and concentration disturbances [ 2 ], a ecting both the health and the academic performance of the students [ 22 ].

Therefore, it is important then to carry out these kinds of studies that will be useful; rstly for the students, because it will allow them to increase the theoretical knowledge on the subject and with it, a possible optimization in their teaching-learning process and secondly, for the institution, because it will allow them to have knowledge about stress of the students who are part of it. Finally, the paper is organized as follows: Section 2 discusses the background and the related works on human emotions. Section 3 presents the architecture and the algorithms used in our stress detector. The description of the experiment and results are presented in Section 4. Finally, conclusions and future work are discussed in Section 6. 2

Human emotions

Emotions are located in many parts of the brain. Cognitive responses are located in the cerebral cortex, mainly in the prefrontal area. Also, they imply changes in human behavior, autonomic nervous system, and neuroendocrine alterations. The cerebral centers involved in these processes are located in subcortical regions, in the limbic system and the brain stem [ 7 ]. The amygdala is a brain structure located in the limbic system that has historically been directly related to emotions, it has the size and shape of an almond and its direct electrical stimulation produces subjective reactions of fear and apprehension [ 7 ]. Additionally, the autonomic nervous system is responsible for the physiological activation of the person. It is a basic survival mechanism that allows us to mobilize many of the resources available for rapid action. Before the perception of a threat activates the sympathetic autonomous nervous system that would produce a series of changes in the viscera that are detailed below. While if there is no perception of threat and everything goes smoothly, the parasympathetic nervous system remains activated. According to speci c stimuli, the autonomic nervous system changes the behavior of a determined physiological signal.

In this context, human emotions recognition has been investigated in di erent computer science elds. For instance, in video games, Tognetti et al. proposed to detect enjoyment in a racing game [ 24 ]. In software engineering, Muller and Fritz presented a method to recognize the perceived di culty of developers [ 17 ] and in other work the frustration and the happiness [ 18 ]. Also, we can nd more proposals in the literature such as Healey and Picard [ 11 ], Tognetti [ 24 ], Muller and Fritz [ 18 ], Lee et al. [ 13 ] or Leon et al. [ 14 ]. Overall, the di erent proposed works use di erent data sources to recognize emotions (e.g., images, microphone data, physiological signals or text). However, according to [ 7 ] physiological signals (e.g., heart rate, electroencephalography, electrodermal activity, electromyogram or electrocardiogram) provides a reliable way to recognize emotions because this theory is based on detecting automatic physiological responses of the body. For our practical case of education, we use Electrodermal activity (EDA) as a source of data because EDA is one of the best real-time correlates of stress [ 11 ]. EDA is a psychophysiological parameter that re ects the activity of the sympathetic nervous system [ 8 ]. It could be interpreted as the level of activation of the subject. In other words, when the subject is very activated (i.e. high emotionality) the electrical conductance of the skin increases; on the contrary, when the subject is little activated (relaxed), the conductance of the skin decreases. 3

Emo-aware education approach

Figure 1 shows the architecture of our proposal; which is addressed not only for one person but also for many students (e.g., students of a course). On the left side, each user uses one or more physiological sensors (e.g., E4-wristband3 or Moodmetric ring4); in case of the E4-wristband, this device is placed on the wrist of the non-dominant hand of the subject. Also, the collected signals are the input of the real-time emotion detector module. This module has the responsibility to determine whether the user feels an emotion or not; for this paper, the physiological stress was selected as target emotion, in other words, the detector will mark a label of "stressed" or "not stressed"; for that objective, we have implemented the pre-processing steps (see Section 4 for details) proposed by Bakker et al. [ 4 ] for arousal detection in an integrated pipeline to enable real-time processing. Next, this information is input for the inference engine component, which according to the set of assessment rules in the rule base, it decides which metrics will be sent to the messaging application module. This last module provides relevant information about each student in the experiment to the stakeholders (e.g.,

3 https://www.empatica.com/research/e4/

4 http://www.moodmetric.com/ professors, teachers or researchers). Finally, it is important to remark that each physiological sensor placed on the subjects is directly connected by Bluetooth to the inference engine module; in other words, the inference engine also works as a server to receive all information.

Real-time emotion detector

Rule base

Inference

engine

Messaging application

Sensors … Subjects In this section, we present the rst stage of experiments to analyze the performance of the primary module (i.e., real-time emotion detector) of our approach. First, it is presented the description of the algorithms used for detecting physiological stress; next, it is presented the details of the experiment and the obtained results. As the real-time emotion detector is the primary module of our approach, then it is necessary to ensure a good performance of this module for the correct working of the complete pipeline. Therefore, the goal of this preliminary experiment is to evaluate the performance of the real-time stress detector in terms of its accuracy. As was mentioned before, the stress detector is based on a change arousal detection approach proposed by Bakker et al. [ 4 ].

As we use signals is required a previous step of noise lter applying a median lter over a moving window of size n = 100 EDA samples. After it is applied an aggregation process of one value for each 240 EDA samples. Next, the data is discretized using the symbolic aggregate approximation (SAX) method [ 15 ]. Finally, we use a change detection algorithm based on ADaptive WINdowing (ADWIN) method [ 6 ]; basically, this algorithm analyzes the statistically signi cant di erence between two consecutive splits. For instance, given 1 and 2 as the means of two splits of a sequence of EDA signals, then j 1 2j >2cut is the condition for a change detection that is computed with the Equation 1. (1) where W2 is the variance of the elements of W. is the desired con dence and 0 = =(ln n) [ 4 ]. Figure 2 shows the output the algorithm detecting a stress change. The goal of this preliminary experiment was to evaluate the accuracy of the real-time stress detector, for that objective the following research question was proposed:

What is the accuracy of the real-time stress detector able to recognize physiological stress changes in semi-controlled conditions?

To achieve this objective, we use two di erent stressful5 (i.e., the Stroop Task [ 12 ] and an environmental noise [ 19 ]) for generating stress on participants, and the stress detector can detect these emotional changes. Also, the used stressful is

5 A stressful is any stimulus that generates stress on the user.

de ned as the independent variable, and as dependent variables, the user stress state (measured by the stress detector) and the reported stress by the subjects.

The experiment involved 14 subjects (i.e., master students and Ph.D. candidates), whose ages ranged between 21 and 32 years old. The experiment lasted about 30 minutes; where the subjects interacted with the two stressful by ve minutes each one. All subjects used the E4-wristband, that is a wearable device that o ers real-time physiological data acquisition. Additionally, after of the interaction with the stressful, all participants are asked to complete a questionnaire about their stress perception (self-report stress). Overall, comparing the results of the stress detector and the reported stress by the subjects, the real-time stress detector obtained an accuracy of 79.17% (to compute the accuracy we use the Equation 2).

accuracy =

T P + T N T P + T N + F P + F N (2)

Where TP indicates true positives, TN true negatives, FP false positive and FN false negatives. In this case, examples where reported stress and stress detector are labeled as stressed are considered as true positive. 5

Applications

The recognition of physiological stress on students is a valuable information for di erent stakeholders and this information could be used with di erent proposes. In line with this notion, in this paper we identify two potential applications, which are explained as follow. 5.1

Evaluation of teaching-learning processes In the last decades, from economic, political and social spheres, one of the main objectives of the education, in any of its levels, is the quality. The legal educational regulations have emphasized this demand and for which various projects and institutions have been launched, in order to achieve the highest levels of quality in Peruvian institutions. In addition, there is a need for accountability, demanded by society, with the interest of planning improvement processes that provide as a result the increase in the quality of the education system. In which undoubtedly, the evaluation plays an important and necessary role.

Initially, the evaluation was an activity carried out by those who were in a position of power, authority or superiority over the people who are evaluated. In this way, the evaluation has served and serves for the selection of people, for the quali cation of the apprenticeships, for the promotion within the system or for the certi cation of socially recognized quali cations. However, the evaluation is not a process that consists of controlling and demanding the evaluated, but it is a process of re ection that requires us all to commit to knowledge and improvement. One of the intrinsic reasons for the need for evaluation is that an educational program cannot be designed and developed e ectively and e ciently without the evaluation phase is naturally present. Therefore, the value of evaluation as a quality factor can hardly be denied. Being aware of the need to promote evaluation procedures that address the needs, already mentioned, of accountability and improvement of teaching, a potential use of this emotional information is in the evaluation of educational processes.

Contextualizing in the educational area, it is possible to evaluate the negative emotions that could generate the methodologies that use a teacher, the teaching process, the selected educational competencies, the educational material, or the number of hours in which the students stay in an institution. In line with this notion, it is important to carry out these type of evaluations together with psychologies who can supportand orient about emotional intelligence topics that could be useful for managing emotions in relation to students behavior. 5.2

Prevention of chronic diseases Stress as a psychophysiological response of the organism due to external or internal factors (classi ed as stressors). Stress can complicate the health of people and consequently, it could result in chronic diseases. Regarding the educational area, stress can a ect both teachers and students even reaching more aggressive pathologies such as Burnout Syndrome [ 9 ] which is an emotional disorder that is linked to the workplace (e.g., schools). This syndrome can have very serious consequences, both physically and psychologically.

Therefore, the collection of emotional information takes on a crucial importance in order to make quick decisions in order to avoid and/or prevent this problem from worsening. In our proposal, the collection of information is obtained objectively, appropriately and in real-time, unlike other conventional procedures, to provide true data that help identify cases that require prompt help and derive this information to the corresponding professionals avoiding so that the problem becomes chronic. This information collection could be done during a month, approximately, both to professors and students and from the information that is obtained begin to plan actions oriented to the prevention and treatment of stress. 6

Conclusions and future works

The academic stress whose source is in the educational environment is a question that a ects the students' learning and their well-being.

In this paper, we present an emo-aware architecture for providing emotional information of students (from high school or university) during a teachinglearning process. We argue that this kind of emotional information is fundamental to understand the di erent behaviors in the student reactions involved in a teaching-learning process. Another important contribution of our approach is for the students, which can use the information collected from themselves to analyze their learning di culties and after to make decisions about how to optimize their educational processes.

In the rst stage of experiments (analyzing only the real-time emotion detector component), an experiment was conducted with 14 subjects using the E4-wristband device to gather physiological data (physiological sensors). Comparing the outcome of our stress detector with the reported by each subject (perceive stress), the real-time stress detector obtained an accuracy of 79.17%. Overall, we can conclude the real-time emotion detector based on stress recognition has had a good performance for detecting physiological stress in semicontrolled conditions (i.e., in a room), because this result show a good accuracy in comparison with other machine-learning based on recognition methods, due to it oscillates between 70% and 85% [ 1, 10 ], values reported in the literature of stress recognition using physiological data.

As part of our future work, it is important to generate interest in future research in which academic stress is the focus of attention, because it is necessary to prevent or even cushion the e ects of stress in students, although it may seem to some to be unimportant in comparison with others, it is closely related to undesirable alterations, such as memory failures at the moment of performing a stressful test, or failures in the learning process itself. Also, we plan to conduct a series of simulation-based experiments to assess our inference rules. Then, we plan to conduct experiments with multiple groups of subjects for evaluating the relevance of the information delivered by our prototype application.

References

1. Alberdi , A. , Aztiria , A. , Basarab , A. : Towards an automatic early stress recognition system for o ce environments based on multimodal measurements: A review . Journal of Biomedical Informatics 59 , 49 {75 (feb 2016 )

2. Amat V , Fernandez C , O.I.P.M.R.M.R.D. : Estres en estudiantes de enfermer a . Rev. Rol enferm . 133 , 75 { 78 ( 1990 )

3. Arias-Gund n , O. , Vizoso-Gomez , C. : Causas de estres academico en estudiantes universitarios ( 2016 )

4. Bakker , J. , Pechenizkiy , M. , Sidorova , N.: What's your current stress level? detection of stress patterns from gsr sensor data . In: Proceedings of the 2011 IEEE 11th International Conference on Data Mining Workshops . pp. 573 { 580 . ICDMW '11, IEEE Computer Society, Washington, DC, USA ( 2011 )

5. Barraza A , Mart nez JL , S.J.C.E.A.R. : Estresores academico y genero: un estudio exploratorio de su relacion en alumnos de licenciatura . VE-IUNAES 5 ( 12 ), 33 { 43 ( 2011 )

6. Bifet , A. , Gavalda , R. : Learning from time-changing data with adaptive windowing . In: Proceedings of the 2007 SIAM International Conference on Data Mining , pp. 443 { 448 . Society for Industrial and Applied Mathematics (apr 2007 )

7. Boucsein , W.: Electrodermal Activity. Springer US ( 2012 )

8. Dawson , M.E. , Schell , A.M. , Filion , D.L. , Berntson , G.G. : The electrodermal system . In: Cacioppo, J.T. , Tassinary , L.G. , Berntson , G . (eds.) Handbook of Psychophysiology, pp. 157 { 181 . Cambridge University Press ( 2007 ). https://doi.org/10.1017/cbo9780511546396.007, https://doi.org/10.1017/cbo9780511546396.007

9. Embriaco , N. , Papazian , L. , Kentish-Barnes , N. , Pochard , F. , Azoulay , E.: Burnout syndrome among critical care healthcare workers . Current Opinion in Critical Care 13 ( 5 ), 482 {488 (oct 2007 ). https://doi.org/10.1097/mcc.0b013e3282efd28a, https://doi.org/10.1097/mcc.0b013e3282efd28a

10. Garcia-Ceja , E. , Osmani , V. , Mayora , O. : Automatic stress detection in working environments from smartphones x2019; accelerometer data: A rst step . IEEE Journal of Biomedical and Health Informatics 20 ( 4 ), 1053 { 1060 ( July 2016 )

11. Healey , J.A. , Picard , R.W.: Detecting stress during real-world driving tasks using physiological sensors . Trans. Intell. Transport. Sys . 6 ( 2 ), 156 {166 (Jun 2005 ). https://doi.org/10.1109/TITS. 2005 . 848368 , http://dx.doi.org/10.1109/TITS. 2005 .848368

12. Lattimore , P. : Stress-induced eating: an alternative method for inducing egothreatening stress . Appetite 36 ( 2 ), 187 {188 (apr 2001 )

13. Lee , M. , Kim , K. , Rho , H. , Kim , S.J.: Empa talk: A physiological data incorporated human-computer interactions . In: CHI '14 Extended Abstracts on Human Factors in Computing Systems . pp. 1897 { 1902 . CHI EA ' 14 , ACM , New York, NY, USA ( 2014 ). https://doi.org/10.1145/2559206.2581370, http://doi.acm. org/10 .1145/2559206.2581370

14. Leon , E. , Clarke , G. , Callaghan , V. , Sepulveda , F. : A userindependent real-time emotion recognition system for software agents in domestic environments . Eng. Appl. Artif. Intell . 20 ( 3 ), 337 {345 (Apr 2007 ). https://doi.org/10.1016/j.engappai. 2006 . 06 .001, http://dx.doi.org/10.1016/j.engappai. 2006 . 06 .001

15. Lin , J. , Keogh , E. , Lonardi , S. , Chiu , B. : A symbolic representation of time series, with implications for streaming algorithms . In: Proceedings of the 8th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery . pp. 2 { 11 . DMKD '03, ACM , New York, NY, USA ( 2003 )

16. Mar n MM , Alvarez CG , L.A.A.A.L.B. : Estres academico en estudiantes: El caso de la facultad de enfermer a de la universidad michoacana . rev. iberoam. produccion academica gest. educ 1 ( 17 ) ( 2014 )

17. Muller, S.C.: Measuring software developers' perceived di culty with biometric sensors . In: Proceedings of the 37th International Conference on Software Engineering - Volume 2 . pp. 887 { 890 . ICSE '15, IEEE Press, Piscataway, NJ, USA ( 2015 ), http://dl.acm.org/citation.cfm?id= 2819009 . 2819206

18. Muller, S.C. , Fritz , T. : Stuck and frustrated or in ow and happy: Sensing developers' emotions and progress . In: Proceedings of the 37th International Conference on Software Engineering - Volume 1 . pp. 688 { 699 . ICSE '15, IEEE Press, Piscataway, NJ, USA ( 2015 ), http://dl.acm.org/citation.cfm?id= 2818754 . 2818838

19. Passchier-Vermeer W , P.W.: Noise exposure and public health . Environ Health Perspect . p. 108 ( suppl 1 ): 123 { 31 ( 2000 )

20. Pulido MA , Serrano ML , V.E.C.M.H.P.V.F.: Estres academico en estudiantes universitarios . Psicolog a y Salud 21 ( 1 ), 31 { 37 ( 2011 )

21. Rivadeneira C , Minici A , D. J.: Algunas puntualizaciones sobre el estres . Revista de terapia cognitivo conductual 23 , 1 { 7 ( 2013 )

22. y Rodrigo Mazo Zea, N.B.G. : Estres academico . Revista de Psicolog a Universidad de Antioquia 3 ( 2 ), 55 { 82 ( 2012 ), https://aprendeenlinea.udea.edu.co/revistas/index.php/psicologia/article/view/11369

23. Rodr guez B , Gonzalez MP , B.L. : Estresores academicos percibidos por estudiantes pertenecientes a la escuela de enfermer a de Avila, centro adscrito a la universidad de salamanca . Rev. enferm. CyL 6 ( 2 ), 98 { 105 ( 2014 )

24. Tognetti , S. , Garbarino , M. , Bonanno , A.T. , Matteucci , M. , Bonarini , A. : Enjoyment recognition from physiological data in a car racing game . In: Proceedings of the 3rd International Workshop on A ective Interaction in Natural Environments . pp. 3 { 8 . AFFINE '10, ACM , New York, NY, USA ( 2010 ). https://doi.org/10.1145/1877826.1877830, http://doi.acm. org/10 .1145/1877826.1877830