3 See http://affect.media.mit.edu/projectpages/iCalm/iCalm-2-Q.html
Aective computing (AfC) is a novel paradigm originally proposed in 1997 by
Rosalind Picard from MIT Media Lab in her paramount book [
In this paper we discuss selected important challenges in designing experiments that lead to data and information collection on aective states of participants. We aim at acquiring data that would be basis to formulate and evaluate computer methods for detection, identication interpretation of such aective states, and ultimately human emotions.
This work is result of cooperation between two research teams: one from AGH UST led by Grzegorz J. Nalepa and second in Department of Games Technology led by Pawe“ Wƒgrzyn. Early ideas were originally presented in the project proposal entitled Knowledge-based models for aective context-aware systems (KAFXS) written by Grzegorz J. Nalepa in cooperation with Jan K. Argasi nski and Piotr Augustyniak. They were later developed and focused on the development of practical experiments presented in this paper. 2
Challenges in Designing Practical AfC Experiments To design and conduct practical AfC experiments that could deliver valuable aective data, a number of challenges has to be addressed.
Dening emotion is a dicult problem. Without consensual
conceptualization and operationalization of exactly what phenomenon is to be studied,
progress in theory and research is hard to achieve and fruitless debates are likely
to proliferate. Modern theories of emotions have their origin in 19th century
William James theorized about aects in terms of reactions to stimuli. He was
precursor to appraisal theory which is among most popular in the community
of computational emotional modeling [
First of all, in our work we initially assume that emotions are results of
non-cognitive processes, as James proposed. More specically, we are aiming at
building on the somatic feedback theory of emotions form Jesse Prinz [
Therefore, we assume that we need to measure number of bodily signals to
detect and identify emotions. We begun our work with an in-depth analysis of
selected works in experimental psychology and biological psychology. A survey
paper [
From our perspective, and considering our needs, the review turned out to be mostly inconclusive and provided little support in designing our experiments. Some of the main challenges are related to: 1. the use of discrete (e.g. Ekman’s faces) vs continuous (e.g. Russel’s Circumex) models of emotions distinction between basic emotions vs high/social emotions while naming emotions is a lot conceptualization and high level semantics involved, that needs to be minimized the experimental assumptions dier importantly, thus to comparison of results is questionable the individual, and cultural dierences of participants need to be considered in some cases how to evoke emotions in experimental setup, what are the optimal stimuli which bodily signals should be used, how to measure and what to measure (e.g. only HR and GSR values) what is the role of of user questionnaires, in fact people need semantics (names of emotions) not numbers (e.g. Valence/Arousal or HR value)
There are also some other practical challenges that we will need to address in the near future. They include (but are not limited to): 1. the quality of data from mobile devices, are the results from bands reliable (is raw signal available?), there is an need for cross validation 2. volatile hardware market, devices change or get discontinued, (e.g. Microsoft
Band 2) 3. reliable hardware and software setup, 4. data synchronization across several devices.
Finally, an important challenge remains a synthetic way of reporting the
measure of emotion. There are number of approaches to do it. Some of them simply
assume reporting Valence/Arousal vales. We are dedicated to delivering more
synthetic methods that would combine data measurements with user reports.
Therefore, we are working on certain aective metrics that would include both
the measurements of bodily responses, as well as report and stimulus
evaluation by participants. Some other related works include the so-called Emotional
Index [
The rst aim of cooperation of our two research groups is to provide: 1. an integrated sensoric framework which will use wearable physiological and biomedical hardware sensors for detection of user aects, 2. computational models for aect identication and interpretation.
To deliver solutions to the rst objective we need to use some small mobile devices to acquire sensoric data. In our current experiments we selected 3 such devices.
Empatica E4. An advanced sensory wristband based on the technologies
previously developed in the Aective Computing division of MIT Media Lab 1.
Blood volume pulse and galvanic skin response sensors, as well as infrared
thermopile and accelerometer are on board. It has already been used in
number of works [
Microsoft Band 2. Band developed mainly for tracking tness goals. Equipped with optical heart rate and skin temperature sensors, accelerometer and galvanic skin response (GSR) sensor available through well documented Software Development Kit (SDK). e-Health Sensor Platform. An open medical monitoring platform supervised by the Cooking Hacks. It is a shield for Arduino/Raspberry Pi and the set of sensors that can be plugged in: pulse, oxygen in blood (SPO2), airow (breathing), body temperature, electrocardiogram (ECG), glucometer, galvanic skin response (GSR), blood pressure (sphygmomanometer), patient position (accelerometer) and muscle/electromyography sensor (EMG). Thanks to build on Arduino, this solution can be combined with various devices and installations.
We are using dierent data acquisition methods for these three devices. Basically, for E4 we use mobile applications delivered by Empatica (however, on a desktop system), for Microsoft Band 22 we developed our own mobile app, and for e-Health we use a basic setup for serial port data acquisition.
Furthermore, sensor recordings are compiled with user surveys (see Figure 1) prepared to determine which emotions were experienced during each stage of a prepared experiment. In fact in our initial phase of research we prepared and conducted several in lab experiments to verify our assumptions. The studies were conducted with the use of Virtual Reality (VR) via Oculus headset to provide emotional stimuli for more immersive user experience. Three experiments were conducted to collect the data for preliminary analysis as well as for the comparison of the signals collected by dierent devices.
How do you feel during watching a movie? unpleasantly boredom neutral neutral Fig. 1. Survey presented for each stage of the experiment (English adaptation for the paper; during experiments Polish version was used). In addition, the participant was able to describe the emotions in his own words below. pleasantly fascination
What do you feel?
Sadness Joy Rage
Anxiety Surprise Fear Irritation Shame Contempt
Guilt Disgust Pleasure
Desperation Pride NONE OF THE ABOVE
All of the experiments were prepared and conducted by the Authors in the Department of Games Technology at Jagiellonian University in July, September, and October 2016.
Experiment 1: Secondary School students. First experiment was conducted in July as a part of a holiday camp for secondary school students. 7 pupils (men) was connected to three devices (Empatica E4, Microsoft Band 2, e-Health Sensor Platform) during the test procedure. Signals measured by all devices: Heart Rate (HR), Galvanic Skin Response (GSR), Skin Temperature.
Experiment consists of 4 stages that took place in Virtual Reality (VR) environment: (a) introductory movie without interaction, (b) three tasks with architecture tool (changing colors or moving furniture around), (c) jump scare, (d) movie without interaction to calm down participants.
Experiment 2: The Scientist’s Night. Second study was conducted during the Scientist’s Night at Jagiellonian University in September. More than 100 participants take part in between-subject procedure: each participant was connected only to one of three devices (Empatica E4, Microsoft Band 2, e-Health Sensor Platform) during the experiment. This plan was adopted due to the nature of the event: a lot of people and chaos do not give a possibility to conduct withinsubject study as before. Signals measured by all devices: HR, GSR.
During experiment, 6 participants were at the scene: three was watching movie in VR and another three was dealing with papers (agreement to participate in the study before movie, survey after movie). In order to simplify and shorten the procedure, only one movie, approx. 5 minutes long, was presented. Experiment 3: The Scientist’s Night follow-up study. The third study was performed with the same procedure using the same tools as during Scientist’s Night, but to make it very calmly, slowly and accurately. The planned group is 30 persons (10 persons x 3 devices).
The data from these experiments is currently being analyzed. The results will be presented during the workshop. 5
During the AfCAI workshop in Murcia we presented in more details the experiments and shared the experiences we got.
Our next steps for future works include cross validation and evaluation of mobile bands data, evaluation restriction to HR, GSR signals, provisioning several conguration of emotion evoking setup, combination of VR experiments with odor stimuli, delivery of several veried data sets for further analysis, and nally position and constraint research with specic applications and projects.
Furthermore, we aim at developing data acquisition layer integrated with
our recent solutions [
The authors would like to thank the participants of the Cognitive Science Seminar for Philosophy PhD Students in spring 2016. Moreover, special thanks go to Dr Micha“ Klincewicz for his remarks on methods in experimental psychology. Finally, we would like to thank our students who helped us with the experiments, and data treatment, especially the AfC group in the KECS course in 2016.