Context-aware systems gained huge popularity over the last decade. Rapid
evolution of personal mobile devices such as smartphones, tablets, smartwatches and
other types of wearables forced researchers and developers to work on ecient
methods for modelling, and processing contextual information. The most recent
research in the area of mobile context-aware systems identies four challenges
that should be met by the every context-aware system. These are [
Context in context-aware systems can be described as any information that
can be used to characterize the situation of an entity [
We assert that a special case of a high-level context may be the emotional
state of the user. In such a case, number of problems to be solved appear. The
most serious are to determine if it is possible to build a high-level emotional state
from low-level contextual data and how to do this. We believe, that Jesse Prinz’s
Embodied Appraisal Theory [
AfC is an interdisciplinary eld of study, where two crucial elements are considered: modes of data collection and ways of interpreting them in correlation with aective states corresponding to emotions. Today most often harvested and processed information are about: speech (prosody: pitch variables, speech rate), body gestures and poses (3D mapping, motion capture techniques), facial expressions (visual analysis and electromyography), physiological monitoring (blood pressure, blood volume pulse, galvanic skin response). 2
Basically, we are aiming at developing a technology to detect, identify and interpret human emotional states. We believe, that it can be provided based on the integration of context-aware systems and aective computing paradigms. We are planning to identify and characterize aective context data, and provide knowledge-based models to identify and interpret aects based on this data. A working name for this technology is simply AfCAI: Aective Computing with Context Awareness for Ambient Intelligence .
The area of aective computing shares some characteristics with the area of mobile context-aware systems that we studied so far. The environment is highly dynamic in both types of systems, as the biomedical information like heart rate reading or blood pressure may change rapidly. There exist various factors that may have impact on the changes of the biomedical parameters. These factors may not always be observed, or may be delivered to the system with a degree of uncertainty that needs to be handled appropriately. Finally, the user has to be aware of what the system does with such sensitive data as biomedical information. He or she has to understand what is the goal of the system, and should have the ability to adjust the system to follow his or her goals.
We consider a certain hierarchy in interpreting aective context data. In the lowest level, we encounter readings including physical context that is delivered by the current mobile phone sensors (temperature, location coordinates), as well as physiological measurements from dedicated biomedical devices (heart rate, skin temperature, galvanic skin response measurement). This data can be considered as objective context. On a higher level, we provide certain conceptualization of this data, that we believe can be provided automatically. Examples include being at home, or resting heart rate of given person. Then, on the next level we would like to provide certain interpretation of that conceptualization, which in the case of the aective data could be relaxed, excited, or anxious. In fact, an ultimate level could include identication of causal relations like explanations and prediction.
Detecting changes in emotional context and explaining it with changes in physiological context could be used to build complex proles of user emotions. Such proles could be further used by a computer system to better t the user preferences, or to react on some alarming changes both in emotional and in biomedical contexts. Furthermore, existence of complex proles of user emotions may lead to creation of general proles for emotions at all or at specic context which could be used for diversity of cases. Awareness of variation in user emotion in relation to general prole could be then classied as another variable for objective context (weather, location, activity etc.). This general information could be then used for better understanding of user emotions in real situations and better reaction to them. Another case is an introduction of user emotions’ awareness into personal articial assistants, suggestions systems or non-invasive advertisement. It is also valid to discover shortcomings of current generation mobile devices for gathering biomedical context, to be aware and ready to develop tools and methods for future validation and accuracy increase inside framework. 3
In our most recent research [
The rst aim of our research is to provide: 1. an integrated sensory framework which will use wearable physiological and biomedical hardware sensors for detection of user aects. 2. computational models for aect identication and interpretation
Our aim is to provide an integrated sensory framework which will use
wearable physiological and biomedical hardware sensors for detection of user aects.
We are also oriented to feed mobile context-aware systems with this data. These
assumptions lead us to the conclusion that we need to use some small mobile
devices to acquire sensory data. Based on our experience with various hardware,
four devices were selected as the most promising ones:
Empatica E4. An advanced sensory wristband based on the technologies
previously developed in the Aective Computing division of MIT Media Lab 1.
Blood volume pulse and galvanic skin response sensors, as well as infrared
thermopile and accelerometer are on board. It has already been used in
number of research projects [
Microsoft Band 2. Band developed mainly for tracking tness goals. Equipped with optical heart rate, skin temperature and galvanic skin response (GSR) sensors as well as accelerometer, available through well documented Software Development Kit (SDK). e-Health Sensor Platform. An open medical monitoring platform supervised by the Cooking Hacks. It is a shield for Arduino/Raspberry Pi and the set of sensors that can be plugged in: pulse, oxygen in blood (SPO2), airow (breathing), body temperature, electrocardiogram (ECG), glucometer, galvanic skin response (GSR), blood pressure (sphygmomanometer), patient position (accelerometer) and muscle/electromyography sensor (EMG). Thanks to build on Arduino, this solution can be combined with various devices and installations.
Jawbone UP3. Is an advanced tness band. While it sensory capabilities are limited compared to the above ones, it is ultra portable and non invasive, thus increasing opportunities for all day person monitoring.
What is also important, besides basic sensor reading we want to capture user reports on their emotional state. Ultimately, information on these states should be acquired semi-automatically by dedicated mobile applications in the form of recommendation and mediation 1. However, at the basic stage at which we are now, sensor recordings are compiled with user surveys prepared to determine which emotions were experienced during each stage of a prepared experiment.
In fact in our initial phase of research we prepared and conducted several in lab experiments to verify our assumptions. The studies were conducted with the use of Virtual Reality (VR) via Oculus headset to provide emotional stimuli for more immersive user experience. Three experiments were conducted to collect the data for preliminary analysis as well as for the comparison of the signals collected by dierent devices.
All of these experiments were prepared with a close collaboration with Dr Jan Argasinski from Jagiellonian University and were conducted with him in the Department of Games Technology at Jagiellonian. They are described in the second paper submitted to the AfCAI workshop.
Based on the data from the experiments we are aiming at building knowledgebased computational models for aect identication and interpretation. 1 An example of such an approach is in Jawbone UP Coach, where the user is asked interactively about his past activities.
During the AfCAI workshop in Murcia we elaborated on the issues introduced above. In particular we discussed in more detail our: vision of AfCAI systems, ideas about emotion detection and identication based on Prinz’s theory, works in the area of context aware systems, proposal of aective context aware architecture, outlook at building knowledge-based computational models for aect identication and interpretation.
This work was partially funded by the National Science Centre, Poland as a part of the KnowMe project (registration number 2014/13/N/ST6/01786). 13. Sano, A., Picard, R.W.: Stress recognition using wearable sensors and mobile phones. In: Aective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. pp. 671676. IEEE (2013) 14. Song, M., DiPaola, S.: Exploring dierent ways of navigating emotionallyresponsive artwork in immersive virtual environments. In: Proceedings of the Conference on Electronic Visualisation and the Arts. pp. 232239. British Computer Society (2015)