Our goal is to develop a computer platform allowing for detection, recording and interpretation of aective data. This platform should enable opportunities to use this data in personalized computer systems. In our work we are interested in ubiquitous computing solutions. As such, we assume the platform uses a range of sensors to acquire data about condition and behavior of an individual user. These sensors should be wearable and least intrusive. Furthermore, we assume
Aective computing (AfC) [
In our previous works, we started with experiments based on wearable
sensors, and a virtual reality environment to stimulate and evoke emotions [
In this overview paper we focus on our recent progress in the work on the mobile platform for AfC. The rest of the paper is organized as follows. In the next section we provide the main assumptions about the platform, as well as describe aective data acquisition and interpretation. We then discuss our most recent experiments and provide an outlook of our future works. harvesting other available data, coming from general use mobile devices, i.e. sensors of mobile phones, etc.
The data processing and fusion is conducted using the context-aware systems
paradigm [
Our current focus is on the physiological data acquired by wearable measurement devices. We are focusing on advanced o the shelf wristbands, as well as an open measurement platform BITalino. Data acquisition from such a range of devices needs a universal software solution that meets several requirements for being a convenient asset for AfC studies: 1. The technology should introduce modular architecture to be easily expanded to new devices support and new functionalities. 2. The solution should give a possibility to save data from all available sensors for each device. 3. The applications should facilitate recording data from several dierent wristbands simultaneously, as it is a common experimental scenario. 4. The software should allow synchronization with other data sources (e.g. procedure markers from PC). 5. There should be a possibility for arbitrary le naming and tagging to provide more control over data and to facilitate future processing. 6. The data should be saved in convenient le format (such as i.e. CSV). 7. The data should be available for other applications that run on the same device, i.e. there should be a service that feeds clients (other applications) with the current stream of data. 8. The application should output raw data gathered from wristband(s).
There is a lack of such state-of-the-art solutions. One attempt to create such
an application was done in 2011, when PsychLog project 1 [
As a response for the need, we developed our own mobile software [
with the band: establishing a connection through the provided API, starting the transmission of raw data, handling errors; and WristBandFragment for denition of UI panel for specic wirstband. Application has also generic modules for handling the gathered data and for serialization of data streams to CSV les. Current version of application supports Empatica E4, Microsoft Band 2 and Scosche Rhythm+ devices.
Developed application was used in several studies (see Sect. 3). In particular,
it was used in our recent analysis aimed at critical comparison of quality of HR
and GSR signals in selected devices that oer real-time monitoring with access
to raw data: Empatica E4, Microsoft Band 2, e-Health Sensor Platform V2.0
and BITalino (r)evolution kit [
First of all, we treat the aective data as context data. We dene two types of
such data. Low-level information which can be referred as physiological signals
delivered by hardware sensors, and high-level information that can be derived
based on low-level context and is interpreted as emotional state of the user. Such
an approach allowed us to adapt methods from the eld of mobile context-aware
systems developed by us in our previous research [
Data from single sensor is read by Context Provider Plugin which is part of AWARE framework 2 that allows to instrument, infer, log and share mobile context information. The Context data delivered by the AWARE plugins is sent to Emotion Recognition module responsible for classication of user emotional state. Emotion classication can have polymorphic form as the system can be dynamically recongured with Conguration Rule-based system . This allows for on-line swapping of emotion recognition modules in context dependent manner. Such modules can be based on face expressions analysis, voice recognition, mobile phone usage patterns, or physiological signals, etc. Many of these approaches use data which interpretation highly depend on the user prole. In some cases such data can be unique to users (e.g. mobile phone usage patterns), making it impossible to build models in advance from anonymous data sets.
Therefore, we argue that the emotion recognition modules should be based on online learning mechanisms that allow to discover knowledge about the user from streaming data. Such data can be additionally uncertain and may evolve
over time exposing so called concept-drift, hence the learning mechanism has to
take all of these into consideration. Furthermore, it should generate models that
are understandable by the user and can be modied by one. In our research we
proposed uncertain decision tree generation algorithm for building decision trees
from innite training sets with uncertain instances and class labels. Additionally,
we proposed a translator that converts these trees into human readable, visual
rule-based representation [
All of the components presented in this section were designed to work on mobile devices. This allows to run the system solely on the mobile device. Such feature is desirable in systems that operate on highly sensitive data (such as physiological information) or is expected to work correctly without access to the Internet or other external resources (e.g. online health monitoring systems). 3
To this day, several experiments for verifying our approach have been conducted. With respect to specic characteristics of the experimental procedure, three versions of the experiment can be distinguished. Throughout each of the studies
conducted, the procedure was gradually improved and expanded with additional sensors to gather data and experimental tasks to provide interesting and meaningful research situation. This section provides an overview of the three experiments handled so far, together with the descriptions of experimental procedures applied to those experiments, respectively.
The rst attempt of experimental evaluation of the approach took place in
late April 2017 (Eurokreator lab, Krakow, Poland). The subjects were 2 male
and 4 female, who participated in the workshops held in the lab at the time.
The sensors used included Empatica E4 and MS Band 2 wristbands, paired with
our data recording application. The computer and the smartphone app are
synchronized during the whole procedure. In this experiment, people were asked to
seat in front of the computer and evaluate their arousal on 7-levels scale [1; 7]
in reaction to visual stimuli (pictures from Nencki Aective Picture System,
see [
The second experiment was conducted in June 2017, with participation of
9 Polish students of AGH University of Science and Technology. This time, a
second phase, which followed the Calibration Phase, was involved. Directly after
the arousal evaluation of presented aective pictures, the subjects played a simple
platform game (created specically for research purposes [
Most recent attempt of experimental evaluation was conveyed in January 2018, again at the AGH University of Science and Technology. The procedure itself was identical as in the June 2017 experiment, aside from one short additional phase included after the Gaming Phase. The participant was asked to simply sit and look at a relaxing picture displayed on full screen of the computer. After 30 seconds, a sudden unpleasant sound (terried female scream) was played without any warning. This step is applied to get a physiological reading corresponding to a very strong aective response of the subject. The sensory setup was augmented by two measurement platforms: e-Health and BITalino, as well as Neurobit Optima, an advanced physiology data acquisition equipment, used to gather referential HR and GSR data for the devices that are anticipated for AfC purposes. Data of 18 subjects in total was gathered. 3.1
Preliminary analysis indicates several issues, but nevertheless provide promising
perspectives. The main problems that occurred during the experiments were
related to the fact that the devices require special conguration, including repeated
pairing attempts before the connection takes place. Tentative observations
suggest that HR indeed varies as a function of arousal. Higher arousal values ascribed
in the Calibration Phase are indicated by HR scores higher than in the neutral
condition, and lower arousal values are associated with lower HR scores. This is
consistent with both the general knowledge and the results of other research [
Our future work regard both extension of the current experimental paradigms, as well as interpretation of the data we are acquiring. We are planning to incorporate audio stimuli, as well as considering the use of virtual reality devices. Moreover, in the work regarding the design of aective games, we would also consider how the interaction during the experiment takes place. This is especial important or the practical development of the aective loop in games [20].
In the near future, we are planning to develop methods of semantic
description of the aective data [
In: Proceedings of the 7th International Conference on Entertainment Computing. pp. 122133. ICEC ’08, Springer-Verlag, Berlin, Heidelberg (2009), http://dx. doi.org/10.1007/978-3-540-89222-9_14 21. Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: Survey of an emerging domain. Image Vision Comput. 27(12), 17431759 (2009), https://doi. org/10.1016/j.imavis.2008.11.007