Aective computing (AfC) [18] is a relatively well established domain interested in the computer-based interpretation and synthesis of emotion, and aect-related signal. To solve its important challenges, it combines results of several domains, including computer science, philosophy, psychology, cognitive science, and more recently neuroscience and biomedical engineering [6].
In our previous works, we started with experiments based on wearable sensors, and a virtual reality environment to stimulate and evoke emotions [14]. Furthermore, we introduced the proposal of using the context-aware systems paradigm to develop a platform for AfC, that could be used in ambient intelligence applications [17]. We described the design and the preliminary work on this platform in [15]. Later on, we turned our attention to computer games [8], as an interesting environment to design aective experiments [16]. As we focus on wearable devices, in our recent paper we provided results of a critical analysis of measurements using the selected devices [9]. Moreover, we developed our own software for mobile phones [10] for aective data acquisition.
In this overview paper we focus on our recent progress in the work on the mobile platform for AfC. The rest of the paper is organized as follows. In the next section we provide the main assumptions about the platform, as well as describe aective data acquisition and interpretation. We then discuss our most recent experiments and provide an outlook of our future works.
Our goal is to develop a computer platform allowing for detection, recording and interpretation of aective data. This platform should enable opportunities to use this data in personalized computer systems. In our work we are interested in ubiquitous computing solutions. As such, we assume the platform uses a range of sensors to acquire data about condition and behavior of an individual user. These sensors should be wearable and least intrusive. Furthermore, we assume harvesting other available data, coming from general use mobile devices, i.e. sensors of mobile phones, etc.
The data processing and fusion is conducted using the context-aware systems paradigm [4]. We realize, that in such a setting the data can be only partially available and reliable. As such we build on our previous work, where we considered uncertainty handling in mobile context-aware systems [5]. We assume a layered approach, with a data acquisition layer from diverse sensors, and interpretation layer incorporating several articial intelligence methods. These two main layers are described next.
Our current focus is on the physiological data acquired by wearable measurement devices. We are focusing on advanced o the shelf wristbands, as well as an open measurement platform BITalino. Data acquisition from such a range of devices needs a universal software solution that meets several requirements for being a convenient asset for AfC studies:
1. The technology should introduce modular architecture to be easily expanded to new devices support and new functionalities. 2. The solution should give a possibility to save data from all available sensors for each device. 3. The applications should facilitate recording data from several dierent wristbands simultaneously, as it is a common experimental scenario. 4. The software should allow synchronization with other data sources (e.g. procedure markers from PC). 5. There should be a possibility for arbitrary le naming and tagging to provide more control over data and to facilitate future processing. 6. The data should be saved in convenient le format (such as i.e. CSV). 7. The data should be available for other applications that run on the same device, i.e. there should be a service that feeds clients (other applications) with the current stream of data.
There is a lack of such state-of-the-art solutions. One attempt to create such an application was done in 2011, when PsychLog project1 [7] was developed. In addition to the module responsible for sensory data collection, it also had survey module for self-report questionnaires and visualization module for presenting gathered signals. Unfortunately, the project has been abandoned and only beta version for Windows Mobile platform was released.
As a response for the need, we developed our own mobile software [10] for data acquisition, which satises most of the identied requirements. It is built as a set of interfaces that should be implemented to communicate with the specic device: WristBand for denition of the band, its possible states and list of sensors available; WristBandService for supporting every stage of communication with the band: establishing a connection through the provided API, starting the transmission of raw data, handling errors; and WristBandFragment for denition of UI panel for specic wirstband. Application has also generic modules for handling the gathered data and for serialization of data streams to CSV les. Current version of application supports Empatica E4, Microsoft Band 2 and Scosche Rhythm+ devices.
Developed application was used in several studies (see Sect. 3). In particular, it was used in our recent analysis aimed at critical comparison of quality of HR and GSR signals in selected devices that oer real-time monitoring with access to raw data: Empatica E4, Microsoft Band 2, e-Health Sensor Platform V2.0 and BITalino (r)evolution kit [9]. Results demonstrated that BITalino remains the most prospective platform for both HR and GSR measurements, especially as the technical support for the e-Health is being phased out. As for secondary HR readings, MS Band 2 can be used. While the BITalino kit does not have the wristband form, it can be turned into a wearable using the BITalino Freestyle kit and 3D-printed boxes.
First of all, we treat the aective data as context data. We dene two types of such data. Low-level information which can be referred as physiological signals delivered by hardware sensors, and high-level information that can be derived based on low-level context and is interpreted as emotional state of the user. Such an approach allowed us to adapt methods from the eld of mobile context-aware systems developed by us in our previous research [15] to the area of AfC. In particular, we integrate the architecture for developing mobile context-aware system with methods for aective data acquisition and machine learning algorithms for emotion recognition to propose the architecture for aective data processing and interpretation. The architecture is given in Fig. 1 Data from single sensor is read by Context Provider Plugin which is part of AWARE framework2 that allows to instrument, infer, log and share mobile context information. The Context data delivered by the AWARE plugins is sent to Emotion Recognition module responsible for classication of user emotional state. Emotion classication can have polymorphic form as the system can be dynamically recongured with Conguration Rule-based system. This allows for on-line swapping of emotion recognition modules in context dependent manner. Such modules can be based on face expressions analysis, voice recognition, mobile phone usage patterns, or physiological signals, etc. Many of these approaches use data which interpretation highly depend on the user prole. In some cases such data can be unique to users (e.g. mobile phone usage patterns), making it impossible to build models in advance from anonymous data sets.
Therefore, we argue that the emotion recognition modules should be based on online learning mechanisms that allow to discover knowledge about the user from streaming data. Such data can be additionally uncertain and may evolve Fig. 1. Architecture of dynamic framework for context aware systems extended with emotion recognition and mediation. [11] over time exposing so called concept-drift, hence the learning mechanism has to take all of these into consideration. Furthermore, it should generate models that are understandable by the user and can be modied by one. In our research we proposed uncertain decision tree generation algorithm for building decision trees from innite training sets with uncertain instances and class labels. Additionally, we proposed a translator that converts these trees into human readable, visual rule-based representation [3]. This representation is based on the same rule-based language, called HMR+ [2], that is used by the Conguration Rule-based system module. Hence, both the dynamic system reconguration and reasoning about emotional condition of a person can be conducted by the use of our dedicated inference engine HeaRTDroid3 . Similarly, both elements can be directly modied by the user with a usage of explicit and implicit mediation techniques [2].
All of the components presented in this section were designed to work on mobile devices. This allows to run the system solely on the mobile device. Such feature is desirable in systems that operate on highly sensitive data (such as physiological information) or is expected to work correctly without access to the Internet or other external resources (e.g. online health monitoring systems).
To this day, several experiments for verifying our approach have been conducted. With respect to specic characteristics of the experimental procedure, three versions of the experiment can be distinguished. Throughout each of the studies conducted, the procedure was gradually improved and expanded with additional sensors to gather data and experimental tasks to provide interesting and meaningful research situation. This section provides an overview of the three experiments handled so far, together with the descriptions of experimental procedures applied to those experiments, respectively.
The rst attempt of experimental evaluation of the approach took place in late April 2017 (Eurokreator lab, Krakow, Poland). The subjects were 2 male and 4 female, who participated in the workshops held in the lab at the time. The sensors used included Empatica E4 and MS Band 2 wristbands, paired with our data recording application. The computer and the smartphone app are synchronized during the whole procedure. In this experiment, people were asked to seat in front of the computer and evaluate their arousal on 7-levels scale [1,7] in reaction to visual stimuli (pictures from Nencki Aective Picture System, see [12]) presented one by one. The subjects were exposed to three series of 18 pictures, where each was displayed for a xed time (3 seconds per stimuli), with 30 second breaks between each session. Before the rst group of pictures, 30 seconds of blank screen was also presented, in order to acquire baseline HR and GSR recordings during participants inactivity. The whole experiment lasted about 15 minutes. Later, the procedure was extended to include another experimental phase, so that picture exposition was engaged as a Calibration Phase. Its purpose is to address the problem of individual dierences between baseline physiological activity and reaction patterns of various subjects.
The second experiment was conducted in June 2017, with participation of 9 Polish students of AGH University of Science and Technology. This time, a second phase, which followed the Calibration Phase, was involved. Directly after the arousal evaluation of presented aective pictures, the subjects played a simple platform game (created specically for research purposes [16]), using the same computer. After pressing Enter key on the notebook to start the game, a 30 seconds of blank screen is presented to get the baseline HR and GSR recordings for the beginning of the Gaming Phase. Following this brief inactivity period, tutorial level appears and the subject is allowed to freely explore the game mechanics, and then plays the game for the desired time, but not exceeding 10 minutes. The data is continuously collected with the wristbands worn by the participants, and recorded with our application running on the smartphone.
Most recent attempt of experimental evaluation was conveyed in January 2018, again at the AGH University of Science and Technology. The procedure itself was identical as in the June 2017 experiment, aside from one short additional phase included after the Gaming Phase. The participant was asked to simply sit and look at a relaxing picture displayed on full screen of the computer. After 30 seconds, a sudden unpleasant sound (terried female scream) was played without any warning. This step is applied to get a physiological reading corresponding to a very strong aective response of the subject. The sensory setup was augmented by two measurement platforms: e-Health and BITalino, as well as Neurobit Optima, an advanced physiology data acquisition equipment, used to gather referential HR and GSR data for the devices that are anticipated for AfC purposes. Data of 18 subjects in total was gathered.
Preliminary analysis indicates several issues, but nevertheless provide promising perspectives. The main problems that occurred during the experiments were related to the fact that the devices require special conguration, including repeated pairing attempts before the connection takes place. Tentative observations suggest that HR indeed varies as a function of arousal. Higher arousal values ascribed in the Calibration Phase are indicated by HR scores higher than in the neutral condition, and lower arousal values are associated with lower HR scores. This is consistent with both the general knowledge and the results of other research [13]: if a participant is more excited, her heart beats faster. Therefore, once technical diculties are resolved, interesting results and modes of data interpretation are anticipated.
Our future work regard both extension of the current experimental paradigms, as well as interpretation of the data we are acquiring. We are planning to incorporate audio stimuli, as well as considering the use of virtual reality devices. Moreover, in the work regarding the design of aective games, we would also consider how the interaction during the experiment takes place. This is especial important or the practical development of the aective loop in games [20].
In the near future, we are planning to develop methods of semantic description of the aective data [1]. We are planning to use ontological modeling to this goal. Furthermore, we are investigating methodological requirements for providing eective classiers of emotional states of the users [19]. Currently we are focusing on the short term changes of emotional condition, and basic emotions. However, in the future we are also considering the analysis of long term moods, as well as social emotions, studied using the social signal processing [21].