=Paper=
{{Paper
|id=Vol-2166/afcai18-paper12
|storemode=property
|title=Development of Mobile Platform for Affect Interpretation. Current Progress
|pdfUrl=https://ceur-ws.org/Vol-2166/afcai18-paper12.pdf
|volume=Vol-2166
|authors=Grzegorz J Nalepa,Krzysztof Kutt,Szymon Bobek,Barbara Giżycka
|dblpUrl=https://dblp.org/rec/conf/afcai/NalepaKBG18
}}
==Development of Mobile Platform for Affect Interpretation. Current Progress==
https://ceur-ws.org/Vol-2166/afcai18-paper12.pdf
Development of Mobile Platform for Aect
Interpretation. Current Progress.
Grzegorz J. Nalepa, Krzysztof Kutt,
Szymon Bobek, and Barbara Gi»ycka
AGH University of Science and Technology
Al. Mickiewicza 30, 30-059 Krakow, Poland
{gjn,kkutt,sbobek,bgizycka}@agh.edu.pl
Keywords: context-aware systems, aective computing, mobile devices.
1 Introduction
Aective computing (AfC) [18] is a relatively well established domain interested
in the computer-based interpretation and synthesis of emotion, and aect-related
signal. To solve its important challenges, it combines results of several domains,
including computer science, philosophy, psychology, cognitive science, and more
recently neuroscience and biomedical engineering [6].
In our previous works, we started with experiments based on wearable sen-
sors, and a virtual reality environment to stimulate and evoke emotions [14].
Furthermore, we introduced the proposal of using the context-aware systems
paradigm to develop a platform for AfC, that could be used in ambient intel-
ligence applications [17]. We described the design and the preliminary work on
this platform in [15]. Later on, we turned our attention to computer games [8],
as an interesting environment to design aective experiments [16]. As we focus
on wearable devices, in our recent paper we provided results of a critical analysis
of measurements using the selected devices [9]. Moreover, we developed our own
software for mobile phones [10] for aective data acquisition.
In this overview paper we focus on our recent progress in the work on the
mobile platform for AfC. The rest of the paper is organized as follows. In the
next section we provide the main assumptions about the platform, as well as
describe aective data acquisition and interpretation. We then discuss our most
recent experiments and provide an outlook of our future works.
2 Outline of the Emotion Interpretation Platform
Our goal is to develop a computer platform allowing for detection, recording
and interpretation of aective data. This platform should enable opportunities
to use this data in personalized computer systems. In our work we are interested
in ubiquitous computing solutions. As such, we assume the platform uses a range
of sensors to acquire data about condition and behavior of an individual user.
These sensors should be wearable and least intrusive. Furthermore, we assume
harvesting other available data, coming from general use mobile devices, i.e.
sensors of mobile phones, etc.
The data processing and fusion is conducted using the context-aware systems
paradigm [4]. We realize, that in such a setting the data can be only partially
available and reliable. As such we build on our previous work, where we con-
sidered uncertainty handling in mobile context-aware systems [5]. We assume a
layered approach, with a data acquisition layer from diverse sensors, and inter-
pretation layer incorporating several arti cial intelligence methods. These two
main layers are described next.
2.1 Aective Data Acquisition
Our current focus is on the physiological data acquired by wearable measurement
devices. We are focusing on advanced o the shelf wristbands, as well as an open
measurement platform BITalino. Data acquisition from such a range of devices
needs a universal software solution that meets several requirements for being a
convenient asset for AfC studies:
1. The technology should introduce modular architecture to be easily expanded
to new devices support and new functionalities.
2. The solution should give a possibility to save data from all available sensors
for each device.
3. The applications should facilitate recording data from several dierent wrist-
bands simultaneously, as it is a common experimental scenario.
4. The software should allow synchronization with other data sources (e.g. pro-
cedure markers from PC).
5. There should be a possibility for arbitrary le naming and tagging to provide
more control over data and to facilitate future processing.
6. The data should be saved in convenient le format (such as i.e. CSV).
7. The data should be available for other applications that run on the same
device, i.e. there should be a service that feeds clients (other applications)
with the current stream of data.
8. The application should output raw data gathered from wristband(s).
There is a lack of such state-of-the-art solutions. One attempt to create such
an application was done in 2011, when PsychLog project1 [7] was developed. In
addition to the module responsible for sensory data collection, it also had survey
module for self-report questionnaires and visualization module for presenting
gathered signals. Unfortunately, the project has been abandoned and only beta
version for Windows Mobile platform was released.
As a response for the need, we developed our own mobile software [10] for
data acquisition, which satis es most of the identi ed requirements. It is built
as a set of interfaces that should be implemented to communicate with the
speci c device: WristBand for de nition of the band, its possible states and list of
sensors available; WristBandService for supporting every stage of communication
1
See: http://sourceforge.net/projects/psychlog/.
with the band: establishing a connection through the provided API, starting the
transmission of raw data, handling errors; and WristBandFragment for de nition
of UI panel for speci c wirstband. Application has also generic modules for
handling the gathered data and for serialization of data streams to CSV les.
Current version of application supports Empatica E4, Microsoft Band 2 and
Scosche Rhythm+ devices.
Developed application was used in several studies (see Sect. 3). In particular,
it was used in our recent analysis aimed at critical comparison of quality of HR
and GSR signals in selected devices that oer real-time monitoring with access
to raw data: Empatica E4, Microsoft Band 2, e-Health Sensor Platform V2.0
and BITalino (r)evolution kit [9]. Results demonstrated that BITalino remains
the most prospective platform for both HR and GSR measurements, especially
as the technical support for the e-Health is being phased out. As for secondary
HR readings, MS Band 2 can be used. While the BITalino kit does not have the
wristband form, it can be turned into a wearable using the BITalino Freestyle
kit and 3D-printed boxes.
2.2 Interpretation of Aective Data
First of all, we treat the aective data as context data. We de ne two types of
such data. Low-level information which can be referred as physiological signals
delivered by hardware sensors, and high-level information that can be derived
based on low-level context and is interpreted as emotional state of the user. Such
an approach allowed us to adapt methods from the eld of mobile context-aware
systems developed by us in our previous research [15] to the area of AfC. In par-
ticular, we integrate the architecture for developing mobile context-aware system
with methods for aective data acquisition and machine learning algorithms for
emotion recognition to propose the architecture for aective data processing and
interpretation. The architecture is given in Fig. 1
Data from single sensor is read by Context Provider Plugin which is part
of AWARE framework2 that allows to instrument, infer, log and share mobile
context information. The Context data delivered by the AWARE plugins is sent
to Emotion Recognition module responsible for classi cation of user emotional
state. Emotion classi cation can have polymorphic form as the system can be
dynamically recon gured with Con guration Rule-based system. This allows for
on-line swapping of emotion recognition modules in context dependent manner.
Such modules can be based on face expressions analysis, voice recognition, mobile
phone usage patterns, or physiological signals, etc. Many of these approaches
use data which interpretation highly depend on the user pro le. In some cases
such data can be unique to users (e.g. mobile phone usage patterns), making it
impossible to build models in advance from anonymous data sets.
Therefore, we argue that the emotion recognition modules should be based
on online learning mechanisms that allow to discover knowledge about the user
from streaming data. Such data can be additionally uncertain and may evolve
2
See: http://awareframework.com
Fig. 1. Architecture of dynamic framework for context aware systems extended with
emotion recognition and mediation. [11]
over time exposing so called concept-drift, hence the learning mechanism has to
take all of these into consideration. Furthermore, it should generate models that
are understandable by the user and can be modi ed by one. In our research we
proposed uncertain decision tree generation algorithm for building decision trees
from in nite training sets with uncertain instances and class labels. Additionally,
we proposed a translator that converts these trees into human readable, visual
rule-based representation [3]. This representation is based on the same rule-based
language, called HMR+ [2], that is used by the Con guration Rule-based system
module. Hence, both the dynamic system recon guration and reasoning about
emotional condition of a person can be conducted by the use of our dedicated in-
ference engine HeaRTDroid3 . Similarly, both elements can be directly modi ed
by the user with a usage of explicit and implicit mediation techniques [2].
All of the components presented in this section were designed to work on
mobile devices. This allows to run the system solely on the mobile device. Such
feature is desirable in systems that operate on highly sensitive data (such as
physiological information) or is expected to work correctly without access to the
Internet or other external resources (e.g. online health monitoring systems).
3 Recent Experiments
To this day, several experiments for verifying our approach have been conducted.
With respect to speci c characteristics of the experimental procedure, three ver-
sions of the experiment can be distinguished. Throughout each of the studies
3
See: https://bitbucket.org/sbobek/heartdroid
conducted, the procedure was gradually improved and expanded with additional
sensors to gather data and experimental tasks to provide interesting and mean-
ingful research situation. This section provides an overview of the three experi-
ments handled so far, together with the descriptions of experimental procedures
applied to those experiments, respectively.
The rst attempt of experimental evaluation of the approach took place in
late April 2017 (Eurokreator lab, Krakow, Poland). The subjects were 2 male
and 4 female, who participated in the workshops held in the lab at the time.
The sensors used included Empatica E4 and MS Band 2 wristbands, paired with
our data recording application. The computer and the smartphone app are syn-
chronized during the whole procedure. In this experiment, people were asked to
seat in front of the computer and evaluate their arousal on 7-levels scale [1, 7]
in reaction to visual stimuli (pictures from Nencki Aective Picture System,
see [12]) presented one by one. The subjects were exposed to three series of
18 pictures, where each was displayed for a xed time (3 seconds per stimuli),
with 30 second breaks between each session. Before the rst group of pictures,
30 seconds of blank screen was also presented, in order to acquire baseline HR
and GSR recordings during participants inactivity. The whole experiment lasted
about 15 minutes. Later, the procedure was extended to include another exper-
imental phase, so that picture exposition was engaged as a Calibration Phase.
Its purpose is to address the problem of individual dierences between baseline
physiological activity and reaction patterns of various subjects.
The second experiment was conducted in June 2017, with participation of
9 Polish students of AGH University of Science and Technology. This time, a
second phase, which followed the Calibration Phase, was involved. Directly after
the arousal evaluation of presented aective pictures, the subjects played a simple
platform game (created speci cally for research purposes [16]), using the same
computer. After pressing Enter key on the notebook to start the game, a 30
seconds of blank screen is presented to get the baseline HR and GSR recordings
for the beginning of the Gaming Phase. Following this brief inactivity period,
tutorial level appears and the subject is allowed to freely explore the game
mechanics, and then plays the game for the desired time, but not exceeding 10
minutes. The data is continuously collected with the wristbands worn by the
participants, and recorded with our application running on the smartphone.
Most recent attempt of experimental evaluation was conveyed in January
2018, again at the AGH University of Science and Technology. The procedure
itself was identical as in the June 2017 experiment, aside from one short ad-
ditional phase included after the Gaming Phase. The participant was asked to
simply sit and look at a relaxing picture displayed on full screen of the computer.
After 30 seconds, a sudden unpleasant sound (terri ed female scream) was played
without any warning. This step is applied to get a physiological reading corre-
sponding to a very strong aective response of the subject. The sensory setup
was augmented by two measurement platforms: e-Health and BITalino, as well
as Neurobit Optima, an advanced physiology data acquisition equipment, used
to gather referential HR and GSR data for the devices that are anticipated for
AfC purposes. Data of 18 subjects in total was gathered.
3.1 Preliminary Evaluation of the Experiments
Preliminary analysis indicates several issues, but nevertheless provide promising
perspectives. The main problems that occurred during the experiments were re-
lated to the fact that the devices require special con guration, including repeated
pairing attempts before the connection takes place. Tentative observations sug-
gest that HR indeed varies as a function of arousal. Higher arousal values ascribed
in the Calibration Phase are indicated by HR scores higher than in the neutral
condition, and lower arousal values are associated with lower HR scores. This is
consistent with both the general knowledge and the results of other research [13]:
if a participant is more excited, her heart beats faster. Therefore, once technical
di culties are resolved, interesting results and modes of data interpretation are
anticipated.
4 Future Works
Our future work regard both extension of the current experimental paradigms,
as well as interpretation of the data we are acquiring. We are planning to in-
corporate audio stimuli, as well as considering the use of virtual reality devices.
Moreover, in the work regarding the design of aective games, we would also
consider how the interaction during the experiment takes place. This is especial
important or the practical development of the aective loop in games [20].
In the near future, we are planning to develop methods of semantic descrip-
tion of the aective data [1]. We are planning to use ontological modeling to this
goal. Furthermore, we are investigating methodological requirements for provid-
ing eective classi ers of emotional states of the users [19]. Currently we are
focusing on the short term changes of emotional condition, and basic emotions.
However, in the future we are also considering the analysis of long term moods,
as well as social emotions, studied using the social signal processing [21].
References
1. Berthelon, F., Sander, P.: Emotion ontology for context awareness. In: 2013 IEEE
4th International Conference on Cognitive Infocommunications (CogInfoCom). pp.
5964 (Dec 2013)
2. Bobek, S.: Methods for modeling self-adaptive mobile context-aware sytems. Ph.D.
thesis, AGH University of Science and Technology (April 2016), Supervisor: Grze-
gorz J. Nalepa
3. Bobek, S., Misiak, P.: Uncertain decision tree classi er for mobile context-aware
and aective computing, submitted to ICAISC 2018
4. Bobek, S., Nalepa, G.J.: Uncertain context data management in dynamic mobile
environments. Future Generation Computer Systems 66, 110 124 (2017), http:
//www.sciencedirect.com/science/article/pii/S0167739X1630187X
5. Bobek, S., Nalepa, G.J.: Uncertainty handling in rule-based mobile context-aware
systems. Pervasive and Mobile Computing 39(August), , August 2017 (2017), http:
//www.sciencedirect.com/science/article/pii/S1574119216302115
6. Calvo, R.A., D'Mello, S.K., Gratch, J., Kappas, A. (eds.): The Oxford Handbook
of Aective Computing. Oxford Library of Psychology, Oxford University Press
(2015)
7. Gaggioli, A., Pioggia, G., Tartarisco, G., Baldus, G., Corda, D., Cipresso, P.,
Riva, G.: A mobile data collection platform for mental health research. Personal
and Ubiquitous Computing 17(2), 241251 (2013), https://doi.org/10.1007/
s00779-011-0465-2
8. Hudlicka, E.: Aective computing for game design. In: Proceedings of the 4th
International North American Conference on Intelligent Games and Simulation
(GAMEON-NA) (2008)
9. Kutt, K., Binek, W., Misiak, P., Nalepa, G.J., Bobek, S.: Towards the development
of sensor platform for aective computing. critical evaluation of measurement de-
vices, submitted to ICAISC 2018
10. Kutt, K., Nalepa, G.J., Gi»ycka, B., Jemioªo, P., Adamczyk, M.: Bandreader a
mobile application for data acquisition from wearable devices in aective comput-
ing experiments, submitted to ICAISC 2018
11. ¦picki, M.Z., Bobek, S.: Aective context-aware systems: architecture of dynamic
mobile solution, submitted to ICAISC 2018
12. Marchewka, A., urawski, ., Jednoróg, K., Grabowska, A.: The Nencki Aec-
tive Picture System (NAPS): Introduction to a novel, standardized, wide-range,
high-quality, realistic picture database. Behavior Research Methods 46(2), 596610
(2014)
13. Mauss, I.B., Robinson, M.D.: Measures of emotion: A review. Cognition and Emo-
tion 23(2), 209237 (2009)
14. Nalepa, G.J., Argasinski, J.K., Kutt, K., Wegrzyn, P., Bobek, S., Lepicki, M.Z.:
Aective computing experiments in virtual reality with wearable sensors. method-
ological considerations and preliminary results. In: Ezquerro, M.T.H., Nalepa, G.J.,
Mendez, J.T.P. (eds.) Proceedings of the Workshop on Aective Computing and
Context Awareness in Ambient Intelligence (AfCAI 2016). CEUR Workshop Pro-
ceedings, vol. 1794 (2016), http://ceur-ws.org/Vol-1794/
15. Nalepa, G.J., Bobek, S., Kutt, K.: Mobile platform for aective context-aware
systems. Future Generation Computer Systems (2017), in review
16. Nalepa, G.J., Gizycka, B., Kutt, K., Argasinski, J.K.: Aective design patterns in
computer games. scrollrunner case study. In: Communication Papers of the 2017
Federated Conference on Computer Science and Information Systems, FedCSIS
2017. pp. 345352 (2017), https://doi.org/10.15439/2017F192
17. Nalepa, G.J., Kutt, K., Bobek, S., Lepicki, M.Z.: AfCAI systems: Aective com-
puting with context awareness for ambient intelligence. research proposal. In: Ez-
querro, M.T.H., Nalepa, G.J., Mendez, J.T.P. (eds.) Proceedings of the Work-
shop on Aective Computing and Context Awareness in Ambient Intelligence (Af-
CAI 2016). CEUR Workshop Proceedings, vol. 1794 (2016), http://ceur-ws.org/
Vol-1794/
18. Picard, R.W.: Aective Computing. MIT Press (1997)
19. Rincon, J.A., Costa, Â., Novais, P., Julián, V., Carrascosa, C.: Using non-
invasive wearables for detecting emotions with intelligent agents. In: Graña,
M., López-Guede, J.M., Etxaniz, O., Herrero, Á., Quintián, H., Corchado, E.
(eds.) International Joint Conference SOCO'16-CISIS'16-ICEUTE'16 - San Se-
bastián, Spain, October 19th-21st, 2016, Proceedings. Advances in Intelligent Sys-
tems and Computing, vol. 527, pp. 7384 (2016), https://doi.org/10.1007/
978-3-319-47364-2_8
20. Tijs, T., Brokken, D., Ijsselsteijn, W.: Creating an emotionally adaptive game.
In: Proceedings of the 7th International Conference on Entertainment Computing.
pp. 122133. ICEC '08, Springer-Verlag, Berlin, Heidelberg (2009), http://dx.
doi.org/10.1007/978-3-540-89222-9_14
21. Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: Survey of an
emerging domain. Image Vision Comput. 27(12), 17431759 (2009), https://doi.
org/10.1016/j.imavis.2008.11.007