=Paper=
{{Paper
|id=None
|storemode=property
|title=xAffect - A Modular Framework for Online Affect Recognition and Biofeedback Applications
|pdfUrl=https://ceur-ws.org/Vol-957/matel12_submission_5.pdf
|volume=Vol-957
}}
==xAffect - A Modular Framework for Online Affect Recognition and Biofeedback Applications==
https://ceur-ws.org/Vol-957/matel12_submission_5.pdf
xAffect – A Modular Framework for Online
Affect Recognition and Biofeedback Applications
Kristina Schaaff, Lars Müller, Malte Kirst, and Stephan Heuer
FZI Forschungszentrum Informatik, Karlsruhe, Germany,
[schaaff,lmueller,kirst,heuer]@fzi.de
Abstract. Providing information about the affective state of a person
is getting more and more important in a wide range of learning appli-
cations. For instance biofeedback can be used to reduce stress level or
increase the performance of a person during a learning task. The de-
velopment of wearable physiological sensors drives the development of
applications that provide online biofeedback using sensor data for on-
line analysis. In this paper we present a Java software framework called
xAffect for complex online biofeedback systems that can be used as a
rapid prototyping middleware between physiological sensors and third
party software. We designed xAffect for financial decision making sup-
port, but due to its easy extensibility it is a useful framework for several
other affective computing applications. Biomedical engineers and com-
puter scientists are invited to use xAffect and extend it according to their
requirements.
1 Introduction
The affective state plays an important role for learning. It does not only influ-
ence well-being but also our decision making [10]. The knowledge of a person’s
affective state can help individuals to gain better self-awareness and improve self
emotion regulation. Since the term ’affective computing’ has first been defined as
’computing that relates to, arises from, or deliberately influences emotions’ [11],
a lot of affective applications have evolved. For instance, information about the
affective state of a person can be used to make tutoring systems more human-like
[3] and therefore can help to increase user acceptance of the system. Moreover,
learning environments can be improved by using affective information, e. g. the
content of the learning environment can be presented according to the current
affective state of a person. Serious Gaming is another area where information
about the affective state of a player can be used to enhance game-play.
However, it is hard to objectively measure the affective state of a person. For
instance questionnaires can only provide a snapshot at a certain time and often
lack of objectivity. Therefore, physiological monitoring solutions can be used
in order to provide online feedback to a user as the affective state is strongly
correlated with the physiological state of a person [2].
The development of applications that use online biofeedback in order to cre-
ate affective applications is mainly driven by the fact that recently physiological
monitoring solutions are getting increasingly unobtrusive and cheap.
Developing a system which provides online feedback about the affective state
of a person often involves testing a large number of combinations of different
sensors or algorithms. Designing a biofeedback system as a rapid prototyping
solution can therefore help to minimize the effort for the design of such a system
and to maximize reusability of single system components.
2 State of the art
Few learning applications already use physiological signals to capure affective
aspects, e.g. AutoTutor [3]. However, various commercial products as well as
non-commercial or open source projects for monitoring and analysis of certain
physiological signals are available. Nevertheless, a closer look at these technolo-
gies reveals different drawbacks that a developer has to deal with. Most com-
mercial systems are closed source solutions which do not allow modifications.
Furthermore, they do not provide an interface for custom data access and anal-
ysis. While the software might provide sufficient analysis functionality for a cer-
tain biofeedback application, the choice of sensors coming with this software is
often limited (e. g. with regard to sampling rate, number of available channels
or parameters) and cannot be extended. Moreover, the documentation about
the measuring methods and algorithms used for the data analysis often remains
unclear, which introduces uncertainty into the interpretation of measurement
results. Restrictive license terms and conditions concerning the rights to publish
results and the ownership of acquired data may further constrain the usability
of experiment outputs.
There exist several open source frameworks for design of multimodal and /
or biofeedback applications. However, most of these frameworks are either de-
signed for affective applications such as biofeedback or for multimodal input but
not both at the same time. For instance, solutions like the widely-used BCI2000
software [12] focus mainly on brain computer interfaces and do not provide algo-
rithms for online feedback of other physiological data than EEG. Other solutions
like ICARE [1], CrossWeaver [14] or ICON [4] are mainly designed for develop-
ment of multimodal systems but lack of platform independency. OpenInterface
[8] is another powerful platform which was designed for rapid prototyping of
multimodal systems. However, it is focussed on multimodal interaction design
in general but does not contain components for biofeedback applications.
3 Example Scenario: Biofeedback in Trading Decisions
The affective state of a person has a large influence on how decisions are made
in financial markets. In a study with 80 day-traders, [9] found, that traders with
more intense emotional reactions to monetary losses or gains showed worse trad-
ing performance than traders with less intense emotional reactions. Moreover,
[5] showed, that persons in a positive affective state tend to make more flexible
decisions. Negative affect, on the other hand, determines more rational decision
making. Helping traders to become aware of their current emotional state and to
regulate their emotions according to the market situation can help to improve de-
cision making at financial markets. Figure 1 shows how biofeedback can be used
to support the trader during the trading day. Physiological signals (e. g. heart
rate or electrodermal activity) are used to compute the current affective state
of a trader. This information is combined with data from the trading platform
about the market situation and the trader’s decisions. Subsequently, a feedback
is provided to the trader in which way to regulate his or her emotions to adapt
to the market situation.
Trading
Trading
Platform
Data
Feedback
Software
Sensor
Data
Decisions Trader
Feedback
Emotion
Regulation
Fig. 1. Usage of biofeedback during trading
Developing and testing such a scenario as described above generates various
requirements for an affective monitoring framework in terms of hardware and
software. The basic requirements are:
– Allow connection to different unobtrusive physiological sensors (e. g. ECG,
EDA, respiration) according to the specific application (off-the-shelf or cus-
tom devices).
– Online processing of acquired data to enable real-time feature extraction and
classification in order to provide information about the affective state of the
user.
– Easy integration of new or improved algorithms.
– Possibility to integrate additional information sources (e. g. a trading plat-
form) and to receive control commands (e. g. to start or stop sensors) from
higher level systems.
– Logging of raw data and derived data from any processing step for further
offline analysis or just for logging purposes.
As a consequence, there was a need for a software framework for online affect
recognition that can act as a middleware between physiological sensors and 3rd
party software and meets the requirements for reconfigurability and extensibility.
4 The xAffect Framework
xAffect is implemented as a modular Java framework to fulfill the requirements
outlined above. Our concept of modularizing the system into reconfigurable com-
ponents that can be easily reused is described in the following section along with
the major architectural decisions.
4.1 Concept
Systems for psychophysiological data analysis can be decomposed into three ba-
sic functionalities which are described by using a simple biofeedback application
that visualizes the arousal level based on an ECG signal. First, data has to be
received from a source, in this case an ECG sensor. Subsequently, an algorithm
processes the data and creates a result such as the arousal level. The last step is
to visualize the result to the user of the system. In more complex use cases, like
presented in section 3, multiple sensors and algorithms are combined to create
visualization and store the data for later analysis.
These combinations result in four basic patterns depicted in Figure 2: (a)
chaining, (b) parallel processing, (c) data fusion and (d) distributed processing.
In the following, these patterns are described as each of them results in spe-
cific requirements towards a modular framework for psychophysiological data
analysis.
(a) Chaining (b) Parallel processing
A B C B
A
C
(c) Data fusion (d) Distributed processing
A A B C
C
B A B C
Fig. 2. Basic patterns for online data analysis
The most basic setup (a) consists of a linear concatenation of components;
the output of one component is the input for the next one. Chains of components
can be built that conduct data processing in a sequential manner. When modules
along these chains are reused or replaced by other components, the interfaces
between components have to be standardized. These interfaces combine two
aspects: (i) control interfaces (e. g. how to configure or start a module) and (ii)
data interfaces (e. g. how to encode multi channel data).
A second common pattern is the parallel processing (b) of data from one
single data source or one data processor. It allows to speed up processing as well
as comparing two processing approaches in near real-time. This requires that
each component is implemented as a single thread or process which has to be
managed by the system.
Data fusion (c) is the major challenge for multimodal systems that combine
multiple data sources. A data fusion component has to deal with multiple inputs
that are often unsynchronized due to hardware specific transmission rates and
network delays, e. g. when sensors are connected by bluetooth.
Distributed processing (d) of data across multiple platforms is becoming
more important for mobile and cloud based applications. The requirements are
varying between high bandwidth transmission of raw data and low bandwidth
connections to multiple recipients of calculated features.
4.2 Architecture
The overall architecture of the framework is illustrated in Fig. 3. xAffect breaks
down the whole signal processing chain into a set of interchangeable Components.
These can be classified into three categories:
Network
Data
Sources
GUI
System Manager
Data Dispatcher
Data Control
Processors
Data
Setups
Sinks
Components Core Setup
Fig. 3. Overall architecture and control flow of the xAffect framework
Data sources reflect all sensors, signal generators or timers that feed new data
into the system.
Data processors are responsible for the processing of incoming data. These in-
clude simple data converters as well as complex signal processing algorithms.
Data sinks are responsible for data logging or data transmission. This does
not only include streaming and storing data but also live visualization of the
data and the computed features.
Each component provides automatically multiple outputs. Additionally, they
can be configured for multiple inputs. The components can be flexibly combined
that users can easily create new systems or experiment with different algorithms
reusing components from other settings. As the whole framework is designed
to process data online, each component is running in an independent thread to
enable parallel computation of features whenever possible.
The system configuration is defined in an application specific Setup, which
specifies the correct order of all required components and their individual con-
figuration parameters. xAffect’s Core includes the basic functionality of initial-
ization, the data dispatcher connects all components with queues. The Control
provides the user interface for the framework – either graphical for stand-alone
applications or via network (UDP) for the integration into other software.
4.3 Data Flow
xAffect uses the ideas of the Unisens 2.0 data format to exchange data between
components. Unisens [7] is a generic data format for multi sensor data. It defines
a human readable meta data format in XML that acts as a container for sen-
sor data and annotations from various recording systems. Unisens 2.0 supports
evenly sampled signals from different sensors with different sampling rates as
well as discrete events like a QRS annotation for ECG data sets and unevenly
sampled time discrete measurement values. Therefore, Unisens can be easily ex-
tended to new data formats and allows a rapid prototyping with different data
sources and formats.
All captured and processed data can be stored in Unisens format. This sim-
plifies later debugging and offline analysis of developed systems supported by
the extensive Unisens framework.
4.4 Network Communication
Distributed processing is a central component of xAffect. Specialized data sources
and data sinks can interlink multiple xAffect instances. Two modes of communi-
cation are available: a high bandwidth connection between two instances based
on UDP and a low bandwidth publish subscribe mechanism based on the eXten-
sible Messaging and Presence Protocol (XMPP). XMPP is an IETF standard
that mainly aims at instant messaging. However, it provides a widespread robust
infrastructure to transmit messages to multiple receivers.
5 Prototyping with xAffect
Due to its modular architecture the xAffect framework can easily be integrated
into psychophysiological studies. The customization of xAffect is limited to the
following steps:
1. Make sure that all required Components for signal generation, sensor inte-
gration, signal processing etc. are available. Missing components e. g. for the
integration of new sensors or for new signal processing algorithms have to
be implemented by extending an existing abstract class.
2. Generate a new Setup for the specific use case by defining the order and
configuration parameters of all required components.
Currently components exist for data acquisition covering electrodermal ac-
tivity, cardiac activity, activity monitoring and mouse button press as well as
signal generators and a Unisens data reader. Accordingly, components for fea-
ture computation from these signals have been integrated, as well as components
for data logging in Unisens data format and for data visualization. So far, the
xAffect framework has been integrated in two scenarios where biofeedback from
physiological sensors was required.
In our study about financial decision making, the xAffect framework has
been used in order to connect different sensors to serious games. Goal of the
study was developing a game to train a person’s emotion regulation capabilities
using biofeedback. For this purpose various ECG sensors had to be evaluated for
suitability for the game. Moreover, different kinds of games with different game
concepts (e. g. [6]) were developed and connected to xAffect using the UDP/IP
interface. Due to the modular architecture of the xAffect framework, it was easy
to test several combinations of sensors and games. Based on the raw ECG signal
from the sensors, heart beats were detected and heart rate computed. Using this
data, an arousal value was computed to be fed back into the game (see Fig. 4).
The games reacted to the changes of the arousal level by adapting the difficulty.
Sources ECG Game
Recorder Data
Processors QRS
Detection
Heartrate
Calculation
Arousal
Calculation
Sinks
Game Logging
Fig. 4. Data flow for two different setups
In comparison to available open source software solutions described in Chap-
ter 2, xAffect is focussed on biofeedback from multimodal sensor data. Therefore,
xAffect closes the gap between highly specialized frameworks like BCI2000 and
general signal processing frameworks like OpenInterface. In result, developers
can choose from a variety of sources, processors and sinks to rapidly prototype
customized biofeedback applications.
Due to the light-weight architecture, some of the components developed for
the serious games setup could also be reused in a driving scenario [13] where
heart rate is computed and displayed while a person is driving a car. xAffect
will be extended by more physiological measures such as electrodermal activity
in the future.
6 Conclusion and Future Work
The xAffect framework offers a flexible way to integrate physiological sensors into
online biofeedback scenarios. One of the main benefits of the xAffect framework
is that the system can be configured for very complex scenarios. The integration
effort of a new component is limited to writing an adapter for this component and
to create or update the corresponding setup. Due to the modular architecture
of the software, no changes regarding the core of the software will be necessary.
Moreover, the modular architecture provides interfaces for easy integration of
additional components such as new algorithms or additional sensors. The xAffect
framework including Java source code, examples and documentation is available
on www.xaffect.org under the original BSD license.
Acknowledgements
The research was carried out and funded as part of the xDelia research project
(www.xdelia.org). We gratefully acknowledge funding from the European Com-
mission under the 7th Framework Program, Grant No. 231830.
References
1. Bouchet, J., Nigay, L., Ganille, T.: Icare software components for rapidly develop-
ing multimodal interfaces. In: Proceedings of the 6th international conference on
Multimodal interfaces. pp. 251–258. ICMI ’04, ACM, New York, NY, USA (2004)
2. Cacioppo, J.T., Berntson, G.G., Larsen, J.T., Poehlmann, K.M., Ito, T.A.: The
psychophysiology of emotion, vol. 2, chap. 11, pp. 173–191. The Guilford Press
(2000)
3. D’Mello, S., Jackson, T., Craig, S., Morgan, B., Chipman, P., White, H., Person,
N., Kort, B., el Kaliouby, R., Picard., R., Graesser, A.: Autotutor detects and
responds to learners affective and cognitive states. In: Workshop on Emotional and
Cognitive Issues at the International Conference of Intelligent Tutoring Systems
(2008)
4. Dragicevic, P., Fekete, J.: Input device selection and interaction configuration with
icon. People and Computers pp. 543–448 (2001)
5. Dreisbach, G., Goschke, T.: How positive affect modulates cognitive control: re-
duced perseveration at the cost of increased distractibility. Journal of experimental
psychology Learning memory and cognition 30(2), 343–53 (2004)
6. Jerčić, P., Astor, P.J., Adam, M.T., Hilborn, O., Schaaff, K., Lindley, C., Senner-
sten, C., Eriksson, J.: A serious game using physiological interfaces for emotion
regulation training in the context of financial decision-making. ECIS 2012 (2012)
7. Kirst, M., Ottenbacher, J.: Unisens – a universal data format (2008),
http://www.unisens.org, http://www.unisens.org
8. Lawson, J.Y.L., Vanderdonckt, J., Macq, B.: OpenInterface: A lightweight open
source platform for rapid prototyping of multimodal applications. In: Proc. of 1st
Int. Workshop on User Interface Extensible Markup Language UsiXML2010 (2010)
9. Lo, A.W., Repin, D.V., Steenbarger, B.N.: Fear and greed in financial markets: A
clinical study of day-traders. NBER Working Papers 11243, National Bureau of
Economic Research, Inc (2005)
10. Loewenstein, G., Lerner, J.S.: The role of affect in decision making. In: David-
son, R.J., Goldsmith, H.H., Scherer, K.R. (eds.) Handbook of Affective Science,
chap. 31, pp. 619–642. Series in Affective Science, Oxford University Press, Oxford,
New York (2003)
11. Picard, R.: Affective computing. Tech. Rep. 321, MIT Media Laboratory, Percep-
tual Computing Section (November 1995)
12. Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., Wolpaw, J.R.:
BCI2000: a general-purpose brain-computer interface (BCI) system. Biomedical
Engineering, IEEE Transactions on 51(6), 1034–1043 (Jun 2004)
13. Schneider, J., Koellner, C., Heuer, S.: An approach to automotive ECG measure-
ment validation using a car-integrated test framework. accepted for publication at
2012 IEEE Intelligent Vehicles Symposium, Alcala de Henares, Spain (2012)
14. Sinha, A.K., Landay, J.A.: Capturing user tests in a multimodal, multidevice in-
formal prototyping tool. In: Proceedings of the 5th international conference on
Multimodal interfaces. pp. 117–124. ICMI ’03, ACM, New York, NY, USA (2003)