Providing information about the a ective state of a person is getting more and more important in a wide range of learning applications. For instance biofeedback can be used to reduce stress level or increase the performance of a person during a learning task. The development of wearable physiological sensors drives the development of applications that provide online biofeedback using sensor data for online analysis. In this paper we present a Java software framework called xA ect for complex online biofeedback systems that can be used as a rapid prototyping middleware between physiological sensors and third party software. We designed xA ect for nancial decision making support, but due to its easy extensibility it is a useful framework for several other a ective computing applications. Biomedical engineers and computer scientists are invited to use xA ect and extend it according to their requirements.
The a ective state plays an important role for learning. It does not only in
uence well-being but also our decision making [10]. The knowledge of a person's
a ective state can help individuals to gain better self-awareness and improve self
emotion regulation. Since the term 'a ective computing' has rst been de ned as
'computing that relates to, arises from, or deliberately in uences emotions' [11],
a lot of a ective applications have evolved. For instance, information about the
a ective state of a person can be used to make tutoring systems more human-like
[
However, it is hard to objectively measure the a ective state of a person. For
instance questionnaires can only provide a snapshot at a certain time and often
lack of objectivity. Therefore, physiological monitoring solutions can be used
in order to provide online feedback to a user as the a ective state is strongly
correlated with the physiological state of a person [
The development of applications that use online biofeedback in order to create a ective applications is mainly driven by the fact that recently physiological monitoring solutions are getting increasingly unobtrusive and cheap.
Developing a system which provides online feedback about the a ective state of a person often involves testing a large number of combinations of di erent sensors or algorithms. Designing a biofeedback system as a rapid prototyping solution can therefore help to minimize the e ort for the design of such a system and to maximize reusability of single system components. 2
Few learning applications already use physiological signals to capure a ective
aspects, e.g. AutoTutor [
There exist several open source frameworks for design of multimodal and /
or biofeedback applications. However, most of these frameworks are either
designed for a ective applications such as biofeedback or for multimodal input but
not both at the same time. For instance, solutions like the widely-used BCI2000
software [12] focus mainly on brain computer interfaces and do not provide
algorithms for online feedback of other physiological data than EEG. Other solutions
like ICARE [
Example Scenario: Biofeedback in Trading Decisions
The a ective state of a person has a large in uence on how decisions are made
in nancial markets. In a study with 80 day-traders, [9] found, that traders with
more intense emotional reactions to monetary losses or gains showed worse
trading performance than traders with less intense emotional reactions. Moreover,
[
Developing and testing such a scenario as described above generates various requirements for an a ective monitoring framework in terms of hardware and software. The basic requirements are: { Allow connection to di erent unobtrusive physiological sensors (e. g. ECG, EDA, respiration) according to the speci c application (o -the-shelf or custom devices). { Online processing of acquired data to enable real-time feature extraction and classi cation in order to provide information about the a ective state of the user. { Easy integration of new or improved algorithms. { Possibility to integrate additional information sources (e. g. a trading platform) and to receive control commands (e. g. to start or stop sensors) from higher level systems. { Logging of raw data and derived data from any processing step for further o ine analysis or just for logging purposes.
As a consequence, there was a need for a software framework for online a ect recognition that can act as a middleware between physiological sensors and 3rd party software and meets the requirements for recon gurability and extensibility.
The xA
ect Framework xA ect is implemented as a modular Java framework to ful ll the requirements outlined above. Our concept of modularizing the system into recon gurable components that can be easily reused is described in the following section along with the major architectural decisions. 4.1
Concept Systems for psychophysiological data analysis can be decomposed into three basic functionalities which are described by using a simple biofeedback application that visualizes the arousal level based on an ECG signal. First, data has to be received from a source, in this case an ECG sensor. Subsequently, an algorithm processes the data and creates a result such as the arousal level. The last step is to visualize the result to the user of the system. In more complex use cases, like presented in section 3, multiple sensors and algorithms are combined to create visualization and store the data for later analysis.
These combinations result in four basic patterns depicted in Figure 2: (a) chaining, (b) parallel processing, (c) data fusion and (d) distributed processing. In the following, these patterns are described as each of them results in speci c requirements towards a modular framework for psychophysiological data analysis.
(a) Chaining
A (c) Data fusion
A B
B C
C (b) Parallel processing
B A A
C
B (d) Distributed processing
A B C
C
The most basic setup (a) consists of a linear concatenation of components; the output of one component is the input for the next one. Chains of components can be built that conduct data processing in a sequential manner. When modules along these chains are reused or replaced by other components, the interfaces between components have to be standardized. These interfaces combine two aspects: (i) control interfaces (e. g. how to con gure or start a module) and (ii) data interfaces (e. g. how to encode multi channel data).
A second common pattern is the parallel processing (b) of data from one single data source or one data processor. It allows to speed up processing as well as comparing two processing approaches in near real-time. This requires that each component is implemented as a single thread or process which has to be managed by the system.
Data fusion (c) is the major challenge for multimodal systems that combine multiple data sources. A data fusion component has to deal with multiple inputs that are often unsynchronized due to hardware speci c transmission rates and network delays, e. g. when sensors are connected by bluetooth.
Distributed processing (d) of data across multiple platforms is becoming more important for mobile and cloud based applications. The requirements are varying between high bandwidth transmission of raw data and low bandwidth connections to multiple recipients of calculated features. 4.2
Architecture The overall architecture of the framework is illustrated in Fig. 3. xA ect breaks down the whole signal processing chain into a set of interchangeable Components. These can be classi ed into three categories:
Components r e h c t a p s i D a t a D
Core r e g a n a M m e t s y S
Data sources re ect all sensors, signal generators or timers that feed new data into the system.
Data processors are responsible for the processing of incoming data. These include simple data converters as well as complex signal processing algorithms. Data sinks are responsible for data logging or data transmission. This does not only include streaming and storing data but also live visualization of the data and the computed features.
Each component provides automatically multiple outputs. Additionally, they can be con gured for multiple inputs. The components can be exibly combined that users can easily create new systems or experiment with di erent algorithms reusing components from other settings. As the whole framework is designed to process data online, each component is running in an independent thread to enable parallel computation of features whenever possible.
The system con guration is de ned in an application speci c Setup, which speci es the correct order of all required components and their individual conguration parameters. xA ect's Core includes the basic functionality of initialization, the data dispatcher connects all components with queues. The Control provides the user interface for the framework { either graphical for stand-alone applications or via network (UDP) for the integration into other software. 4.3
Data Flow xA ect uses the ideas of the Unisens 2.0 data format to exchange data between components. Unisens [7] is a generic data format for multi sensor data. It de nes a human readable meta data format in XML that acts as a container for sensor data and annotations from various recording systems. Unisens 2.0 supports evenly sampled signals from di erent sensors with di erent sampling rates as well as discrete events like a QRS annotation for ECG data sets and unevenly sampled time discrete measurement values. Therefore, Unisens can be easily extended to new data formats and allows a rapid prototyping with di erent data sources and formats.
All captured and processed data can be stored in Unisens format. This simpli es later debugging and o ine analysis of developed systems supported by the extensive Unisens framework. 4.4
Network Communication Distributed processing is a central component of xA ect. Specialized data sources and data sinks can interlink multiple xA ect instances. Two modes of communication are available: a high bandwidth connection between two instances based on UDP and a low bandwidth publish subscribe mechanism based on the eXtensible Messaging and Presence Protocol (XMPP). XMPP is an IETF standard that mainly aims at instant messaging. However, it provides a widespread robust infrastructure to transmit messages to multiple receivers. 5
Due to its modular architecture the xA ect framework can easily be integrated into psychophysiological studies. The customization of xA ect is limited to the following steps: 1. Make sure that all required Components for signal generation, sensor integration, signal processing etc. are available. Missing components e. g. for the integration of new sensors or for new signal processing algorithms have to be implemented by extending an existing abstract class. 2. Generate a new Setup for the speci c use case by de ning the order and con guration parameters of all required components.
Currently components exist for data acquisition covering electrodermal activity, cardiac activity, activity monitoring and mouse button press as well as signal generators and a Unisens data reader. Accordingly, components for feature computation from these signals have been integrated, as well as components for data logging in Unisens data format and for data visualization. So far, the xA ect framework has been integrated in two scenarios where biofeedback from physiological sensors was required.
In our study about nancial decision making, the xA ect framework has
been used in order to connect di erent sensors to serious games. Goal of the
study was developing a game to train a person's emotion regulation capabilities
using biofeedback. For this purpose various ECG sensors had to be evaluated for
suitability for the game. Moreover, di erent kinds of games with di erent game
concepts (e. g. [
Sources Processors Sinks
ECG Recorder
QRS Detection Heartrate Calculation
Arousal Calculation
Game
Game
Data
Logging
In comparison to available open source software solutions described in Chapter 2, xA ect is focussed on biofeedback from multimodal sensor data. Therefore, xA ect closes the gap between highly specialized frameworks like BCI2000 and general signal processing frameworks like OpenInterface. In result, developers can choose from a variety of sources, processors and sinks to rapidly prototype customized biofeedback applications.
Due to the light-weight architecture, some of the components developed for the serious games setup could also be reused in a driving scenario [13] where heart rate is computed and displayed while a person is driving a car. xA ect will be extended by more physiological measures such as electrodermal activity in the future. 6
The xA ect framework o ers a exible way to integrate physiological sensors into online biofeedback scenarios. One of the main bene ts of the xA ect framework is that the system can be con gured for very complex scenarios. The integration e ort of a new component is limited to writing an adapter for this component and to create or update the corresponding setup. Due to the modular architecture of the software, no changes regarding the core of the software will be necessary. Moreover, the modular architecture provides interfaces for easy integration of additional components such as new algorithms or additional sensors. The xA ect framework including Java source code, examples and documentation is available on www.xa ect.org under the original BSD license.
The research was carried out and funded as part of the xDelia research project (www.xdelia.org). We gratefully acknowledge funding from the European Commission under the 7th Framework Program, Grant No. 231830. 7. Kirst, M., Ottenbacher, J.: Unisens { a universal data format (2008), http://www.unisens.org, http://www.unisens.org 8. Lawson, J.Y.L., Vanderdonckt, J., Macq, B.: OpenInterface: A lightweight open source platform for rapid prototyping of multimodal applications. In: Proc. of 1st Int. Workshop on User Interface Extensible Markup Language UsiXML2010 (2010) 9. Lo, A.W., Repin, D.V., Steenbarger, B.N.: Fear and greed in nancial markets: A clinical study of day-traders. NBER Working Papers 11243, National Bureau of Economic Research, Inc (2005) 10. Loewenstein, G., Lerner, J.S.: The role of a ect in decision making. In: Davidson, R.J., Goldsmith, H.H., Scherer, K.R. (eds.) Handbook of A ective Science, chap. 31, pp. 619{642. Series in A ective Science, Oxford University Press, Oxford, New York (2003) 11. Picard, R.: A ective computing. Tech. Rep. 321, MIT Media Laboratory, Perceptual Computing Section (November 1995) 12. Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., Wolpaw, J.R.: BCI2000: a general-purpose brain-computer interface (BCI) system. Biomedical Engineering, IEEE Transactions on 51(6), 1034{1043 (Jun 2004) 13. Schneider, J., Koellner, C., Heuer, S.: An approach to automotive ECG measurement validation using a car-integrated test framework. accepted for publication at 2012 IEEE Intelligent Vehicles Symposium, Alcala de Henares, Spain (2012) 14. Sinha, A.K., Landay, J.A.: Capturing user tests in a multimodal, multidevice informal prototyping tool. In: Proceedings of the 5th international conference on Multimodal interfaces. pp. 117{124. ICMI '03, ACM, New York, NY, USA (2003)