A Framework for Rapid Prototyping of Multimodal Interaction Concepts Ronny Seiger Florian Niebling Mandy Korzetz Technische Universität Technische Universität Technische Universität Dresden Dresden Dresden Dresden, Germany Dresden, Germany Dresden, Germany ronny.seiger@tu-dresden.de florian.niebling@tu-dresden.de mandy.korzetz@tu-dresden.de Tobias Nicolai Thomas Schlegel Technische Universität Technische Universität Dresden Dresden Dresden, Germany Dresden, Germany tobias.nicolai@mailbox.tu- thomas.schlegel@tu- dresden.de dresden.de ABSTRACT to the high degree of embeddedness of pervasive systems, Ubiquitous systems provide users with various possibilities of ubiquitous systems are characterized by a high level of mobil- interacting with applications and components using different ity, often consisting of a large number of heterogeneous and modalities and devices. To offer the most appropriate mode possibly resource-limited devices, which are loosely-coupled of interaction in a given context, various types of sensors are into dynamic network infrastructures. The emergence of combined to create interactive applications. Thus, the need smart spaces (smart homes, smart factories, smart offices) for integrated development and evaluation of suitable inter- shows the increasing importance and spreading of ubiquitous action concepts for ubiquitous systems increases. Creation systems throughout different areas of everyday life. of prototypes for interactions is a complex and time consum- As user interaction with devices that disappear into the back- ing part of iterative software engineering processes, currently ground often cannot be realized using traditional metaphors, not well supported by tools as prototypes are considered to new ways of interaction have to be explored during the be short-living software artifacts. In this paper, we introduce software engineering process for creating ubiquitous system the framework Connect that enables rapid prototyping of in- components. The development and improvement of multi- teraction concepts with a focus on software engineering as- modal interaction concepts (i. e., interactions using one or pects. The framework allows the definition and modification more input modalities [21]) is thereby not limited to initial of event-to-action mappings for arbitrary interaction devices prototyping, but equally important during implementation, and applications. By applying Connect, model-based proto- testing and evaluation stages. types of multimodal interaction concepts involving multiple devices can be created, evaluated and refined during the en- Existing tools for interaction design and rapid prototyping tire engineering process. of ubiquitous user interaction can be successfully employed during the initial prototyping phases. In later development ACM Classification Keywords phases as well as in iterative engineering processes such as H.5.2 User Interfaces: User-centered design user-centered design (UCD), their applicability is often re- stricted due to their limited ability to automatically propagate Author Keywords changes in prototypes to subsequent stages of development. interaction framework; interaction concepts; multimodal The mostly informal descriptions and implementations of in- interaction; rapid prototyping; software engineering teraction concepts and interactive applications limit their ex- tensibility with respect to new interaction devices and modal- INTRODUCTION ities. The lack of models and formalism also prevents pro- Ubiquitous Systems as coined by Weiser [22] describe user- totypes for interactions from being used and matured in later centered systems at the intersection of mobile and pervasive stages of the development process, which is why prototypes computing combined with ambient intelligence. In addition are usually considered to be short-living software artifacts. In this paper, we propose a model-driven framework for pro- totyping of interaction concepts that can be applied through- Workshop on Large-scale and model-based Interactive Systems: Approaches and out the different phases of iterative software engineering pro- Challenges, June 23 2015, Duisburg, Germany. Copyright © 2015 for the individual papers by the papers’ authors. Copying permitted cesses. The focus of the introduced Connect framework for only for private and academic purposes. This volume is published and copyrighted by interaction design is placed on software engineering aspects its editors. and models. It enables the rapid development of prototypes Interface workbench [13] provides developers with a compre- and enhancement at runtime during expert evaluation and user hensive toolset for configuring components and interactions. studies. Extensibility concerning new types of interaction de- Our aim is to provide an easy to use tool also enabling non- vices as well as interactive components is easily achieved by programmers to rapidly create interaction prototypes. inheritance in the object-oriented framework architecture. As a result of a high-level, model-based design of interaction In [10] Hoste et al. describe the Mudra framework for fusing concepts, modifications to the interactions–even of very early events from multiple interaction devices in order to leverage prototypes–can be reused and advanced from beginning to multimodal interactions. Mudra provides means for process- end of the development cycle. The framework supports indi- ing low-level sensor events and for inferring high-level se- mantic interaction events based on a formal declarative lan- vidualizations concerning different groups of users as well as guage. The framework is primarily focused on combining distinct scenarios by customizing interaction concepts using multiple interaction devices to enable advanced forms of in- specialization and configuration. We introduce a tool based teraction input. It can be used complementary to the Connect on the Connect framework that facilitates the creation and framework as part of defining and integrating low-level and customization of interaction concepts at runtime even by non- programmers. The framework is demonstrated developing an high-level sensor components. However, the application of interaction prototype within a Smart Home–a ubiquitous en- the formalism and semantic models that Mudra is based on vironment consisting of various devices for multimodal inter- increases the effort and complexity for rapidly prototyping actions with physical and virtual entities. interaction concepts and introduces a number of additional components to the lightweight Connect framework. RELATED WORK In addition to the work of Dey et al. [6], one of the first Prototypes are useful tools for assessing design decisions and conceptual frameworks for the rapid prototyping of context- concepts in early but also in advanced stages of the software aware applications predominant in ubiquitous systems, the engineering process. Especially in iterative design processes iStuff toolkit [3] and its mobile extension by Ballagas et al. [2] for creating usable systems, future users have to be involved represent further related research our Connect framework is continuously to provide feedback and to improve concepts based on. These toolkits offer a system for connecting in- and software artifacts [9]. According to Weiser, one of the teractive physical devices and interaction sensors with digi- essential research and development methods for interactive tal applications and software components. The iStuff toolkit ubiquitous systems is the creation of prototypes [23]. The suite supports multiple users, devices and applications as rapid prototyping technique aims at creating prototypes in a well as interaction modalities. However, due to the limited time and resource efficient way to mature artifacts in agile software models applied within these tools, the set of sup- software engineering processes [15]. The focus of our work ported interaction devices is rather static. A model-based ap- is on providing a framework for the rapid development and proach for dynamically creating multimodal user interfaces evaluation of multimodal interactions for ubiquitous systems. composed of several components is described by Feuerstack Especially within the UCD process, prototypes are needed to and Pizzolato [8] as part of their MINT framework. Mappings evaluate design ideas and improve the usability of interac- between user interface elements, user actions and actions to tive systems [17]. Complex interaction scenarios involving be executed can be formally defined with help of this frame- multimodal interactions require the use of technically mature work and used for dynamically generating interactive appli- prototypes to improve the usability of the system or applica- cations. Both the iStuff and MINT framework are intended tion [14]. to be used in the design and development process of user in- terfaces whereas our focus lies on the prototyping and evalu- The basis for our prototyping framework is the OpenInter- ation of interaction concepts (i. e., event-to-action mappings) face platform developed by Lawson et al. [13]. The plat- in different stages of the UCD process. However, in order to form allows the integration of arbitrary heterogeneous com- prototype and develop interactive applications including in- ponents as interaction devices and applications. OpenInter- teraction concepts and user interfaces, the iStuff and Connect face provides a lightweight framework and runtime environ- framework can be used complementary. ment leveraging the prototyping of multimodal interaction concepts using high-level data fusion and pipeline concepts The ProtUbique framework by Keller et al. facilitates the to connect interaction devices with interactive applications. rapid prototyping of interactive ubiquitous systems [11]. It In contrast to other frameworks for prototyping of interac- supports an extensible set of interaction channels transmitting tions [7, 4], OpenInterface is based on a platform and tech- low-level interaction events from heterogeneous devices and nology independent description of a component’s functional- sensors. These interaction events are unified by the frame- ity and interface. Our framework adapts these concepts with work and accessible programmatically in order to prototype a stronger focus on the underlying models and component interactive applications. As ProtUbique offers interfaces to descriptions in order to facilitate the extension, runtime adap- access its interaction channels, it is possible to directly com- tation and reuse of components and interactions during the bine both the ProtUbique and Connect framework. Interac- iterative stages of UCD. Other existing interaction and proto- tion channels are integrated into Connect in the form of sen- typing frameworks (e. g., CrossWeaver [20] and STARS [16]) sors supplying interaction events. Connect can then be used are realized in a more informal way for specific scenarios and to map these events to actions that will be executed by actua- therefore lack extensibility of software components, interac- tors or applications. tion devices and modalities as well as reusability. The Open- With the emergence of ubiquitous systems, users play a cen- tral role in the software development process. Prototypes are well suited for involving users in the design process and for improving concepts and software artifacts based on user feedback. Various frameworks for the prototyping of inter- active applications including user interfaces and interaction concepts exist. These frameworks are often focused on the use of prototyping techniques in early development stages and limited in the set of supported software components and interaction modalities. However, agile and iterative software engineering processes are required for developing interactive Figure 1. Class diagram of the Connect framework ubiquitous systems. Therefore, we propose a model-driven framework for the rapid prototyping of multimodal interac- tion concepts. By applying models for the definition of inter- interaction devices and arbitrary event sources can be inte- active components and their interrelations, extensibility and grated into the framework. In order to integrate new types reusability of interaction concepts and interactive prototypes and corresponding instances of sensor components into the in multiple design stages is facilitated. In that way, the us- framework, an adapter for receiving the sensors’ events has to ability and user experience of applications and systems for be implemented. On receiving an event from the sensor, the ubiquitous environments can be increased. state of the corresponding event port is updated, i. e., the port is activated or deactivated. An event port can be connected INTERACTION FRAMEWORK to one or more action ports of one or more actuator compo- nents. The event port’s activation leads to the activation of the Structure connected action ports. As arbitrary event sources are sup- We designed the Connect framework from a software engi- ported, interactive devices independent of modality and num- neering point of view using abstract models and their build- ber can be combined to be used for multimodal interactions, ing blocks as a starting point. The framework adheres to a i. e., using one or more input modalities. Currently, only bi- basic class structure consisting of multiple types of compo- nary states for events (active/inactive) without additional data nents, which are interconnected with each other. Fig. 1 shows payload are supported by the Connect framework. the class diagram using UML notation. A Component is a An example of a locally integrated sensor component is the software entity having a defined interface and known behav- computer’s keyboard. The event ports correspond to the set ior [13]. In analogy to control systems linking physical sen- of the individual keys. A smartphone device sending touch sors with actuators, we distinguish between SensorCompo- events via a dedicated app to an instance of the Connect run- nents representing entities that are able to produce interaction time is an example of a remotely integrated sensor compo- events and ActuatorComponents able to consume interaction nent. The set of touch events provided by the smartphone and events and trigger subsequent actions. ComplexComponents supported by the app represent the event ports. combine these capabilities. In addition, specializations of complex components are used to enable the logical and tem- poral combination of sensor and actuator components. Ports Actuator Components describe the components’ interfaces in order to define interac- Actuator components represent devices and applications that tions and connections between multiple components. Event- are able to actively perform and execute actions. Analogous Ports define the types of events a sensor component is able to a sensor component, the ActuatorComponent is an inter- to produce and ActionPorts represent the types of actions an face actuators have to implement in order to connect the ac- actuator component is able to perform. The activation of an tuator to the Connect framework. An ActionPort is a wrapper event port leads to the activation of the action ports the event for an action or method the actuator component is able to ex- port is connected to. A central Manager class handles the in- ecute. New types of actuator components can be integrated stantiation of components and maintains a list of all active into the framework as implementations of the interface Ac- component instances, which are accessible from within the tuatorComponent. Adapters for calling the actuator compo- scope of the framework. nents’ particular operations from inside the framework have to be implemented for every type of actuator. An action port can be connected to one or more event ports of one or more Sensor Components sensor components. Upon receiving an activation from an Sensor components represent devices and applications that event port connected to an action port, the actuator compo- are able to detect interactions and produce corresponding in- nent activates the action port and executes the corresponding teraction events. The SensorComponent is an interface sen- method. Arbitrary local and remote devices and applications sors have to implement, e. g., by an adapter connecting the can be integrated into the framework as actuator components. sensor device via its API to the Connect framework. An Currently, we support the activation of methods without the EventPort is a wrapper for every type of event the sensor com- processing of input or output parameters. ponent is able to trigger. The sensor component maintains a list of all its events and creates corresponding event ports. By An example of an actuator component is a service robot implementing the sensor component interface, new types of whose movement functionality is provided in the form of directed movement actions (e. g., forward, backward, left, right). For each direction there is a corresponding action port. Complex Components Complex components represent devices and applications that combine sensor and actuator functionalities. These entities can contain multiple event and action ports, i. e., they are able to produce events for actuator components and receive events from sensor components. The ComplexComponent class is Figure 2. Extensions of the SensorComponent to support BCI input modes viewed as an abstract class that has to implement both the SensorComponent and the ActuatorComponent interfaces in order to be integrated into the Connect framework. of an actuator’s callable methods and their parameters, which An example of a complex component is a smartphone sending can be parsed in order to automatically create an actuator touch interaction events and providing executable operations component implementing the ActuatorComponent interface (e. g., for taking pictures or switching on the its light). and the corresponding action ports. Logical and Temporal Components PROTOTYPING MULTIMODAL INTERACTION CONCEPTS Logical Components are viewed as specializations of com- Exemplary Sensors plex components. They are used for creating logical connec- Brain Computer Interface tions (AND, OR, NOT, XOR, etc.) between multiple event In order to show the framework’s capability of supporting ports of one or more sensor components. The logical compo- multimodal interactions and its applicability within the Smart nent’s action ports are used as input ports for events from the Home scenario, we extended the core sensor component by sensor components and its event ports are used as output ports the Emotiv EPOC1 EEG brain computer interface (BCI) act- producing events for the activation of subsequent actuator ing as a source of interaction events [19]. The BCI used in our components. By cascading these logical components, com- setting provides interaction modes enabling the recognition of plex logical circuits for sensor events triggering actions of thoughts (Cognitive), facial expressions (Expressive), emo- actuator components can be created. In addition, we integrate tions (Affective) and head movement (Gyroscope). Each of flip-flop and trigger components for saving of and switching these modes is introduced as a subclass of the abstract Brain- between states. That way, it is possible to define more com- ComputerInterface class, which implements the SensorCom- plex interaction concepts involving multiple interaction de- ponent interface (see Fig. 2). Event ports are created for ev- vices in advanced stages of the engineering process and also ery possible type of sensor event produced by the BCI in each to introduce modes of interaction (i. e., state-dependent be- mode (e. g., for blink, wink left, look right, smile, and laugh havior of the interactive prototypes). in the Expressive mode). Upon instantiation of an object of Besides logical components, Temporal Components for de- one of the interaction mode classes, a listener for event ports scribing temporal dependencies between sensor events are corresponding to the sensor events is initialized. supported as extensions of complex components. That way Tablet we are able to define the activation of higher level events, The second exemplary sensor component from the Smart e. g., after a defined number of repeating sensor events or after Home domain that we integrated into our test setting is an the appearance of an event within a defined time frame. The Android-based tablet device. A dedicated app sends inter- functions and algorithms–including additional attributes–that action events regarding the pressing of specific buttons and are executed when the logical or temporal component is acti- events detected by the tablet’s gyroscope sensor to an instance vated have to be provided for each new type of complex com- of the Connect framework. In order to support this event ponent. As new types of components are introduced into the source, we introduce the abstract Tablet class implementing framework’s underlying class model by inheritance, only the the SensorComponent interface. From that class, the Button base classes’ methods have to be overwritten to use instances and Gyroscope modes are derived as subclasses (see Fig. 3). of these new components. Event ports representing the particular buttons and gyroscope Dynamic Components movement directions (i. .e., forward, backward, left, right) en- Thus far we are able to extend the set of sensor, actuator and able the detection of the corresponding interaction events and complex components by introducing implementations and connection to other components. specializations of the appropriate classes into the model at Exemplary Actuators design time. In order to add new types of components at run- time, we extend the framework by the concept of Dynamic Service Robot Components. These components are created by Connect’s A TurtleBot 22 service robot plays the role of an actuator in runtime based on a formal model of a component’s function- the context of our Smart Home scenario [19]. We abstracted ality and ports. Currently, we support the use of a WSDL its movement functionality into two operational modes ex- (Web Services Description Language [5]) document describ- tending the abstract ServiceRobot class: Manual Movement ing the available operations of a service-based actuator com- 1 https://emotiv.com/epoc.php 2 ponent. The WSDL format provides a suitable formalization http://www.turtlebot.com/ Figure 3. Extensions of the SensorComponent to support tablet input Figure 5. Complex network of input and output components forming an modes interactive prototype interactions and listeners for new events. The security com- ponent’s second operation resets all ports to the inactive state and re-enables the event listeners to continue with the inter- action. Both operations can be connected to the event ports of an arbitrary–preferably reliable–sensor component. Prototyping Tool We implemented a Java application based on the Connect framework. The tool allows the graphical instantiation of known types of sensor, actuator and complex components. The lists of available ports, individual attributes as well as the Figure 4. Extension of the ActuatorComponent to support a service robot component’s graphical representation are coded into the class actuator structure and component’s data model. Instantiated compo- nents can be configured using the tool. In addition, it is pos- sible to generate service-based actuator components from a and Automatic Movement (see Fig. 4). The manual movement WSDL file. Connections between component ports are cre- mode supports the fine-grained control of the robot platform ated and modified graphically at runtime using drag and drop by direct movement commands (i. e., forwards, backwards, to gestures (cf. Pipeline metaphor [13]). That way, circuits for the left, to the right). Using the automatic movement mode, interactions consisting of sensors, actuators, logical compo- the robot can be send to specific locations in a room or build- nents and temporal components can bes designed. For certain ing. In automatic mode, driving, path planning and obstacle types of sensor events there are sliders that are used for setting avoidance are handled by the robot itself. The action ports for activation thresholds. As many interaction devices provide these two actuator component modes correspond to the avail- sensor data in the form of numerical values–not just Boolean able movement directions (manual mode) and to the specific values for the active/inactive states–the definition of activa- target locations (automatic mode). tion thresholds increases the accuracy of event detection/acti- Service-based Light Switch vation and supports individual user configurations. The capability of dynamically adding new components at run- Fig. 5 shows a screenshot of the configuration tool’s user in- time is an important feature of the Connect framework. As it terface containing three instances of sensor components (BCI supports the automated generation of an actuator component modes), logical components and an actuator component (ser- based on a WSDL document, we implemented a web service vice robot). The tool’s user interface provides visual feedback for the remote control of a light switch providing a switch on regarding currently active sensor events, connections, and ac- and a switch off operation. Upon parsing of the WSDL file tions as well as numerical values for sensor input. and creation of action ports for both operations, the Connect runtime acts as a client sending requests to the web service. Prototype Configurations and User Profiles Security Component The composition of components as well as their intercon- In order to prevent incorrect behavior and actions caused nections, attributes and port thresholds can be persisted in by imprecise interaction devices and unintended user inter- individual prototype configurations and user profiles based actions at runtime, a Security Component is introduced into on the class model presented in the previous section. These the framework as an implementation of the actuator compo- settings are saved in and loaded from XML-based files. In nent. This component provides an operation for deactivating this way, individual interaction concepts can be created for all event and action ports and thereby disabling the current specific prototypes, component configurations, scenarios and users according to their capabilities. These model-based con- By creating user profiles for specific users, user groups and figurations can then be used as templates for creating new scenarios, the corresponding prototypes can be used for user- interaction concepts or for refinement at a later stage of the centered evaluation methods (e. g., user studies and expert development process [18]. Listing 1 shows an extract of a evaluation) during various stages of the software development prototype configuration describing an event port of a sensor process. In addition, user profiles facilitate the creation of in- component connected to an action port. teractive applications according to a user’s individual cogni- tive capabilities and preferences. Listing 1. Extract from a prototype configuration The extensibility of the framework’s underlying models for components and interactions allows for the integration of new interactive devices and applications at later engineer- IO . T a b l e t B u t t o n C o m p o n e n t T a b l e t B u t t o n C o m p o n e n t ing stages. That way, interaction concepts can be developed

starting from simple event-to-action mappings and evolved

to more complex models and scenarios for interactive ubiq- Core . E v e n t P o r t uitous systems. Changes within these models can be propa- Up f a l s e gated to the affected software artifacts. By adding a mech- anism for model versioning, traceability for the evolution of these interaction models in iterative stages of the design pro- T u r t l e B o t cess is achieved and templates of interaction concepts can be

MoveForward created for reuse in new projects. DISCUSSION The introduced Connect framework for the creation and iter- ative advancement of interaction concepts is built on model- ... based components and configurations. Our chosen design al- lows for convenient extension through the inclusion of addi- tional arbitrary interaction components (i. e., sensors and ac- Prototyping and Evaluation tuators) via concepts of object-orientation. Basic design pat- With the help of the Connect framework and the configuration terns reduce the effort for developers to extend the set avail- tool, interaction concepts describing mappings between inter- able interaction devices and applications. At runtime, interac- action devices and active controllable components can be cre- tion components can be rapidly combined into pipelines and ated. Once integrated into the framework, multiple instances circuits to form complex interaction concepts. Modifications of multimodal sensor, actuator and complex components are of interaction concepts as well as persisting and loading of ready to be used and loosely-coupled at runtime for a partic- interaction and component configurations are supported by ular setting. Connections between sensors and actuators and the framework. Compared to related research, this approach their respective ports are modifiable at runtime (create, up- allows for a high level of reusability and refinement of inter- date, delete). Compared to hard-wired event-to-action map- active prototype applications and configuration in succeeding pings, model-based prototypes of interaction concepts can be development stages. With the help of the graphical configura- created and modified quickly with the help of Connect in or- tion tool it is possible to rapidly create working prototypes for der to test and evaluate their usability and suitability for con- multimodal interactive applications and use these prototypes crete use cases as part of the software engineering process. for test and evaluation purposes. Connect can be part of various user-centered prototyping Due to the use of known metaphors, e. g., pipelines and cir- and evaluation methods and stages reusing models that de- cuits, the creation and configuration of complex interaction scribe components and their interrelations. As the prototyp- concepts can be achieved even by non-programmers such as ing tool follows known metaphors from the WIMP paradigm interaction designers or end users, supporting different phases and integrates easy to understand graphical metaphors of iterative software engineering processes like UCD. By pro- (e. g., pipelines and circuits), it is also possible for designers, viding model-based abstractions for component design and non-programmers and end-users to understand and define in- data flow, interaction design may be advanced from sim- teraction concepts for a given scenario. That way, future users ple prototypes during requirement analysis to complex mul- can be involved in early stages of the design process lead- timodal interaction concepts containing numerous different ing to more intuitive interactions and usable applications for sensors and actuators in subsequent phases of development. ubiquitous systems. This iterative improvement of existing concepts proved to be especially helpful during usability testing using expert evalu- Due to the model-driven approach for describing components ation and end user studies. and their interrelations applied in Connect, it is possible to persist and reload component configurations and their con- On the one hand, the introduction of logical and tempo- nections. Prototypes of interaction concepts can be repeat- ral components into more complex interaction concepts al- edly tested, reused and refined in order to increase the usabil- lows for the use of multiple interactive input devices. On ity of the interactive application that is under development. the other hand, these components enable interaction modes and the definition of interaction sequences to prevent unin- nents. The framework follows a model-driven software en- tended interaction events. These mechanisms become neces- gineering approach enabling the extension and integration of sary when using imprecise and “always on” interaction de- new types of components into interactive prototypes as well vices (e. g., brain-computer interfaces and eye trackers) to as the reuse of component and prototype configurations dur- prevent Midas touch. With respect to the BCI, a double-blink ing development. Related frameworks and concepts generally within a certain timeframe could, for example, be used to trig- lack extensibility, flexibility and reusability due to the limited ger an action instead of a simple “natural” blink. As shown use of models and other formalisms. Therefore, interaction in the prototyping tool, thresholds defining the activation of frameworks often support only a static set of interaction de- sensor ports can be defined for sensor events containing Inte- vices and applications that can be used for the development ger or Double values. This mechanism also helps interaction of short-living interaction prototypes at early design stages. designers with the integration of imprecise devices. Lastly, a With Connect, the set of software components can be easily security component can be added to the interaction concept extended to support new types of devices and applications, and test environment to stop all ongoing interactions in case which can be combined to create multimodal interactions. of any malfunctions. This is especially helpful when interact- Connect’s runtime and user-friendly prototyping tool facili- ing with real world physical devices in ubiquitous systems. tate the use of multiple input and output devices as entities involved in interactions as well as dynamic reconfiguration The Connect framework is a prototyping tool for engineer- of interactions at runtime. The model-based descriptions of ing interactive applications. It can be employed during the components and interactions leverage the reuse and iterative development process for designing and testing of interaction refinement of components, concepts and prototypes for the concepts. In combination with other frameworks for the pro- user-centered software engineering process of ubiquitous sys- totyping of user interfaces (e. g., MINT [8], iStuff [2]), in- tems. teractions (e. g., OpenInterface [13]) and interaction devices (e. g., ProtUbique [11]) as well as for the fusion of high-level Regarding future work, we will extend the component models sensor events (e. g., Mudra [10]), Connect represents an ad- in order to be able to process non-Boolean input and output dition to the engineering toolchain for interactive ubiquitous values and states for event and action ports. The use of a systems. Due to its extensibility and model-driven develop- formal definition language for describing the interfaces of a ment approach, interaction and configuration models created sensor component will add the ability to also introduce new with Connect could be used as input for subsequent tools and types of sensor components to Connect at runtime. By com- phases in the software engineering process. bining the prototyping framework ProtUbique [11] for defin- ing interaction sources and high-level interaction events with The current development state of the Connect framework still service-based communication for sending interaction events contains some shortcomings that will be improved in con- to Connect, we will create a flexible toolchain for developing secutive versions. Until now, user defined data types beyond prototypes of interactive multimodal applications. In addi- simple Boolean values at event and action ports are not sup- tion, dynamic component platforms (e. g., OSGi [1]) can be ported. To be able to accommodate analogue sensors, at least employed to introduce additional runtime flexibility concern- some form of floating point data and complex data types will ing the support of new types of software components. We will have to be included into the extensible data model. In ad- also look into the distributed communication among several dition, aggregation of components attached to distributed in- instances of the Connect runtime. An instance of Connect stances of the framework is not yet possible. With the avail- running on one computer could be used as a sensor com- ability of active components that contain significant process- ponent within another Connect instance, which allows the ing power themselves such as smartphones, integration of preprocessing and derivation of higher order events on local preprocessed sensor values can be simplified by combining computers in order to save resources and simplify the mod- multiple networked Connect systems. eling of complex interactions. To evaluate the framework’s applicability, we will use Connect for the prototyping of in- CONCLUSION & FUTURE WORK teractive software components as part of the engineering pro- Engineering software for interactive ubiquitous systems re- cess for Smart Home applications [19, 12]. quires flexible and iterative development processes. The con- tinuous involvement of future users throughout the entire REFERENCES process is a key aspect of the user-centered design and de- 1. OSGi Alliance. 2003. Osgi service platform, release 3. velopment methodology for ubiquitous systems. Prototypes IOS Press, Inc. are a well suited tool for the development, test and evalua- tion of theoretical concepts in almost all stages of the soft- 2. Rafael Ballagas, Faraz Memon, Rene Reiners, and Jan ware engineering process. We developed an extensible and Borchers. 2007. iStuff mobile: rapidly prototyping new easy to use framework that supports rapid prototyping, evo- mobile phone interfaces for ubiquitous computing. In lution and evaluation of interaction concepts for ubiquitous Proceedings of the SIGCHI conference on Human systems. The Connect framework is based on a modular factors in computing systems. ACM, 1107–1116. object-oriented software model, which views interaction de- vices as sensor components and interactive applications as 3. Rafael Ballagas, Meredith Ringel, Maureen Stone, and actuator components. Interaction concepts can be defined, Jan Borchers. 2003. iStuff: a physical user interface modified and tested at runtime by connecting these compo- toolkit for ubiquitous computing environments. In Proceedings of the SIGCHI conference on Human 16. Carsten Magerkurth, Richard Stenzel, Norbert Streitz, factors in computing systems. ACM, 537–544. and Erich Neuhold. 2003. A multimodal interaction framework for pervasive game applications. In 4. Jullien Bouchet and Laurence Nigay. 2004. ICARE: a Workshop at Artificial Intelligence in Mobile System component-based approach for the design and (AIMS), Fraunhofer IPSI. development of multimodal interfaces. In CHI’04 extended abstracts on Human factors in computing 17. Martin Maguire. 2001. Methods to support systems. ACM, 1325–1328. human-centred design. International journal of 5. Erik Christensen, Francisco Curbera, Greg Meredith, human-computer studies 55, 4 (2001), 587–634. Sanjiva Weerawarana, and others. 2001. Web services 18. Nicolai Marquardt, Robert Diaz-Marino, Sebastian description language (WSDL) 1.1. (2001). Boring, and Saul Greenberg. 2011. The proximity 6. Anind K Dey, Gregory D Abowd, and Daniel Salber. toolkit: prototyping proxemic interactions in ubiquitous 2001. A conceptual framework and a toolkit for computing ecologies. In Proceedings of the 24th annual supporting the rapid prototyping of context-aware ACM symposium on User interface software and applications. Human-computer interaction 16, 2 (2001), technology. ACM, 315–326. 97–166. 19. Ronny Seiger, Tobias Nicolai, and Thomas Schlegel. 7. Pierre Dragicevic and Jean-Daniel Fekete. 2001. Input 2014. A Framework for Controlling Robots via device selection and interaction configuration with Brain-Computer Interfaces. In Mensch & Computer ICON. In People and Computers XVInteraction without 2014–Workshopband: 14. Fachübergreifende Konferenz Frontiers. Springer, 543–558. für Interaktive und Kooperative Medien–Interaktiv unterwegs-Freiräume gestalten. Walter de Gruyter 8. Sebastian Feuerstack and Ednaldo Pizzolato. 2011. GmbH & Co KG, 3. Building multimodal interfaces out of executable, model-based interactors and mappings. In 20. Anoop K Sinha and James A Landay. 2003. Capturing Human-Computer Interaction. Design and Development user tests in a multimodal, multidevice informal Approaches. Springer, 221–228. prototyping tool. In Proceedings of the 5th international conference on Multimodal interfaces. ACM, 117–124. 9. John D Gould. 2000. How to design usable systems. Readings in Human Computer Interaction: Towards the 21. Wolfgang Wahlster. 2006. SmartKom: foundations of Year (2000), 93–121. multimodal dialogue systems. Vol. 12. Springer. 10. Lode Hoste, Bruno Dumas, and Beat Signer. 2011. 22. Mark Weiser. 1991. The computer for the 21st century. Mudra: a unified multimodal interaction framework. In Scientific american 265, 3 (1991), 94–104. Proceedings of the 13th international conference on 23. Mark Weiser. 1993. Some computer science issues in multimodal interfaces. ACM, 97–104. ubiquitous computing. Commun. ACM 36, 7 (1993), 11. Christine Keller, Romina Kühn, Anton Engelbrecht, 75–84. Mandy Korzetz, and Thomas Schlegel. 2013. A Prototyping and Evaluation Framework for Interactive Ubiquitous Systems. In Distributed, Ambient, and Pervasive Interactions. Springer, 215–224. 12. Suzanne Kieffer, J-YL Lawson, and Benoit Macq. 2009. User-centered design and fast prototyping of an ambient assisted living system for elderly people. In Information Technology: New Generations, 2009. ITNG’09. Sixth International Conference on. IEEE, 1220–1225. 13. Jean-Yves Lionel Lawson, Ahmad-Amr Al-Akkad, Jean Vanderdonckt, and Benoit Macq. 2009. An open source workbench for prototyping multimodal interactions based on off-the-shelf heterogeneous components. In Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems. 245–254. 14. Linchuan Liu and Peter Khooshabeh. 2003. Paper or interactive?: a study of prototyping techniques for ubiquitous computing environments. In CHI’03 extended abstracts on Human factors in computing systems. ACM, 1030–1031. 15. Luqi. 1989. Software evolution through rapid prototyping. Computer 22, 5 (1989), 13–25.