Smart environments bring together multiple users, (interaction) resources and services. This creates complex and unpredictable interactive computing environments that are hard to understand. Users thus have difficulties to build up their mental model of such interactive systems. To address this issue users need possibilities to evaluate the state of these systems and to adapt them according to their needs. In this work we describe the requirements and functionalities for evaluating and controlling interactive spaces in smart environments from the system and the user perspective. Furthermore we present a model-based implementation of these capabilities which is accessible for the user in form of a meta user interface.
smart
The ongoing realization of the ubiquitous computing paradigm and the creation of environments holding multiple networked (interaction) resources lead to new forms of human-computer interaction. While current systems support multiple applications through multi-tasking and multiple users one after the other or via web-based applications, their interfaces are usually build for one user using one service with one limited and fixed set of interaction resources. Future interaction in smart environments however Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Conference’04, Month 1–2, 2004, City, State, Country.
Copyright 2004 ACM 1-58113-000-0/00/0004…$5.00. brings together multiple users, multiple interaction resources and multiple services (applications). This raises the need to manage and control the assignment of resources, users and services and leads to the complex problem of considering the multiplicity in three dimensions (Figure 1). Considering multiple services simultaneously (1) e.g. requires the distribution of screen space among them, the shared usage of interaction resources like microphones or loudspeakers as well as the exchange of semantics and information between the services to reach a useful level of interconnection. Multiple simultaneous users (2) require e.g. the shared or alternating usage of interaction resources, the resolution of conflicts, the collaborative usage of resources and services, the possibility to exchange information between multiple users as well as the consideration of privacy issues. Finally the multiple available interaction resources (3) drive new forms of interaction, but this also requires e.g. the possibility to directly select and address resources according to the needs of users and services, the management of resources (occupied resources), the distribution of information across multiple resources or the adaptation to the resource properties. As different resources can also support different modalities this involves the utilization of multimodal interaction capabilities. In this paper we mainly focus on the latter aspect (3), being a facilitator for the former two. Without the possibility to manage the utilization of interaction resources, it is very unlikely that multi-user and multi-application scenarios can benefit from the availability of multiple interaction resources.
In the remainder of the paper, we first introduce the user perspective by explaining the functionalities users need to manage the utilization of interaction resources. Thereby they can determine which services or parts of the services are presented on or controlled through which interaction resources. Following this we elaborate on the system perspective by explaining how the system manages services and interaction resources and provides the functionalities to establish connections between both entities. In section 4 we introduce our implementation. Based on a runtime system using a user interface model with multiple levels of abstraction to describe such multimodal, distributed user interfaces for smart environments, we present how the user can manage the utilization of interaction resources. A comparison to the related work and a summary and outlook complete the paper.
The utilization of multiple interaction resources (IRs) at the same
time poses new demands on users. The user needs the possibility
to keep track of the user interfaces from the different services
(service UIs) spread across different IRs and should be provided
functionalities to alter the configuration according to her needs.
We refer to this as the configuration of the (personal) interactive
space of the user. An ambient interactive space has been defined
as a dynamic assembly of physical entities coupled with
computational and communicational entities to support human
activities [
From the user perspective the utilization of one or several services currently available in a smart environment, thus requires the configuration of her interactive space to determine the IRs she wants to utilize the services. Two possibilities can be addressed to configure the personal interactive space: (1) the configuration of a single IR and (2) the configuration of a set of multiple IRs. In the first case, the user uses a given IR to control the utilization of this very interaction resource. This means the IR provides access to a meta-level of the user interface, allowing the alteration of its presentation. For a specific IR this includes adding or removing parts of the service UI to/from the IR. In the second case, the user again uses an IR to access a meta-level of the user interface. In this case however, the configuration via the IR also affects other IRs. The user can move or clone part of the service UI between IRs or add and remove elements to an IR, different from the one currently used. This second configuration requires access to the complete environment and the available services and provides a freely configurable interactive space.
To make these functionalities available for the user, a
configuration interface is required that has to be provided
independently from the services. Meta user interfaces (meta-UIs)
have been proposed to provide such common facilities for user
interfaces and thus a generic control on a meta-level [
To address this issue we utilize a runtime system providing distributed multimodal user interfaces. As described in the next section, this runtime system is aware of the context-of-use and manages the service UIs. It also controls the connections between service UI parts and interaction resources. The meta-UI is provided as the control interface to configure the runtime system.
From the perspective of the system, the configuration of a users’
interactive space, involves the management of the service UIs and
their current states as well as the available IRs. To support the
distribution of (multimodal) user interfaces, the underlying
runtime system is also responsible for the assignment of IRs to
service user interfaces, so that the user can interact with the
service. In our work, we assume a server side system that is aware
of the available services (in form of a UI model for each service)
and of the available IRs (represented in a context model). Figure 2
shows the elements of this runtime system. The interaction with a
service is defined by a UI model that is combined of different
levels of abstraction. Similar to the CAMELEON Reference
Framework [
The runtime system also continuously senses the environment for
new IRs and manages them in a context model. The model
comprises information about users, environment and IRs, where
representations of the IRs define the available resources internally
for the system. The runtime system uses the IR representations to
push CUI elements to these resources as we described in [
In the simplest case, this leads to UI elements connected to a
single IR, e.g. the presentation of a user interface on a screen. The
ability of the system to maintain and alter this connection and the
possibility to push the UI elements to any connected IR now also
allows changing the target IR, leading to the migration of the UI
e.g. to another screen. Redirecting the elements to an IR of
another modality could e.g. also lead to starting a voice dialog.
However, to realize multimodal interaction we aim at the
simultaneous utilization of multiple interaction resources. This in
turn requires the distribution of the available UI elements to
multiple IRs simultaneously (see also [
The different configuration scenarios described above can
technically be brought down to the atomic operations of creating a
connection between a CUI element and an IR or removing such
connections. For example changing the interaction modality of a
task from graphical to vocal includes the removal of connections
between IRs and graphical CUI elements of that task and the
creation of new connection between the voice CUI elements and
the appropriate IR (or IRs). When a "CUI to IR" connection is
established our runtime system sends the element to the IR, which
then creates the final user interface and delivers it to the user. If a
CUI element should no longer be accessible through an IR, the
appropriate connection between both is destroyed, which results
in the removal of the corresponding FUI from the IR. It must be
said, that the association between the CUI elements and the
elements at higher levels of abstraction (task and AUI) are always
preserved. This is necessary for the state synchronization of all
elements as described in [
In the next section, we describe our implementation of the described system. The Multi-Access Service Platform (MASP), a modal based runtime system, provides the basis to provide users a meta-UI allowing them to control multimodal interaction.
To evaluate the described approach for the control of multimodal interaction we have implemented a first version of a meta-UI with the MASP, our implementation of the UI runtime system described above. Providing a model-based framework for the development and execution of multimodal multi-device user interfaces, the MASP provides the means to develop interactive services for smart environments. Combined with the capability of the MASP to automatically discover interaction resources in the environment the prerequisites are fulfilled to implement a meta-UI service allowing the user to evaluate and control multimodal interaction.
Figure 3 shows a screenshot of the current implementation of the
meta-UI. In the upper, left corner the user can requests the
currently available services. In the upcoming list the user can
choose which one she wants to connect to the currently used
screen. Once the user selects a service the UI of the selected
service is shown in the centre and the configuration options at the
bottom of the screenshot can be used to configure the current
service UI. Here we distinguish four features the user can utilize
to configure her interactive space. (1) The migration feature
provides possibilities to migrate a service UI from one interaction
resource to another to e.g. transfer the UI to another screen better
viewable from the users’ current position. Through the
distribution feature (2) the user can distribute parts of the user
interface to other IRs. Thereby the user can also specify if the
selected parts should be cloned or moved to the target IR. The
third configuration feature is called multimodality (3) and
provides possibilities to configure the utilized modalities within
the interaction. This allows users to e.g. switch off audio output of
the MASP if it is currently disturbing the user. The adaptation
feature (4) allows the user to configure further functions of the
MASP. E.g. the MASP supports a so called “FollowMe” modus
which can be configured through the adaptation feature. The
activation of the “FollowMe” modus leads to an automatic
configuration of the interactive space by the MASP over time.
The MASP senses for changes in the available interaction
resources for the user (resources made available or are no more
available to the user) and reconfigure the interactive space
according to the new resource combination by trying to support a
broad range of interaction possibilities.
These configuration options allow adapting the interactive space
according to the possible changes defined e.g. by Coutaz [
Moreover the status symbols in the upper centre of the screen allow the user to observe which modalities are currently enabled. In the bottom right corner the user can “release” her interactive space, which results in removing all service UIs from all interaction resources.
Several approaches exists which enable the configuration of the
relationship between services and IRs by some means or other.
Most of them also provide some kind of meta-UI to allow the user
to access the configuration possibilities Molina et al. [
In the mentioned approaches the meta-UIs are not the focus but
are developed to give access to exactly the specific described
configuration possibilities. They do not consider other
functionalities which could improve the possibilities of users to
simplify the control of their interactive space. The work described
by Vanderhulst [
We presented our approach to control multimodal interaction in smart environments. The functionalities to keep the user in control of the interaction by configuring her interactive space as well as the prerequisites from the system perspective were described. Furthermore a first implementation of the described concept allowing the user to access these capabilities through a meta-UI was presented. However, there are still some aspects which deserve further investigation.
At the moment the user has to configure the interactive space based on the provided information by herself. But the system with its knowledge about the available IRs and services as well as the user and further environment information can at least help the user by providing useful configuration possibilities. Furthermore the system can automatically configure the interactive space of the user (as we started to implement with the “FollowMe”-feature). However, the automatic configuration can also reduce the satisfaction of the user if it does not exactly match her preferences and requirements and should therefore be used careful. Another aspect that arises with the configurability of the interactive space is the persistence of the user configuration. When the user (re)configures its interactive space, she adapts it to her preferences and needs in the current situation. It thus appears to be suitable that the system utilizes this knowledge by providing the same configuration to the user in the same situation, so the user does not need to do the same configuration over and over again. However, the automatic analyzing of the current situation and the detection of the relevant context parameters is a difficult task which needs further investigation. A first solution could be to let the user specify the relevant situation parts if she wants the system to be able to restore a given interactive space. Another direction for future work are the problems occurring when considering multiple users and multiple services.