Many different models and tools exist for supporting the design of ubiquitous interactive systems. Each of them deals with a different point of view. As a result designing such systems has to involve a set of models rather than just one. In this paper we first provide an overview on existing models dedicated to Mixed Interactive Systems, one form of ubiquitous systems. Then, to facilitate the elicitation of the most appropriate model, we organize them along the steps of the development process. Finally, to smoothly guide the use of these different design resources along the development process, we provide an overview of different linking mechanisms between design models for ubiquitous systems and highlight their characteristics.
INTRODUCTION
Among the most recent forms of interactive techniques,
one aims at taking advantage of physical objects to support
the interaction with a computer system: physical artifacts
surrounding users during their activity become part of the
loop. Users’ everyday objects thus constitute an extension
of their body to communicate with the system. Such
systems are either called tangible UI, mixed or augmented
reality, etc.: we hereafter refer to them with the generic
term of Mixed Interactive System (MIS). Such systems are
emerging in many different domains, ranging from specific
application such as surgery [
EXISTING MODELS IN MIS Designing Mixed Interactive Systems (MIS) requires considering many specifics facets: the nature of physical artifacts, the links between these physical objects and digital data and the variety of devices and technologies which can be involved. Consequently, adapted design resources have been developed. Hereafter, we review a set of design resources dedicated to MIS.
First, conceptual frameworks [
Taxonomies and models [
All these modeling approaches cover different but complementary design considerations. Although their levels of abstraction vary, they offer a clear definition of the development space and constitute a common terminology supporting interdisciplinary communication. As depicted in Figure 1, we also highlight that the different design models and resources, used to develop a MIS, can be organized along the traditional phases of an interactive software design cycle. Existing models dedicated to WIMP can therefore easily be put in parallel with models dedicated to MIS, and either be compared to or used in addition to these dedicated models.
The main limits are however that these models and approaches are almost exclusively usable by MIS experts and remain highly compartmentalized. Indeed, even if high-level resources used during the design should guide the implementation, concrete and systematic links have not been clearly expressed yet. To support the design through the four development phases of a MIS and to highlight a chain of design models and tools from the earliest design considerations to the latest in the development process, connections among models are required. We propose in the following section, an overview of different linking mechanisms we have been exploring over the past years. We depict the goal, source and target of the links between two models and describe the resulting overall mechanism (see Figure 7). The presented linking mechanisms are also positioned in Figure 1.
BASIS OF OUR INSTRUMENTED DESIGN PROCESS The three pillars of our articulating efforts respectively support the abstract description of the user’s interaction with a MIS, the software level decomposition required to implement the designed MIS and the concrete component based implementation of the final MIS. We first summarize these models and illustrate them on a case study: the notepad assisted slideshow. For oral presentations, sequential slideshow systems like PowerPoint are largely used. The prototype we propose is a physical enhancement of a slideshow system: it involves the use of a notepad as “remote control” and feedback source, and associates each page of the notepad to one digital slide (see Figure 2). The speaker can thus write his own comments on the notepad and easily access to the corresponding slide. Potential animation steps of each slide are controlled through user’s tap on the notepad. In the next sections, this prototype is used to illustrate the different models used to develop this system and how they have been linked.
Interaction model
Overview
ASUR [
Goal The goal of this model is to describe the different types of elements and data exchange required to support the interaction with a MIS. Both entities and channels are further characterized by attributes such as the type of entity (real object, adapter bridging the two worlds, etc.), the medium and language of the channels. Additional elements are expressible such as physical constraints among entities, links between a physical and digital entity, etc.
Example
The model of the notepad assisted slideshow is presented in
Figure 3. The “user” interacts with the “notepad” for which
each page is detected by an adapter (“PageDetector”).
Another interaction with a second adapter detects user’s tap
(“StepDetection”). These two adapters deliver to the
“slideshow” some digital data from which the current slide
and step in the animation are identified. Finally, the state of
the slideshow is rendered through three different adapters
to the “user” and the “attendees”.
the “slideshow” expressed in ASUR. This sub-assembly
involves four components: two controllers, one model and
one view.
Tools
A metamodel of ASUR has been defined [
Goal The goal of this second model is to promote the integration of design considerations related to component-based specificities (port, data flow, component) and to software architecture of interactive system (functional core, views and controller). ASUR-IL is thus composed of two types of sub-assemblies: • •
Adapters describing the required devices and API to implement the link between physical and digital world
Entities describing system-dependent components; entities are decomposed according to the MVC pattern. It thus contains a Model, View(s) and Controller(s) In comparison to ASUR, ASUR-IL adopts a software point of view on the interaction with the MIS. It therefore provides a list and description of the software bricks required, interfaces, ports, data types, etc.
Example
Left side of Figure 4 illustrates the adapter sub-assembly
required to implement the “page detector” expressed with
ASUR (Figure 3). It involves a digital camera component
and a marker detection component. The right side of Figure
4 describes the entity sub-assembly required to implement
The ASUR-IL model which entirely covers the case study
is represented in Figure 5.
Tools
As for ASUR, a metamodel has been defined, and a
graphical editor has been defined as an Eclipse plug-in.
Software component model
Overview
To implement the prototypes that we design with ASUR
and ASUR-IL we rely on two existing prototyping
platforms: Open Interface (OI) [
Example The Notepad Assisted Slideshow has been implemented as an assembly of WComp components. This is illustrated in Figure 6. Each element of the assembly corresponds to one of the ASUR-IL element defined in the ASUR-IL model of the system. Tools Metamodels of WComp and OI include the three concepts of component based architecture: components, ports and data flows. Each platform also offers a graphical editing environment for creating the appropriate assemblies. Outcomes and limitations There are obviously links between concepts expressed in each of these three pillars. But there is no clear constraint that drives their respective use and it is not ensured that the designer will conform to design recommendations made with the other models. For example, so far interaction design decisions expressed with the ASUR model are not constraining the development of the software architecture with ASUR-IL. And yet, these three pillars are required to drive a MIS from its early specification to its final implementation. We therefore complement them with linking mechanisms, such as model transformations or model coupling, in order to support the transitions between several phases of the development process. The goal is to better take advantage of the design choices expressed with the different models.
ARTICULATING DESIGN MODELS MIS models have been developed to cover different phases of the development process: transformations and couplings are thus required at different places in the process. From requirement to interaction design Overview We explored different linking mechanisms at this level: • •
Linking models resulting of a KMAD task analysis to
the ASUR interaction model [
Stimulating creativity session with the ASUR model
[
From interaction design to software architecture
Overview
This transformation, represented in the left side of Figure 7,
converts an abstract specification of the interaction
technique into a structured set of required software
components: the generation of this software components
structure is driven by the type of ASUR entity, attributes
and channels involved in the model [
Tool To support this transformation, ATL rules, repositories of components, ontology and an interactive wizard have been developed. All these technologies have been packed into Eclipse plug-ins.
From development models to evaluation
Overview
We developed this linking mechanism in order to relate
ergonomic recommendations to part of an interaction
model describing the MIS [
Goal The main objective of this link between evaluation and design model is to facilitate the identification on the model, of part of the solution that is affecting (positively or negatively) the usability of the system. Such links thus potentially reduce the duration of one cycle of the four phases development process.
Tool So far the navigation through the recommendations is only supported by a multiple criteria query on a web site. Refined tools would be useful so that usability recommendations pop out as soon as one relevant elements of the interaction model is added or selected.
CONCLUSION In this position paper we highlighted the diversity of design-time and run-time models existing in the field of mixed interactive system, one form of ubiquitous interactive system. This diversity is partly explained by the amount of design considerations to handle when it comes to designing such systems: indeed most models covers only one specific aspect or at the best a limited subset of relevant considerations. However, following our analysis of existing works, we have been able to identify for each of these development resources one of the different phases of a development cycle for which the development resource is dedicated. This is thus classifying these design resources. To go past the comparison of models through a classification, it is required to chain one model to another. Indeed one model provides one view on the system to design; chaining one to another provides a support for considering different complementary views without leaving one aspect aside. Furthermore, one unique and integrated design platform would be hard to propose because of the multiplicity of options, situations, technologies and usages potentially involved in a MIS. Chaining models to each other allows the definition of different ways in the design process: for example, going from A to B through a transformation in model C (A Æ C Æ B), may very flexibly be replaced by a longer transformation chain involving two other models instead of model C (A Æ D ÆE Æ– B). The result is the same, but the specialists of models D and E are no longer enforced to use model C. Based on the different linking mechanisms between models of different phases of the development process that we have investigated, this paper showed that different forms of transformation exist: they use repositories of partial solutions, graphical representations, manual application of rules, methodological principles or transformation language. Among them, those exploiting Model Driven Engineering (MDE) approach and tools (ASUR to ASURIL to WComp/OI) appear to be the most promising: they use a standard language; they are easily supported by tools; they contribute to the definition and diffusion of the metamodels; they support the generation of multiple representations of the same model; they define transformation mechanisms, constitute guides through the design process; finally MDE has already proven its efficiency in classical software engineering. However, using MDE raised new challenges to investigate. First in terms of properties, what happens to system or interaction properties settled in one model when a transformation is applied to the model? And more generally, are there properties of a transformation that are particularly important for “modiquitous” activities? Managing retroactive loops in the design process also raises questions: if a model B is generated from a model A, how to ensure that modifications on B are still in line with A? How to send back to A modifications performed on B? Given that ubiquitous systems are still quickly evolving and adding considerations to new dimensions, technologies, artifacts, etc., how MDE might help integrates these emerging new considerations? What would be the relevant characteristic of an ubiquitous interactive situation that could help identify the most relevant design path among the available models and transformations? Finally, evaluating ubiquitous systems is a challenge in itself, but “modiquitous” activities are definitely contributing to this challenge through the elicitation of the most relevant design aspect of ubiquitous system, thus emphasizing the need to base the design of ubiquitous system on models.