=Paper=
{{Paper
|id=Vol-198/paper-9
|storemode=property
|title=Constraint Adaptability of Multi-Device User Interfaces
|pdfUrl=https://ceur-ws.org/Vol-198/paper8.pdf
|volume=Vol-198
}}
==Constraint Adaptability of Multi-Device User Interfaces==
https://ceur-ws.org/Vol-198/paper8.pdf
Proceedings of CHI 2006 Workshop “The Many Faces of Consistency in Cross-Platform Design”
8 Constraint Adaptability of Multi-Device User
Interfaces
Kris Luyten, Hasselt University _ Transnationale Universiteit Limburg Expertise
Centre for Digital Media _ IBBT, Wetenschapspark, 2, 3590 Diepenbeek, Belgium,
kris.luyten@uhasselt.be
Jo Vermeulen, Hasselt University _ Transnationale Universiteit Limburg Expertise
Centre for Digital Media _ IBBT, Wetenschapspark, 2, 3590 Diepenbeek, Belgium,
jo.vermeulen@uhasselt.be
Karin Coninx, Hasselt University _ Transnationale Universiteit Limburg Expertise
Centre for Digital Media _ IBBT, Wetenschapspark, 2, 3590 Diepenbeek, Belgium,
karin.coninx@uhasselt.be
Methods to support the creation of multi-device user interfaces typically use some type of
abstraction of the user interface design. To retrieve the final user interface from the
abstraction a transformation will be applied that specializes the abstraction for a particular
target platform. The User Interface Markup Language (UIML) offers a way to create multi-
device user interface descriptions while maintaining the consistency of certain aspects of a
user interface across platforms. We extended the UIML language with support for layout
constraints. Designers can create layout templates based on constraints that limit the ways
a user interface can rearrange across platforms. This results in a higher degree of
consistency and reusability of interface designs.
THE USER INTERFACE MARKUP LANGUAGE (UIML)
The UIML specification [2] is a high-level canonical markup language to describe the structure, style,
content and behavior of a user interface. The declarative nature of UIML allows a clear separation of
the user interface, its content, the mapping of its abstract concepts onto concrete widgets and the
application logic. UIML’s separation of concerns enables reuse of the user interface and promotes
consistency across different platforms.
UIML does not contain metaphor-specific tags (e.g. ), but only generic tags (e.g. ,
). A set of abstractions can be defined in a vocabulary and allows the designer to
specify a user interface by referring to these abstractions. The vocabulary defines how abstractions
can be translated into a concrete presentation. This is a typical example of an interface being
specified in terms of abstract interaction objects which will be mapped onto concrete interaction
objects afterward [7]. Nevertheless, it is challenging to choose the set of abstractions defined in a
UIML vocabulary so a wide range of different platforms can be supported. Since these are specified
separately, UIML is extendable to new devices and UI metaphors, when they become available.
Several levels of abstraction can be supported by UIML.
An UIML document exists of several parts [1] that are shown in figure 1. Together they make up the
Meta-Interface Model (MIM).
© 2006 for the individual papers by the papers' authors. Copying permitted for private and
scientific purposes. Re-publication of material on this page requires permission by the
copyright owners.
Page 40 of 81
Proceedings of CHI 2006 Workshop “The Many Faces of Consistency in Cross-Platform Design”
Figure 1. The UIML Meta-Interface Model
Interface describes four parts of the user interface: structure, style, content and behavior. The
structure describes UIML does not contain metaphor-specific tags (e.g. ), the “hierarchy” of
the user interface. It defines the different parts that are contained in the user interface. Style
describes properties of the parts defined in the structure. This allows changing properties of the
interactors like text, color, font, etc. The content component separates the content of the interface
(e.g. the list of items that has to appear in a list presentation) of the other parts. The behavior of a
UIML document defines rules with actions that are triggered when some condition is met. Some kind
of event mechanism is offered to the user interface designer this way.
Peers define mappings to entities external to the UIML document, and are divided into the
presentation and logic. Presentation contains the mapping with the concrete user interface toolkit. It
defines a “vocabulary” to be used with a UIML document. Finally, the logic component defines how to
bind the user interface with the application logic.
Listing 1 shows the structure and style components of an example UIML document. For more details
about the UIML language, we refer to the specification [1].
Listing 1. Structure and style section for a simple dictionary user interface
We have developed a multi-device UIML renderer for the .Net platform: Uiml.net [6]. Our renderer
can be deployed on desktop computers, tablet PCs, digital television and on mobile devices such as
PDA’s, Mobile Phones... It works on different implementations of the .Net framework (The Microsoft
Page 41 of 81
Proceedings of CHI 2006 Workshop “The Many Faces of Consistency in Cross-Platform Design”
standard framework, the Microsoft Compact framework and Mono1), by consequence it can be
considered platform independent. This renderer enables us to create one user interface design in a
high-level XML-based language and reuse it on different platforms. The renderer takes advantage of
the widget sets that are available on the platform to transform the UIML document into the final user
interface.
UIML enables reuse of large parts of the user interface across different platforms. However, there is
no abstraction for the layout of a user interface, resulting in UIML documents that can only be reused
for a limited set of target devices. This is equivalent to what can be achieved with a generic
vocabulary for the same family of devices, as was described by Ali et al. in [3]. Although the language
also enforces consistency in various manners (a detailed discussion will follow in the next section),
the absence of a layout abstraction causes inconsistencies in the layout of the user interface and in
the part hierarchy.
SUPPORT FOR CONSISTENCY IN UIML
The structure section in a UIML document essentially defines the containment hierarchy. Although
this should mainly stay the same for a wide range of devices, the same hierarchy can not be reused
for more extreme conditions such as very small screens or multiple screens. In practice the structure
specification also contains widget-set specific layout information. This results in multiple alternative
structure specifications for the same user interface to keep them consistent for different widget-sets
on different devices.
The separation of style and structure allows decoupling the user interface structure from its visual
presentation. Different visual representations for the user interface elements can be provided which
do not affect the user interface structure: the structure is kept consistent independent of changes in
appearance.
The decoupling of the content from the other parts of the interface makes it easy to update one
without altering the others [3]. The content of a user interface has to be specified only once and can
be referenced many times by different platform-specific versions of the same interface. This elimi-
nates inconsistencies between these different versions.
The behavior element consists of a sequence of rules, each with a condition and a list of actions. A
condition can hold when an event fires. Events are independent of the supported widget sets and use
the same class and name mapping concepts as the mapping mechanism of parts. This ensures that
we can use the same event class for the user interface on different platforms. This way, UIML
guarantees that the interaction of each platform-specific version of the user interface behaves
consistently. The event class is in turn mapped onto the specific type of event used by the target
platform. For instance, we could define a selection event class, which gets mapped to an OnClick
event for direct manipulation interfaces and to an onfocus event for a speech interface.
The logic component describes the functionality of the application while hiding the application itself
and the communication between the user interface and the application with it. UIML abstracts the
way the application logic can be addressed from within the user interface. UIML enables a consistent
and platform-independent binding between the user interface and the application logic since this
abstraction can be reused for other UIML-based user interfaces.
DEVICE-INDEPENDENT UIML (DI-UIML)
UIML achieves abstraction in many ways. The specification of user interface elements, interaction,
content and the application logic are all platform-independent. However, UIML has no support for
plastic layout management, which results in platform-specific layout adjustments for each of the
target devices. Our solution supports a consistent layout in a wide range of circumstances, while still
being flexible enough to adjust to extreme conditions.
For this purpose, Device-Independent UIML (DI-UIML) extends standard UIML with a high-level way
to describe the graphical user interface layout. The designer is no longer bothered with widget-set
1
http://mono-project.com/
Page 42 of 81
Proceedings of CHI 2006 Workshop “The Many Faces of Consistency in Cross-Platform Design”
dependent details. Our approach is based on the combination of spatial constraints and a constraint
solving algorithm. The interface designer specifies the layout by defining constraints on the user
interface components, such as buttonA left-of labelB. Afterwards, the constraint solver tries to find a
solution that adheres to these constraints. Constraints are resolved on the level of the a bstract
interaction objects, so are independent of the concrete representation of the widgets.
Constraints allow us to specify the layout in a declarative manner and integrate smoothly with UIML.
The designer can focus only on what is the desired layout, rather than how this layout is to be
achieved. Furthermore, constraints allow partial specification of the layout, which can be combined
with other partial specifications in a predictable way [5]. This is useful in our case to define the layout
at several levels, taking advantage of an interface containment hierarchy (e.g. the parts in the
structure section of listing 1 specify a containment hierarchy). For example, we can define that
container selection is left-of container content. The selection and content containers can then each
on its own specify the layout of their children. When a change in this layout requires the containers to
grow, shrink or move, the upper-level layout constraints will be reevaluated. This allows us to define
generic layout patterns. These define the layout of a number of containers, which can afterwards be
filled in with a specific widget hierarchy using its own layout specification.
We developed Cassowary.net, a port of the Cassowary constraint solving toolkit [4] to the .NET
platform. This solver is used by our UIML renderer to realize the specified layout and maintain a
consistent layout of the user interface across devices.
USER INTERFACE CREATION WITH DI-UIML
Figure 2 shows the Rhythmbox2 interface redone using UIML. The interface can now be used on
different platforms and with different screen sizes (figure 2). Before, the designer could reuse most of
the UIML document for different platforms, except the layout specification and some changes in the
structure specification. Our solution solves this problem and makes it possible to reuse the interface
design with different widget sets for different screen sizes without manual intervention. If other
behavior is required, a designer can add or remove a constraint to obtain the envisioned effect, and
is no longer bothered by any platform-specific problems while designing the interface.
Listing 2 shows how the designer (or design tool) can use the new tag in a UIML
document. A layout pattern is a set of spatial constraints that can be applied to a part subtree. In
listing 2 the GTK VBox layout pattern (cfr. listing 1) can be obtained by specifying that Part1 is placed
above Part2 and that the parts are left-aligned. Constraints can be specified either directly in the
UIML document, or included by referring to an external layout pattern by using a predefined alias.
This can be compared with the usage of CSS for XHTML, but is not limited to a particular widget set
nor dependent on a web browser.
Listing 2. Specifying the layout in a UIML document
The layout template presented in listing 2 is a parameterized layout template. A layout template can
be applied to a part hierarchy by inserting it as a child of a part hierarchy. Notice the class attribute
defines the required type of interactor. At the moment of writing we have full support for constraint-
based layout management in Uiml.net, the parameterized layout templates are still work in progress
2
http://www.gnome.org/projects/rhythmbox/
Page 43 of 81
Proceedings of CHI 2006 Workshop “The Many Faces of Consistency in Cross-Platform Design”
however. This extension to the UIML language allows achieving a greater level of consistency and
reusing while reaching a higher level of abstraction in the user interface specification. Listing 3 shows
how a template can be applied onto a part hierarchy. Notice the part class Combo will be matched by
the Choice from the layout template in listing 2, since Combo is a possible instantiation for the
Choice class according to the vocabulary that is used. More details about this mapping scheme can
be found in the future work section.
Listing 3. Applying a layout template on listing 1
Previously, designers had to specify a platform-specific layout for every instantiation of the user
interface. This process relied on the careful and precise work of the designer in order to keep the
different layouts consistent. Furthermore, this process introduced a lot of work, because for every
new target platform, the layout had to be almost completely redesigned. One could even wonder if
using an abstract user interface specification like UIML was advantageous, since there was still a
large part of the interface that had to be rewritten for every platform.
Our method enables designers to create new and reuse existing layout templates. Layout templates
support consistent layouts for multi-device user interfaces. Under most conditions this layout can be
kept consistent across platforms. When more extreme circumstances arise (e.g. the interface is
deployed on a PDA, a cell phone or even distributed among several screens), the layout can adapt to
the new environment by remapping abstract interactors to more basic concrete interactors and
ignoring low-priority layout constraints.
FUTURE WORK
The current UIML vocabulary uses a one-to-one mapping of abstract interaction objects (AIOs) onto
concrete interaction objects (CIOs). Unfortunately, this is not very flexible. We have been
investigating a rule-based extension to the mapping mechanism, based on XSLT’s choice element.
This allows selecting an appropriate concrete interactor for a given abstract interactor, according to a
particular context of use. We call this method 1-to-N mapping.
Figure 2. UIML-based Rhythmbox interface rendered for different platforms.
These context-sensitive selection rules allow a more significant adaptation of the user interface to the
target platform. A rule that states “If the screen space is too small for the preferred user interface,
map the abstract range widget to a spin box instead of a slider.” would result in a user interface that
Page 44 of 81
Proceedings of CHI 2006 Workshop “The Many Faces of Consistency in Cross-Platform Design”
utilizes the slider widget when there is enough screen space (e.g. on a desktop PC), and would
otherwise (e.g. on a PDA) use the more compact spin box. Under normal conditions rich CIOs are
used, while under more extreme conditions (in this example, a small amount of screen space) we fall
back on basic CIOs. In contrast with other approaches, this approach requires the designer to specify
her preferences and excludes unexpected adaptations. This is often required by designers and gives
them the final decision over the final appearance of the user interface.
CONCLUSIONS
The User Interface Markup Language offers the user interface designer a means to create device
independent interface designs. The separation of concerns (structure, style, behavior and content)
allows keeping certain aspects of the user interface consistent across devices, while other aspects
can differ according to the target device, toolkit or user. However, UIML lacks an abstraction for the
layout of the user interface resulting in UIML documents that can only be reused for a limited set of
target devices.
In this position paper we addressed how DI-UIML, an extension of UIML, supports the creation of
consistent multi-device user interfaces. The use of spatial layout constraints leads to user interfaces
that adapt according to the given screen space and preserve the same structure and style for a wide
range of display resolutions. However, in more extreme circumstances, such as very small displays
or multiple displays, the user interface can have an inconsistent representation while still being valid
according to the constraints. Since the designer can add or remove constraints and define custom
layout patterns with these constraints, the limits for consistency are exactly defined for the range of
screen sizes for which the constraint solver can find a solution.
The work presented here is available as free software and can be found at
http://sf.net/projects/uimldotnet/. The Uiml.net renderer for UIML and DI-UIML can be obtained here,
as well as the Cassowary.net library that is used to solve the spatial layout constraints included in DI-
UIML.
ACKNOWLEDGMENTS
Part of the research at EDM is funded by ERDF (European Fund for Regional Development), the
Flemish Government and the Flemish Interdisciplinary institute for Broadband technology (IBBT).
REFERENCES
[1] Marc Abrams and James Helms. User Interface Markup Language (UIML) Specification
version 3.1. Technical report, Oasis UIML TC, 2004.
[2] Marc Abrams, Constantinos Phanouriou, Alan L. Batongbacal, Stephen M. Williams, and
Jonathan E. Shuster. UIML: An Appliance-Independent XML User Interface Language. WWW8
/ Computer Networks, 1999.
[3] Mir Farooq Ali, Manuel A. Pérez-Quiñones, Marc Abrams, and Eric Shell. Building Multi-
Platform User Interfaces with UIML. In Christophe Kolski and Jean Vanderdonckt, editors,
CADUI 2002, volume 3, pages 255–266. Kluwer Academic, 2002.
[4] Greg J. Badros, Alan Borning, and Peter J. Stuckey. The cassowary linear arithmetic
constraint solving algorithm. ACM Trans. Computer-Human Interaction, 8(4):267–306, 2001.
[5] Greg J. Badros, Alan Borning, Kim Marriott, and Peter J. Stuckey. Constraint cascading style
sheets for the web. In UIST ’99: 12th annual ACM symposium on User interface software and
technology, pages 73–82, 1999.
[6] Kris Luyten and Karin Coninx. Uiml.net: an Open Uiml Renderer for the .Net Framework. In
Computer-Aided Design of User Interfaces, 2004.
[7] Jean Vanderdonckt and Franc¸ois Bodart. Encapsulating Knowledge for Intelligent Automatic
Interaction Objects Selection. In ACM Conference on Human Aspects in Computing Systems
InterCHI’93, 1993.
Page 45 of 81