=Paper=
{{Paper
|id=None
|storemode=property
|title=Automated Optimization of UIs for Screens with Limited Resolution
|pdfUrl=https://ceur-ws.org/Vol-617/MDDAUI2010_Paper04.pdf
|volume=Vol-617
}}
==Automated Optimization of UIs for Screens with Limited Resolution==
https://ceur-ws.org/Vol-617/MDDAUI2010_Paper04.pdf
Automated Optimization of User Interfaces
for Screens with Limited Resolution
Sevan Kavaldjian, David Raneburger, Roman Popp, Michael Leitner, Jürgen Falb,
Hermann Kaindl
Institute of Computer Technology
Vienna University of Technology, Austria
{kavaldjian, raneburger, popp, leitner, falb, kaindl}@ict.tuwien.ac.at
ABSTRACT
More and more devices with different screen resolutions are
used to run the same application. In order to reduce usabil-
ity problems, user interfaces (UIs) specific to resolution are
needed, but it is time consuming and costly to implement
all the different UIs manually. Automated generation of UIs
has the potential to reduce time and costs in case of many
such devices. Still, model-driven generation of UIs may not
be flexible enough to include optimization for various reso-
lutions.
We extended the straight-forward approach to model-driven
generation by including optimization according to maximum
usage of available space for a given resolution, minimum
amount of clicks, and minimum scrolling. For these opti-
mizations, we also use automated layouting and calculate
the space needs of the possible variants. In effect, our new
approach generates UIs optimized for screens with limited
Figure 1. A discourse model excerpt
resolution, in order to reduce related usability problems.
displayed on a screen with limited resolution, to minimize
INTRODUCTION
the number of navigation clicks, and to minimize scrolling.
Automated generation of UIs has certainly advanced in re-
All this is important for reducing usability problems. Since
cent years, especially based on model-driven approaches.
more and more devices with different screen resolutions are
Still, such generated UIs pose many usability problems. We
used to run the same application, we can automatically opti-
think that this is partly due to insufficient flexibility of the
mize the generated UI for the given (limited) resolution.
current generation approaches.
In particular, straight-forward model-driven generation only BACKGROUND
allows for matching a single transformation rule for each The input for our UI generation approach is a discourse
source pattern. We extend this approach by taking up means model [2]. Such a discourse model serves as an interaction
from rule-based programming, that have been around for design on a high level of abstraction based on concepts of
a long time. We allow matching of several transformation human language theories. A small excerpt of a larger dis-
rules for any source pattern, and we use so-called conflict course model for flight booking is shown in Figure 1. We
resolution to determine which rule to apply (fire). Based on use this discourse model as a running example throughout
that, we implement a simple form of optimization in the con- the remainder of this paper.
text of model-driven UI generation.
Our Discourse Models
It allows us to maximize the amount of information to be
The main ingredients of our discourse models are commu-
nicative acts derived from speech acts [7]. A communicative
act is represented as a rounded rectangle and models an utter-
Pre-proceedings of the 5th International Workshop on Model Driven Development of
Advanced User Interfaces (MDDAUI 2010): Bridging between User Experience and
ance of one of the communication partners. In our example,
UI Engineering, organized at the 28th ACM Conference on Human Factors in the application asks the customer closed questions, while the
Computing Systems (CHI 2010), Atlanta, Georgia, USA, April 10, 2010. customer provides answers to the questions. In our example,
the yellow (or light gray) communicative acts are uttered by
Copyright © 2010 for the individual papers by the papers' authors. Copying permitted
for private and academic purposes. Re-publication of material from this volume the application and the green (or dark gray) ones are uttered
requires permission by the copyright owners. This volume is published by its editors. by the customer.
MDDAUI 2010.
1
13
Model2Model Model2Code
Discourse Transformation Structural UI Transformation Final UI
Model Model
(a) Our Basic Transformation Process
Further Transformation Possible
Fitting UI
Model2Model Check if UI fits or Model2Code
Discourse Transformation Structural UI Screen Resolution No Further Transformation Final UI
Model Model Transformation
Possible
(b) Our Extended Transformation Process
Figure 2. Our Transformation Process
Additionally, some communicative acts, like Question and matches each of the two Question–Answer adjacency pairs.
Answer form a so-called adjacency pair which is represented For each adjacency pair a panel containing a label for a head-
by a diamond in our discourse models and defines the turn- ing, a list of radio buttons together with item labels, and a
taking and thus the sequence of utterances. submit button on the bottom is added to the Structural UI
Model.
Adjacency pairs can be related with each other to build a
tree structure. In our example, two Question–Answer pairs In the second step this Structural UI Model is used to gen-
are related by a Joint relation. This Joint relation states that erate source code for a particular target platform, e.g., Java
the Question–Answer pairs in both nucleus branches are of Swing in our running example.
equal importance. Further, it does not imply a temporal or-
der per se. For instance, both pieces of information can be OPTIMIZED RENDERING FOR LIMITED RESOLUTIONS
presented in parallel if there is enough space on the screen. The problem tackled by our extended approach is to fit a
Otherwise they can be uttered in sequence. given amount of information optimally (in the following
sense) into screens with limited resolution.
Our Basic Transformation Process
We have developed a user interface generation process [5] Optimization Objectives and Approach
that transforms such discourse models into WIMP-based We assume that the following optimization objectives im-
graphical user interfaces (Windows, Icons, Menu and Point- prove the usability of the generated user interfaces:
ers). Our basic user interface generation process is illus-
trated in Figure 2(a) and consists of two steps. • maximum use of the available space for the given resolu-
tion,
The first step transforms a discourse model into a Struc-
tural UI Model [4] by applying transformation rules to dis- • minimum amount of navigation clicks, and
course model elements. The resulting Structural UI Model • minimum scrolling (except list widgets).
represents the user interface’s widgets and their structure,
but still abstracts from details of the final UI. We do not Whenever the given information to be displayed does not fit
use a common UI description language (e.g. UsiXML1 ) be- into a single screen with default widgets, we try to display
cause our runtime environment is based on the exchange of it with widgets that use less space. If it still does not fit
Communicative Acts. In our running example, the following into a single screen, then we split its display to two or more
transformation rules are applied to elements of the discourse screens. Splitting increases the number of navigation clicks
model excerpt in Figure 1 for generating a model represent- but it minimizes scrolling. We exclude list widgets from this
ing the structure of the final UI in Figure 3(a). last optimization objective because the number of list entries
can vary extremely at runtime and determines whether the
First, a Joint Rule gets applied that matches the Joint relation list is scrollable or not. This information is not known during
and adds a panel to the Structural UI Model. This panel acts our rendering process.
as a container for the Radio Button lists in Figure 3(a), which
correspond to the two nucleus branches of the Joint relation. Our Extended Transformation Process
Our basic transformation process looks like straight-forward
Second, a Closed Question Rule gets applied twice that
model-driven generation that only allows for matching a sin-
1
http://www.usixml.org gle transformation rule for each source pattern. We are not
2
14
aware of any optimization strategy in such a context. There-
fore, we extend the straight-forward approach by allowing
that several transformation rules may match for each source
pattern, and by applying so-called conflict resolution to se-
lect which rule to apply (fire) in the next model-to-model
transformation. Our extended generation process shown in
Figure 2(b) illustrates the resulting possibility of trying out
several rules for optimization purposes. In this approach,
the rules need not to be specifically designed for a particular
screen resolution. It is rather the way the rules are applied
that achieves the given optimization objectives.
In order to implement such an optimization, the conflict res-
olution mechanism needs to select the rules in a certain de-
fined order. For achieving the optimization objectives given
(a) 640×480
above, this selection order is according to the space that the
widgets the rule creates occupy in the final UI. Therefore,
all rules matching the same discourse element for transfor-
mation have to be ranked by the designer according to this
space need.
Each target device we render for has an abstract device spec-
ification that contains all style data used by the transforma-
tion rules. These data specify default sizes for all input and
output widgets on the target device that can be overwritten
in a transformation rule. They are used to set the size for
each final UI element and allow us to calculate the exact size (b) 480×320
of each container (e.g., panel). For example, we set the size
of the list widget explicitly. This makes it independent from
the number of entries. If the list widget is not able to display
all entries, it becomes scrollable.
After the size calculation we try to layout each generated
screen to fit into the given resolution. However, we mod-
ify only the arrangement of the widgets that has not been (c) 320×180
fixed explicitly in a transformation rule. Therefore, we do
not change the layout specified by the Closed Question Rule Figure 3. Generated User Interfaces
(i.e., the layout of the heading label, the radio button list
and the submit button in Figure 3(a)). In this example, we
modify the position of the complete radio button lists in the
panel created by the Joint relation, since the Joint Rule does rules does not fit. As long as a lower ranked rule can be ap-
not contain any layout information. plied, we initiate another generation cycle. First, the Small
Closed Question Rule is used in our example. This rule
Now let us explain how to apply this approach to automati- matches the same source element (Question–Answer adja-
cally generate user interfaces for three target resolutions. As cency pair) as the Closed Question Rule but it creates a UI
input we use the discourse model excerpt shown in Figure 1. structure which occupies less space on the screen. A combo
box element presents the content of the Closed Question
Our first GUI is rendered for the resolution 640×480. The communicative act to the user and a submit button is gen-
first cycle of the model-to-model transformation uses the erated to confirm the selection of the user. The user inter-
highest ranked rules (i.e., the ones with the highest space face shown in Figure 3(b) is the result of two more cycles,
need) for each discourse element. These are the same rules because in each cycle only one lower ranked rule is applied.
that have been applied in our basic transformation process. The resulting UI fits and still presents the same information,
After the first transformation cycle we calculate the size for but using widgets with less space needs (combo boxes in-
each panel in the corresponding Structural UI model. We can stead of radio buttons). However, the list widgets do not fit
place them next to each other without exceeding the screen next to each other and the layouter arranges them vertically.
resolution. So, this is a fitting UI and we trigger the model-
to-code transformation. The result is shown in Figure 3(a). In a third run, we generate a user interface for the resolution
320×180. Even after all rules for widget selection have been
Next we generate a UI for the resolution 480×320. This tried out, the generated GUI still does not fit the given res-
time, the UI resulting from application of the highest ranked olution. Therefore, we start using rules that split the screen
in order to increase the number of navigation clicks before
3
15
we make use of scrolling. In our example, this means that in same rule set to be used for generating UIs for devices with
the next cycle the Small Joint Rule is applied instead of the different resolutions. Through the automatic calculation of
Joint Rule. The Small Joint Rule matches the same source space need, it may even have an advantage in this respect as
element (Joint relation) but creates a different UI structure (a compared to a human interface designer. We implemented
tabbed pane element instead of a panel). Figure 3(c) shows a simple optimization approach that allows us to optimize
the outcome for the resolution 320×180. The Small Joint generated UIs for devices with limited resolution in such a
Rule and the Small Closed Question Rule have been applied way as to utilize the given space and to minimize naviga-
and a fitting UI has been generated after a third cycle of rule tion and scrolling. This should pave the way to optimized
application. This time no layout modifications are necessary multidevice UI generation.
because each tab contains only one panel.
ACKNOWLEDGMENTS
The worst case in our extended generation process occurs This research has been carried out in the CommRob project
if and when no more rules are available and the generated (http://www.commrob.eu), partially funded by the
screen still does not fit the given screen resolution. In this EU (contract number IST-045441 under the 6th framework
case, we stop the optimization loop and rely on scrolling. programme).
RELATED WORK REFERENCES
A transformation system that fits web pages automated and 1. B. Collignon, J. Vanderdonckt, and G. Calvary.
on-the-fly to screens of small devices is presented in [8]. The Model-driven engineering of multi-target plastic user
transformations are performed in order to minimize naviga- interfaces. In Proceedings of the Fourth International
tion and scrolling like in our approach. In contrast, however, Conference on Autonomic and Autonomous Systems
this process alters an already existing UI. (ICAS 2008), pages 7–14, Washington, DC, USA, 2008.
IEEE Computer Society.
Declarative user interface specifications are used as input for
multi-target UI generation in [3]. The user interface adap- 2. J. Falb, H. Kaindl, H. Horacek, C. Bogdan, R. Popp, and
tion is treated as an optimization problem based on a user- E. Arnautovic. A discourse model for interaction design
and device-specific cost function. Compared to such user in- based on theories of human communication. In Extended
terface specifications, our interaction models are on a higher Abstracts on Human Factors in Computing Systems
level of abstraction. (CHI ’06), pages 754–759, New York, NY, USA, 2006.
ACM Press.
The model-driven approach for engineering multi-target UIs 3. K. Gajos and D. S. Weld. SUPPLE: Automatically
presented in [1] supports switching between predefined pre- generating user interfaces. In Proceedings of the 9th
sentations during runtime. Our approach, in contrast, is in- International Conference on Intelligent User Interface
tended to automatically generate presentations for different (IUI ’04), pages 93–100, New York, NY, USA, 2004.
resolutions from a single discourse model. ACM Press.
An advanced approach for generating multi-device UIs is 4. S. Kavaldjian, C. Bogdan, J. Falb, and H. Kaindl.
based on task models [6]. Such a Task Model specifies the Transforming discourse models to structural user
temporal relations among tasks and has to be adapted ac- interface models. In Models in Software Engineering,
cording to the screen space available on the target device. LNCS 5002, volume 5002/2008, pages 77–88. Springer,
Therefore, any optimization and screen splitting has to be Berlin / Heidelberg, 2008.
done explicitly during the creation of the Task Model.
5. S. Kavaldjian, J. Falb, and H. Kaindl. Generating content
We are not aware of any other approach that performs op- presentation according to purpose. In Proceedings of the
timization in the course of model transformations. Neither 2009 IEEE International Conference on Systems, Man
are we aware of any model-driven GUI transformation pro- and Cybernetics (SMC2009), San Antonio, TX, USA,
cess that takes the resolution for transformation rule selec- Oct. 2009.
tion into account. 6. F. Paternò, C. Santoro, and L. D. Spano. Model-based
design of multi-device interactive applications based on
CONCLUSION web services. In INTERACT (1), pages 892–905, 2009.
Our new and extended approach introduces an optimization
technique into model-driven generation of UIs to reduce us- 7. J. R. Searle. Speech Acts: An Essay in the Philosophy of
ability problems. However, we only deal with relatively sim- Language. Cambridge University Press, Cambridge,
ple usability aspects (minimum amount of clicks and mini- England, 1969.
mum scrolling). We do not (yet) optimize layout according 8. X. Xiao, Q. Luo, D. Hong, H. Fu, X. Xie, and W.-Y. Ma.
to, e.g., aesthetic criteria. Therefore, our optimization ap- Browsing on small displays by transforming web pages
proach as presented above is not suitable for large screens into hierarchically structured subpages. ACM
with high resolution. Transactions on the Web, 3(1):1–36, 2009.
Overall we introduce a UI generation process that allows the
4
16