More and more devices with different screen resolutions are used to run the same application. In order to reduce usability problems, user interfaces (UIs) specific to resolution are needed, but it is time consuming and costly to implement all the different UIs manually. Automated generation of UIs has the potential to reduce time and costs in case of many such devices. Still, model-driven generation of UIs may not be flexible enough to include optimization for various resolutions. We extended the straight-forward approach to model-driven generation by including optimization according to maximum usage of available space for a given resolution, minimum amount of clicks, and minimum scrolling. For these optimizations, we also use automated layouting and calculate the space needs of the possible variants. In effect, our new approach generates UIs optimized for screens with limited resolution, in order to reduce related usability problems.
Automated generation of UIs has certainly advanced in recent years, especially based on model-driven approaches. Still, such generated UIs pose many usability problems. We think that this is partly due to insufficient flexibility of the current generation approaches.
In particular, straight-forward model-driven generation only allows for matching a single transformation rule for each source pattern. We extend this approach by taking up means from rule-based programming, that have been around for a long time. We allow matching of several transformation rules for any source pattern, and we use so-called conflict resolution to determine which rule to apply (fire). Based on that, we implement a simple form of optimization in the context of model-driven UI generation.
It allows us to maximize the amount of information to be Copyright © 2010 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. Re-publication of material from this volume requires permission by the copyright owners. This volume is published by its editors. MDDAUI 2010. displayed on a screen with limited resolution, to minimize the number of navigation clicks, and to minimize scrolling. All this is important for reducing usability problems. Since more and more devices with different screen resolutions are used to run the same application, we can automatically optimize the generated UI for the given (limited) resolution.
The input for our UI generation approach is a discourse
model [
The main ingredients of our discourse models are
communicative acts derived from speech acts [
Model2Model Transformation
Structural UI
Model Model2Code Transformation
Final UI (a) Our Basic Transformation Process Further Transformation Possible (b) Our Extended Transformation Process Additionally, some communicative acts, like Question and Answer form a so-called adjacency pair which is represented by a diamond in our discourse models and defines the turntaking and thus the sequence of utterances.
Adjacency pairs can be related with each other to build a tree structure. In our example, two Question–Answer pairs are related by a Joint relation. This Joint relation states that the Question–Answer pairs in both nucleus branches are of equal importance. Further, it does not imply a temporal order per se. For instance, both pieces of information can be presented in parallel if there is enough space on the screen. Otherwise they can be uttered in sequence.
We have developed a user interface generation process [
The first step transforms a discourse model into a
Structural UI Model [
First, a Joint Rule gets applied that matches the Joint relation and adds a panel to the Structural UI Model. This panel acts as a container for the Radio Button lists in Figure 3(a), which correspond to the two nucleus branches of the Joint relation. Second, a Closed Question Rule gets applied twice that matches each of the two Question–Answer adjacency pairs. For each adjacency pair a panel containing a label for a heading, a list of radio buttons together with item labels, and a submit button on the bottom is added to the Structural UI Model.
In the second step this Structural UI Model is used to generate source code for a particular target platform, e.g., Java Swing in our running example.
The problem tackled by our extended approach is to fit a given amount of information optimally (in the following sense) into screens with limited resolution.
We assume that the following optimization objectives improve the usability of the generated user interfaces: • maximum use of the available space for the given resolution, • minimum amount of navigation clicks, and • minimum scrolling (except list widgets).
Whenever the given information to be displayed does not fit into a single screen with default widgets, we try to display it with widgets that use less space. If it still does not fit into a single screen, then we split its display to two or more screens. Splitting increases the number of navigation clicks but it minimizes scrolling. We exclude list widgets from this last optimization objective because the number of list entries can vary extremely at runtime and determines whether the list is scrollable or not. This information is not known during our rendering process.
Our basic transformation process looks like straight-forward model-driven generation that only allows for matching a single transformation rule for each source pattern. We are not aware of any optimization strategy in such a context. Therefore, we extend the straight-forward approach by allowing that several transformation rules may match for each source pattern, and by applying so-called conflict resolution to select which rule to apply (fire) in the next model-to-model transformation. Our extended generation process shown in Figure 2(b) illustrates the resulting possibility of trying out several rules for optimization purposes. In this approach, the rules need not to be specifically designed for a particular screen resolution. It is rather the way the rules are applied that achieves the given optimization objectives.
In order to implement such an optimization, the conflict resolution mechanism needs to select the rules in a certain defined order. For achieving the optimization objectives given above, this selection order is according to the space that the widgets the rule creates occupy in the final UI. Therefore, all rules matching the same discourse element for transformation have to be ranked by the designer according to this space need.
Each target device we render for has an abstract device specification that contains all style data used by the transformation rules. These data specify default sizes for all input and output widgets on the target device that can be overwritten in a transformation rule. They are used to set the size for each final UI element and allow us to calculate the exact size of each container (e.g., panel). For example, we set the size of the list widget explicitly. This makes it independent from the number of entries. If the list widget is not able to display all entries, it becomes scrollable.
After the size calculation we try to layout each generated screen to fit into the given resolution. However, we modify only the arrangement of the widgets that has not been fixed explicitly in a transformation rule. Therefore, we do not change the layout specified by the Closed Question Rule (i.e., the layout of the heading label, the radio button list and the submit button in Figure 3(a)). In this example, we modify the position of the complete radio button lists in the panel created by the Joint relation, since the Joint Rule does not contain any layout information.
Now let us explain how to apply this approach to automatically generate user interfaces for three target resolutions. As input we use the discourse model excerpt shown in Figure 1. Our first GUI is rendered for the resolution 640×480. The first cycle of the model-to-model transformation uses the highest ranked rules (i.e., the ones with the highest space need) for each discourse element. These are the same rules that have been applied in our basic transformation process. After the first transformation cycle we calculate the size for each panel in the corresponding Structural UI model. We can place them next to each other without exceeding the screen resolution. So, this is a fitting UI and we trigger the modelto-code transformation. The result is shown in Figure 3(a). Next we generate a UI for the resolution 480×320. This time, the UI resulting from application of the highest ranked (a) 640×480 (b) 480×320 rules does not fit. As long as a lower ranked rule can be applied, we initiate another generation cycle. First, the Small Closed Question Rule is used in our example. This rule matches the same source element (Question–Answer adjacency pair) as the Closed Question Rule but it creates a UI structure which occupies less space on the screen. A combo box element presents the content of the Closed Question communicative act to the user and a submit button is generated to confirm the selection of the user. The user interface shown in Figure 3(b) is the result of two more cycles, because in each cycle only one lower ranked rule is applied. The resulting UI fits and still presents the same information, but using widgets with less space needs (combo boxes instead of radio buttons). However, the list widgets do not fit next to each other and the layouter arranges them vertically. In a third run, we generate a user interface for the resolution 320×180. Even after all rules for widget selection have been tried out, the generated GUI still does not fit the given resolution. Therefore, we start using rules that split the screen in order to increase the number of navigation clicks before we make use of scrolling. In our example, this means that in the next cycle the Small Joint Rule is applied instead of the Joint Rule. The Small Joint Rule matches the same source element (Joint relation) but creates a different UI structure (a tabbed pane element instead of a panel). Figure 3(c) shows the outcome for the resolution 320×180. The Small Joint Rule and the Small Closed Question Rule have been applied and a fitting UI has been generated after a third cycle of rule application. This time no layout modifications are necessary because each tab contains only one panel.
The worst case in our extended generation process occurs if and when no more rules are available and the generated screen still does not fit the given screen resolution. In this case, we stop the optimization loop and rely on scrolling.
A transformation system that fits web pages automated and
on-the-fly to screens of small devices is presented in [
Declarative user interface specifications are used as input for
multi-target UI generation in [
The model-driven approach for engineering multi-target UIs
presented in [
An advanced approach for generating multi-device UIs is
based on task models [
Our new and extended approach introduces an optimization technique into model-driven generation of UIs to reduce usability problems. However, we only deal with relatively simple usability aspects (minimum amount of clicks and minimum scrolling). We do not (yet) optimize layout according to, e.g., aesthetic criteria. Therefore, our optimization approach as presented above is not suitable for large screens with high resolution.
Overall we introduce a UI generation process that allows the same rule set to be used for generating UIs for devices with different resolutions. Through the automatic calculation of space need, it may even have an advantage in this respect as compared to a human interface designer. We implemented a simple optimization approach that allows us to optimize generated UIs for devices with limited resolution in such a way as to utilize the given space and to minimize navigation and scrolling. This should pave the way to optimized multidevice UI generation.
This research has been carried out in the CommRob project (http://www.commrob.eu), partially funded by the EU (contract number IST-045441 under the 6th framework programme).