Programming graphical user interfaces is hard and expensive, while automatic generation is still quite challenging and faces even more usability problems. One of the issues involved in automatic generation is the presentation of content from the domain of discourse according to its purpose in the current state of the human-machine dialogue. We address this issue through including the intention of a given content presentation as indicated by communicative acts, and through generating it specifically according to the type of communicative act. This results in fully automatically generated user interfaces with content presentations specific to intentions.
The remainder of this paper is organized in the following manner. First, we sketch our discourse models. Then we elaborate on the model transformations from such a discourse model to a structural GUI model including content presentation. Based on such a derived model, we explain automatic screen generation for the content presentation. Finally, we compare our approach with related work.
MDDAUI 2009
User
OpenQuestion get CreditCard::cc
Opening
Alternative
Shop
ClosedQuestion get CreditCard::cc
Opening Nucleus
Nucleus Closing Closing Answer
Answer
HIGH-LEVEL DISCOURSE MODELS
The starting point for our automatic GUI generation is a
discourse model. According to [
Communicative acts represent basic units of language communication. Thus, any communication can be seen as enacting of communicative acts, acts such as making statements, giving commands, asking questions and so on. Communicative acts indicate the intention of the interaction, e.g., asking a question or issuing a request. Figure 1 shows such examples in a small excerpt of a discourse model for a simple online shop, which specifies checkout with a credit card. Communicative acts typically refer to propositional content. In this example, it is about getting the credit card information of the customer, see the text given below the type of communicative act. In fact, it is the same propositional content for both communicative acts. However, it should be presented differently, depending on the type of the communicative act, which indicates the intention of presenting this information. (c) Input string textbox rule (d) Input enum rule combobox (a) Open question rule (b) Closed question rule (e) Output open question (f) Output closed question label label rule rule
Propositional content is specified in our approach in a model of the domain of discourse. Figure 3 shows a very small excerpt of such a model in a UML class diagram.1 It specifies a (logical) CreditCard with its five attributes.
Adjacency pairs are sequences of talk “turns” that are specific to human (oral) communication, e.g., a question should have a related answer. Figure 1 shows two examples of such adjacency pairs.
RST relations specify relationships among text portions and associated constraints and effects. The relationships in a text are organized in a tree structure, where the rhetorical relations are associated with non-leaf nodes, and text portions with leaf nodes. In our work we make use of RST for linking communicative acts and further structures made up of RST relations. Figure 1 shows an example of an Alternative RST relation linking two adjacency pairs.
MODEL TRANSFORMATION TO STRUCTURAL UI MODEL Our model-driven approach transforms discourse models into structural UI models that are close to the final user interface but still GUI toolkit-independent. It is a process consisting of two interleaved transformation steps: 1. The first step applies rules that generate an overall UI structure based on patterns matched in the discourse model. These rules generate abstract widgets like labels for headings and placeholders for data of the propositional content. They also select the parts of the propositional content to be rendered and associates them with the placeholders. 2. The second step executes specific content transformation 1At the time of this writing, the specification of UML is available at http://www.omg.org. rules within the context of the rules of the first step. This allows the selection of abstract widgets depending on the content type, the content’s referring communicative act type and the current context the communicative act is embedded in as defined by the enclosing rule.
We explain this process in more detail by means of our running example. For transforming the discourse model excerpt in Figure 1, we need structural transformation rules for transforming the question-answer adjacency pairs, as well as content transformation rules for transforming strings, enumerations and numbers dependent on the communicative act. The two structural rules below are applied first to the discourse model.
Open Question Rule: The rule in Figure 2(a) matches an adjacency pair relating an open question and an answer (upper part of the figure) and transforms it to a panel containing a label for the question text, input and output widget placeholders for the content, and a submit button for submitting the answer. The rule also assigns all attributes of the open question’s propositional content type to both placeholders. In our example, these attributes are the ones of a credit card. Closed Question Rule: The rule in Figure 2(b) transforms each closed question-answer adjacency pair to a list with each list entry consisting of a heading label, an output widget placeholder for the description of one item of the closed question and a button to select it.
The following four rules are used in the online shop example to transform content types depending on the type of communicative act referred from.
Basic Input String TextBox Rule: The rule in Figure 2(c) matches the string content type and is constrained to input widget placeholders. It generates a text box for each string. In our example, it generates for each credit card string attribute associated with the input widget placeholder of the open question a text box. Afterwards the placeholder gets removed.
Basic Input Enum ComboBox Rule: The rule in Figure 2(d) is similar to the Basic Input String TextBox Rule, but matches enumerations and creates a combo box with the literals of the enumeration as selection list.
Basic Output Open Question Label Rule: The rule in Figure 2(e) matches any content type of content attributes assigned to output widget placeholders generated for open question communicative acts (specified by the two type constraints). For each matched content attribute, e.g., credit card number, the rule generates a label and sets the label text to the attribute name. This label is used to identify the attribute’s corresponding input widget generated by one of the rules described above.
Basic Output Closed Question Label Rule: The rule in Figure 2(f) matches also any content type of content attributes assigned to output widget placeholders generated for closed question communicative acts. In contrast to the Basic Output Open Question Label Rule, this rule assigns the actual propositional content to the generated label.
After these rules are applied to our small running example discourse, we get the generated structural UI model illustrated in Figure 4. The resulting structure of the open question and closed question adjacency pairs marked by the rounded rectangle in Figure 4 corresponds to the structures shown in Figure 2(a) and (b) with the input and output placeholder widgets replaced by the application of the other rules. The surrounding structure in Figure 4 corresponds to the Alternative RST relation and is generated by rules not presented in this paper.
SCREEN GENERATION
A structural UI model resulting from our model
transformation process, like the one in Figure 4, contains already the
complete structure and layout information of the GUI but is
still GUI toolkit-independent. Screen generation is our final
step that transforms the structural model into GUI
toolkitspecific windows and dialogs and generates code for them.
Currently, we support the Java Swing2 and Eclipse SWT3
GUI toolkits. This screen generation step solves three tasks:
It maps the abstract widgets of the structural model to
toolkit-specific widgets,
it maps the generic structural UI layout to a toolkit-specific
layout, and
it generates the event handling and the binding to the user
interface behavior (represented as a generated finite-state
machine as described in [
Figure 5 displays the screen resulting from the structural UI model in Figure 4 by applying the screen generation for Eclipse SWT. In this example, we achieved a one-to-one mapping between structural model and SWT widgets. In some cases, the mapping process is more complex, e.g., our structural UI metamodel contains an image map widget which 2http://java.sun.com/products/jfc/ 3http://www.eclipse.org/swt requires, for example, an image, a label widget and the implementation of multiple active areas within SWT. For transforming the layout information of the structural UI model, we implemented an algorithm that calculates layout data required for the toolkit-specific layout managers. E.g., for the Java Swing GridBagLayout we calculate the weight of each grid cell, which is important for resizing of the window, depending on the widget types and the layouting rule applied to the discourse relation. For example, the Alternative relation used in our running example weighs both branches equally, thus the actual resize behavior depends only on the widgets used.
In addition, the screen generation results in toolkit-specific event handlers that collect information from the input widgets, modify content objects provided by the application logic appropriately or generate new ones. Afterwards, the event handlers create input for the application logic and hand them over to the finite-state machine that implements the user interface behavior, which then selects the next screen according to the advancement of the dialogue.
Finally, the screen generation process also internationalizes and localizes the generated GUI, e.g., it externalizes strings and provides translation files with default suggestions derived from the discourse model, the content types and the transformation rules.
RELATED WORK
Our approach to GUI generation is similar to TERESA by
Mori et al. [
ACKNOWLEDGMENT This research has been carried out in the CommRob project (http://www.commrob.eu) and is partially funded by the EU (contract number IST-045441 under the 6th framework programme).