=Paper=
{{Paper
|id=Vol-439/paper-6
|storemode=property
|title=Fully Automatic Content Presentation Specific to Intentions
|pdfUrl=https://ceur-ws.org/Vol-439/paper6.pdf
|volume=Vol-439
}}
==Fully Automatic Content Presentation Specific to Intentions==
https://ceur-ws.org/Vol-439/paper6.pdf
Fully automatic content presentation specific to intentions
Sevan Kavaldjian, Jürgen Falb, Hermann Kaindl
Institute of Computer Technology
Vienna University of Technology, Austria
{kavaldjian, falb, kaindl}@ict.tuwien.ac.at
ABSTRACT
Programming graphical user interfaces is hard and expen- Alternative
sive, while automatic generation is still quite challenging User Shop
and faces even more usability problems. One of the issues
involved in automatic generation is the presentation of con-
OpenQuestion ClosedQuestion
tent from the domain of discourse according to its purpose
in the current state of the human-machine dialogue. We ad- get CreditCard::cc get CreditCard::cc
dress this issue through including the intention of a given
Opening Opening
content presentation as indicated by communicative acts, and
through generating it specifically according to the type of Nucleus Nucleus
communicative act. This results in fully automatically gen-
erated user interfaces with content presentations specific to
intentions. Closing Closing
Answer Answer
INTRODUCTION
Manual creation of graphical user interfaces (GUIs) is hard
and expensive, so we strive for automatic generation. In-
stead of generating them from simple abstractions, we let an Figure 1. Excerpt of an online shop discourse model.
interaction designer (and even end users) model discourses
in the sense of dialogues (supported by a tool) [1]. From
such a high-level declarative discourse model, we have been HIGH-LEVEL DISCOURSE MODELS
able to automatically generate the overall structure and the The starting point for our automatic GUI generation is a dis-
“look” of a GUI [3] as well as its behavior [8]. This auto- course model. According to [2], such a declarative discourse
matic generation employs model transformations. model has the following key ingredients:
Still, we had to deal with the presentation of content of the • communicative acts as derived from speech acts [10],
domain of discourse. In particular, the concrete presentation
needs to depend on the intention of the envisaged interaction, • adjacency pairs adopted from Conversation Analysis [4],
since the GUI’s usability would clearly not be satisfactory and
otherwise. In this paper, we present our approach to auto-
• RST relations inherited from Rhetorical Structure Theory
matic content generation specific to intentions, which also
(RST) [5].
employs model transformations.
Communicative acts represent basic units of language com-
The remainder of this paper is organized in the following
munication. Thus, any communication can be seen as enact-
manner. First, we sketch our discourse models. Then we
ing of communicative acts, acts such as making statements,
elaborate on the model transformations from such a discourse
giving commands, asking questions and so on. Communica-
model to a structural GUI model including content presenta-
tive acts indicate the intention of the interaction, e.g., asking
tion. Based on such a derived model, we explain automatic
a question or issuing a request. Figure 1 shows such exam-
screen generation for the content presentation. Finally, we
ples in a small excerpt of a discourse model for a simple
compare our approach with related work.
online shop, which specifies checkout with a credit card.
Communicative acts typically refer to propositional content.
In this example, it is about getting the credit card information
of the customer, see the text given below the type of commu-
nicative act. In fact, it is the same propositional content for
both communicative acts. However, it should be presented
differently, depending on the type of the communicative act,
which indicates the intention of presenting this information.
MDDAUI 2009
CEUR
Workshop
ceur-ws.org
ISSN 1613-0073
1
Proceedings
(c) Input string textbox (d) Input enum combobox
rule rule
(a) Open question rule (b) Closed question rule (e) Output open question (f) Output closed question label
label rule rule
Figure 2. Transformation rules for open questions (a) and closed questions (b), for the ContentData input widget of the open question depending on
the content type string (c) and enumeration (d), and rules for the output widgets of the open question (e) and the closed question (f).
rules within the context of the rules of the first step. This
allows the selection of abstract widgets depending on the
content type, the content’s referring communicative act
type and the current context the communicative act is em-
bedded in as defined by the enclosing rule.
We explain this process in more detail by means of our run-
Figure 3. Excerpt of a domain of discourse model. ning example. For transforming the discourse model excerpt
in Figure 1, we need structural transformation rules for trans-
Propositional content is specified in our approach in a model forming the question-answer adjacency pairs, as well as con-
of the domain of discourse. Figure 3 shows a very small ex- tent transformation rules for transforming strings, enumera-
cerpt of such a model in a UML class diagram.1 It specifies tions and numbers dependent on the communicative act. The
a (logical) CreditCard with its five attributes. two structural rules below are applied first to the discourse
model.
Adjacency pairs are sequences of talk “turns” that are spe-
cific to human (oral) communication, e.g., a question should Open Question Rule: The rule in Figure 2(a) matches an
have a related answer. Figure 1 shows two examples of such adjacency pair relating an open question and an answer (up-
adjacency pairs. per part of the figure) and transforms it to a panel containing
a label for the question text, input and output widget place-
RST relations specify relationships among text portions and holders for the content, and a submit button for submitting
associated constraints and effects. The relationships in a text the answer. The rule also assigns all attributes of the open
are organized in a tree structure, where the rhetorical rela- question’s propositional content type to both placeholders.
tions are associated with non-leaf nodes, and text portions In our example, these attributes are the ones of a credit card.
with leaf nodes. In our work we make use of RST for link-
ing communicative acts and further structures made up of Closed Question Rule: The rule in Figure 2(b) transforms
RST relations. Figure 1 shows an example of an Alternative each closed question-answer adjacency pair to a list with
RST relation linking two adjacency pairs. each list entry consisting of a heading label, an output wid-
get placeholder for the description of one item of the closed
question and a button to select it.
MODEL TRANSFORMATION TO STRUCTURAL UI MODEL
Our model-driven approach transforms discourse models into The following four rules are used in the online shop example
structural UI models that are close to the final user interface to transform content types depending on the type of commu-
but still GUI toolkit-independent. It is a process consisting nicative act referred from.
of two interleaved transformation steps:
Basic Input String TextBox Rule: The rule in Figure 2(c)
1. The first step applies rules that generate an overall UI matches the string content type and is constrained to input
structure based on patterns matched in the discourse model. widget placeholders. It generates a text box for each string.
These rules generate abstract widgets like labels for head- In our example, it generates for each credit card string at-
ings and placeholders for data of the propositional con- tribute associated with the input widget placeholder of the
tent. They also select the parts of the propositional content open question a text box. Afterwards the placeholder gets
to be rendered and associates them with the placeholders. removed.
2. The second step executes specific content transformation
Basic Input Enum ComboBox Rule: The rule in Figure
1
At the time of this writing, the specification of UML is available 2(d) is similar to the Basic Input String TextBox Rule, but
at http://www.omg.org.
2
Figure 5. Screenshot of the final UI representing the alternative closed
question and open question.
After these rules are applied to our small running exam-
ple discourse, we get the generated structural UI model il-
lustrated in Figure 4. The resulting structure of the open
question and closed question adjacency pairs marked by the
rounded rectangle in Figure 4 corresponds to the structures
shown in Figure 2(a) and (b) with the input and output place-
holder widgets replaced by the application of the other rules.
The surrounding structure in Figure 4 corresponds to the
Alternative RST relation and is generated by rules not pre-
sented in this paper.
SCREEN GENERATION
A structural UI model resulting from our model transforma-
tion process, like the one in Figure 4, contains already the
complete structure and layout information of the GUI but is
Figure 4. Structural UI model corresponding to the online shop dis-
still GUI toolkit-independent. Screen generation is our final
course model excerpt. step that transforms the structural model into GUI toolkit-
specific windows and dialogs and generates code for them.
Currently, we support the Java Swing2 and Eclipse SWT3
GUI toolkits. This screen generation step solves three tasks:
matches enumerations and creates a combo box with the lit-
• It maps the abstract widgets of the structural model to
erals of the enumeration as selection list.
toolkit-specific widgets,
Basic Output Open Question Label Rule: The rule in • it maps the generic structural UI layout to a toolkit-specific
Figure 2(e) matches any content type of content attributes layout, and
assigned to output widget placeholders generated for open
question communicative acts (specified by the two type con- • it generates the event handling and the binding to the user
straints). For each matched content attribute, e.g., credit card interface behavior (represented as a generated finite-state
number, the rule generates a label and sets the label text machine as described in [8]).
to the attribute name. This label is used to identify the at-
tribute’s corresponding input widget generated by one of the Figure 5 displays the screen resulting from the structural UI
rules described above. model in Figure 4 by applying the screen generation for
Eclipse SWT. In this example, we achieved a one-to-one
Basic Output Closed Question Label Rule: The rule in mapping between structural model and SWT widgets. In
Figure 2(f) matches also any content type of content attributes some cases, the mapping process is more complex, e.g., our
assigned to output widget placeholders generated for closed structural UI metamodel contains an image map widget which
question communicative acts. In contrast to the Basic Out-
2
put Open Question Label Rule, this rule assigns the actual http://java.sun.com/products/jfc/
3
propositional content to the generated label. http://www.eclipse.org/swt
3
requires, for example, an image, a label widget and the im- ACKNOWLEDGMENT
plementation of multiple active areas within SWT. This research has been carried out in the CommRob project
(http://www.commrob.eu) and is partially funded by
For transforming the layout information of the structural UI the EU (contract number IST-045441 under the 6th frame-
model, we implemented an algorithm that calculates layout work programme).
data required for the toolkit-specific layout managers. E.g.,
for the Java Swing GridBagLayout we calculate the weight REFERENCES
of each grid cell, which is important for resizing of the win- 1. C. Bogdan, H. Kaindl, J. Falb, and R. Popp. Modeling
dow, depending on the widget types and the layouting rule of interaction design by end users through discourse
applied to the discourse relation. For example, the Alter- modeling. In Proceedings of the 2008 ACM
native relation used in our running example weighs both International Conference on Intelligent User Interfaces
branches equally, thus the actual resize behavior depends (IUI 2008), Maspalomas, Gran Canaria, Spain, 2008.
only on the widgets used.
2. J. Falb, H. Kaindl, H. Horacek, C. Bogdan, R. Popp,
In addition, the screen generation results in toolkit-specific and E. Arnautovic. A discourse model for interaction
event handlers that collect information from the input wid- design based on theories of human communication. In
gets, modify content objects provided by the application logic CHI ’06 Extended Abstracts on Human Factors in
appropriately or generate new ones. Afterwards, the event Computing Systems, pages 754–759, New York, NY,
handlers create input for the application logic and hand them USA, 2006. ACM Press.
over to the finite-state machine that implements the user in- 3. S. Kavaldjian, C. Bogdan, J. Falb, and H. Kaindl.
terface behavior, which then selects the next screen accord- Transforming discourse models to structural user
ing to the advancement of the dialogue. interface models. In Models in Software Engineering,
LNCS, volume 5002/2008, pages 77–88. Springer,
Finally, the screen generation process also internationalizes
Berlin / Heidelberg, 2008.
and localizes the generated GUI, e.g., it externalizes strings
and provides translation files with default suggestions de- 4. P. Luff, D. Frohlich, and N. Gilbert. Computers and
rived from the discourse model, the content types and the Conversation. Academic Press, London, UK, January
transformation rules. 1990.
5. W. C. Mann and S. Thompson. Rhetorical Structure
RELATED WORK Theory: Toward a functional theory of text
Our approach to GUI generation is similar to TERESA by organization. Text, 8(3):243–281, 1988.
Mori et al. [6]. Both start from high-level models, but our
discourse models have a focus on dialogues and seem to be 6. G. Mori, F. Paterno, and C. Santoro. Design and
even on a higher level than the task models in [6]. development of multidevice user interfaces through
multiple logical descriptions. IEEE Transactions on
The UI Pilot approach by Puerta et al. [9] is semi-automatic Software Engineering, 30(8):507–520, 8 2004.
by requiring the designer to specify tasks and a wireframe 7. I. Pederiva, J. Vanderdonckt, S. España, I. Panach, and
for the user interface. Afterwards, the tool can suggest wid- O. Pastor. The beautification process in model-driven
gets for each user interface element. This approach provides engineering of user interfaces. In Proceedings of the
more flexibility to the user interface designer than our ap- 11th IFIP TC 13 International Conference on Human -
proach, which allows fully automatic content presentation. Computer Interaction — INTERACT 2007, pages
411–425, Rio de Janeiro, Brazil, Sept. 2007. Springer
Pederiva et al. [7] describe a beautification process that
Berlin / Heidelberg.
helps a designer to improve a generated user interface via
a constrained user interface editor. This editor allows apply- 8. R. Popp, J. Falb, E. Arnautovic, H. Kaindl,
ing beautification operations to specific UI elements, result- S. Kavaldjian, D. Ertl, H. Horacek, and C. Bogdan.
ing in model-to-model transformation. Since our approach Automatic generation of the behavior of a user interface
involves content presentation specific to intentions, beautifi- from a high-level discourse model. In Proceedings of
cation should be less important for this part of UI generation. the 42nd Annual Hawaii International Conference on
System Sciences (HICSS-42), Piscataway, NJ, USA,
CONCLUSION 2009. IEEE Computer Society Press.
We address the problem of presentation of content from the 9. A. Puerta, M. Micheletti, and A. Mak. The UI pilot: A
domain of discourse according to its purpose in the current model-based tool to guide early interface design. In
state of the human-machine dialogue. This purpose relates Proceedings of the 10th International Conference on
to the intention indicated by the type of the communicative Intelligent User Interfaces (IUI’05), pages 215–222,
act that refers to propositional content to be presented. Our New York, NY, USA, 2005. ACM Press.
approach takes this type into account in the course of auto-
matic content presentation. In this way, it leads to generated 10. J. R. Searle. Speech Acts: An Essay in the Philosophy
user interfaces with content presentations specific to inten- of Language. Cambridge University Press, Cambridge,
tions. England, 1969.
4