In previous work [
More recently, in the OntoUCP1 project, we wanted to work with models that are more understandable to and possibly more easily to build for humans. Therefore, we studied several theories of human communication from various elds. Based on insights from some of these theories, we focus on high-level speci cations of discourse in the form of models. These models specify discourse in the sense of dialogues, where monologues are embedded and connected.
From our previous work, we inherit the use of communicative acts (and references to domain knowledge). Communicative acts are derived from Speech Act Theory and express intentions in the sense of desired e ects on the environment.
By integrating communicative acts with some results from Rhetorical Structure Theory (RST) and Conversation Analysis, we developed a new discourse metamodel. The meta1OntoUCP (A Uni ed Communication Platform both for Machine-Machine and Human-Machine Interaction based on Ontologies), partially funded by the FIT-IT Program of the Austrian FFG as project number 809254/9312. We also acknowledge the ( nancial) support of the PSE division of Siemens AG Osterreich. model de nes what the discourse models should look like in our approach.
So, we strive for high-level modeling of discourse, including dialogues. Such a discourse model is inspired by human communication and serves as an interaction design for a traditional information system. Currently we do not support the generation of UIs with direct manipulation.
From such an interaction design, user interfaces for several devices are to be generated automatically. Since we knew already how to generate them from a kind of abstract UI model, we strived for generating an abstract UI from our new interaction design models. We explain our model-driven transformation approach on the basis of our metamodels and self-de ned transformation rules. 2.
TRANSFORMATION APPROACH
Our approach to fully automated UI generation is a twostep process illustrated in Figure 1. Model-to-model and model-to-code transformations are necessary to transform a discourse model to an abstract UI model and the abstract UI model to multiple concrete UIs for diverse platforms. In the following, we will explain the input (discourse model structure), the output (structure of the abstract UI model) and the transformation rules for the model-to-model transformation step.
Our discourse models use a self-de ned Domain Speci c Language (DSL) for specifying the classes of possible dialogues or interactions between the human and the machine. The abstract syntax of the DSL is based on the metamodel shown in Figure 2, which illustrates the used concepts. Every discourse is composed of a tree where leaf nodes are Communicative Acts and inner nodes are Rhetorical Relations based on RST. The conceptual UML class diagram Discourse Model relates > +rootNode 1 shown in Figure 2 is not as restrictive as our interpretation, since it allows to create other kinds of graphs besides tree structures. The association class is needed to model the inserted sequence.
The Communicative Acts are used to model the intention of a communication and refer to elements of the domain of discourse. Figure 3 shows a selection of the most important Communicative Acts used in our approach. Two corresponding Communicative Acts, like O er and Accept, form a sequence, which is called Adjacency Pair. The Adjacency Pairs build up the dialogue structure.
Node Communicative Act + contentCondition: + degreeOfStrength: Assertion
Directive
Commissive Informing
Answer
Question
Request
Accept
Offer isadjacent to isadjacent to isadjacent to
The Rhetorical Relations are used to connect
Communicative Acts or Rhetorical Relations with each other. They
represent the dependencies between single interactions of
dialogues. More detailed information about our discourse
metamodel can be found in [
Figure 4 shows a small part of an online shop discourse model, that is typical for discourse models and which we will use as a running example throughout the paper. The example describes the interaction between the user and the online shop. The nucleus branch N of the Background relation conveys the main interaction sequence. The online shop system o ers a list of product categories to the user. The user accept s one of them. During the o ering process the satellite branch S provides background information about the product categories to the user. This part of an online shop discourse model gets transformed to the abstract UI model shown in Figure 5 by applying the rules Adjacency Pair, O er-Accept and Informing in the listed order. Details on each rule are described below.
The abstract UI model is basically a tree representing the UI structure. It is not completely independent of the target device, since the device's real estate is considered for building up the abstract UI structure. However, our abstract user interface is completely independent of the considered UI toolkit (e.g. Web, Java Swing, etc.). This tree structure will be transformed to a toolkit-speci c concrete UI. The concepts which are used in an abstract UI model are speci ed in the abstract UI metamodel shown in Figure 6. The most important concept of the metamodel is the Widget class. It is specialized into two functional categories, the OutputWidget s which have the function of only presenting information and the InputWidget s which have the function of presenting and gathering information from the user. Thus, they are actually input/output widgets, but Figure 6 puts the focus on gathering information.
Style Textbox +style 0
Widget +elements ++ vniasmibele:: inintt[[00..11]]{{oorrddeerreedd}} 0. * +widgets +sc0r.e*ens 0. *
Panel + layout: int [0. 1] {ordered}
The main issue that we address in this paper is how to transfer models as exempli ed in Figure 4 to user interface models at the abstract widget level. In particular, it means a transformation from a mainly declarative model to a user interface featuring procedural behaviour.
The general principle behind our approach is that the abstract UI model is made up of \presentation" units that are set visible when the logic of the interaction with the user so requires. Once this principle is established, our problem can be speci ed as follows:
Given a discourse tree with communicative acts as
leafs, generate the possible set of presentation units,
and the transitions between these presentation units.
Since a presentation unit has to be a coherent
discourse, it corresponds to a subtree of the overall
discourse tree. As such, we call this problem the discourse
tree partitioning problem. This problem and a solution
to it is described in [
Given a presentation unit as a discourse subtree, generate an abstract UI model based on heuristic rules. Since this e ectively \pre-renders" a discourse tree into an abstract UI model, we call this problem the prerendering problem.
In the abstract UI model, a complete tree or subtree with a Panel as root element represents a presentation unit. Hence, a complete abstract UI model can be a forest. Our example in 4 represents exactly one presentation unit that corresponds to the tree shown in Figure 5.
Figure 7 illustrates that the transformation is ful lled by mapping elements of the discourse metamodel to elements of the abstract UI metamodel. Both metamodels are based on the Ecore2 meta-metamodel. Transformation languages like the ATLAS Transformation Language (ATL) support this transformation concept. At the same time a state machine is derived from the discourse model which controls the sending and receiving of Communicative Acts.
Ecore Instance of
Instance of Discourse Metamodel mapping Instance of
Instance of Discourse Model transformation
After having introduced the general transformation principles, we concentrate only on the pre-rendering problem in the remainder of this paper and introduce some rules that are speci c to certain structural patterns occurring in the discourse models. We have found many such patterns during our modeling experience, and we continue to nd new 2Essential MOF like core meta model of the Eclipse Modeling Framework (http://www.eclipse.org/emf/) ones. Due to limited space, we only exemplify ve rules which we believe illustrate the principle.
Heavy Background Rule: Figure 8 shows a rule for a \Heavy Background" relation, relating a large satellite subtree with a nucleus subtree. The \nuclear" part is rendered directly, but if there is no space for its background information (which is presumed to be heavy for this rule to apply), the background information is rendered in a separate presentation unit, to which a link is presented.
Discourse Model Background
S N heavy subtree subtree
Concrete UI
Light Background Rule: Figure 9 shows another rule for a Background relation on an interface. The satellite is rendered on the right side of the presentation unit, while the nucleus occupies the left area. In accordance to the rule above, the \most nuclear part" takes the interface space that is of highest surface and most central to the user focus. This rule is used in our example to generate the basic tree structure of Figure 5. The Light Background Rule can also be localized (adapted), e.g., for cultures that write from right to left, where it may be more suitable to place the satellite at the left side.
Discourse Model Background
S N light subtree subtree
Panel ( ow layout)
Widget (nucleus) Widget (satellite)
Concrete UI
x widget for widget for nucleus N satellite S (in most (in most cases a cases a panel) panel)
Adjacency Pair Rule: Every adjacency pair is transformed to a Panel element of the abstract UI model containing widgets according to the actual related communicative acts. In our example, the rst panel on the second level in Figure 5 results from the O er-Accept adjacency pair. If a communicative act does not take part in an adjacency pair, as it is the case with the Informing in Figure 4, a Panel element is also created for the communicative act.
O er-Accept Rule: Every O er -Accept adjacency pair is transformed either to a Button element or to a ListWidget element containing a Button element depending on the number of content elements o ered. Because our example o ers more than one product category, the ListWidget element is needed to model an unde ned number of categories. Since the acceptance of an O er requires a user action, a Button element is used.
Informing Rule: Every Informing communicative act is transformed either to a Label element or to a ListWidget element containing a Label element, depending on the number of content elements o ered. This rule assumes that the information will be forwarded in textual form, otherwise, e.g., a PictureBox or Audio element will be used. In the online shop example, Label elements are used.
More detailed information about the automatic generation
that is used as a basis for this approach can be found in [
Model-based UI design methods developed and published
in the nineties including OVID [
In contrast, UI Frameworks like XUL3 (XML User Interface Language) are able to generate UIs automatically but they rely on UI models at the abstract widget level, which is on a lower level than our discourse models.
An advanced approach to specifying multi-device user
interfaces based on task models instead of discourse models
is presented in [
Florins et.al. describe in [
Botterweck shows in [
In this paper, we present a new approach to generating abstract user interface models by applying model-driven transformations to discourse models. Our discourse models are derived from results of human communication theories, cognitive science and sociology and are used for specifying interaction design of human-computer interaction of information systems. Thus, they contain additional metainformation, like the intention of an interaction, which allows us to dene sophisticated pre-rendering rules to transform the discourse models to abstract UI models. Our transformation takes already device constraints into account to generate a UI structure well suited for the target device, but the resulting abstract UI models are still independent of UI toolkits. Taking this together with our previous work on automatically generating concrete UIs, this paves the way for automatic generation of concrete UIs from our new interaction design models.