Generating High-Level Interaction Models out of Ontologies Dominik Ertl, Hermann Kaindl, Edin Arnautovic, Jürgen Falb, and Roman Popp Vienna University of Technology Institute of Computer Technology (ertl, kaindl, arnautovic, falb, popp)@ict.tuwien.ac.at ABSTRACT The remainder of this paper is organized as in the follow- Generating user interfaces out of semantic models is still ing manner. First we give a brief background on our previ- an issue because of the semantic gap between ontologies ous work relevant for this paper, and compare it with some and user interfaces. We bridge this gap through semantic of the related work in the field. Then we present our ap- model-driven development. More precisely, we show how proach for generating interaction models out of (annotated) to automatically generate high-level interaction models (in ontologies. This approach contains two parts: the generation the form of communication models representing discourses) of discourse models representing the flow of communica- out of (annotated) ontologies, using model-transformation tion between the user and the computer, and the generation rules. From these discourse models, user interfaces can be of domain-of-discourse models representing what they “talk generated (semi-)automatically. about”. Finally we conclude and provide an outlook of our future work in this direction. INTRODUCTION BACKGROUND AND RELATED WORK The most important elements of any interactive system are Our previous work focused on manual modeling of interac- the information it contains and the user interface through tion designs [1], where even end users created interaction which this system communicates with its users. The infor- designs in the form of discourse models using the graphi- mation may be represented with (formal) semantic models cal editor developed for this purpose. These discourse mod- (e.g., based on ontologies), and the user interface is typi- els are based on several theories of human communication cally created manually on top of such models. This requires [2]. The key parts of our discourse models are Commu- a lot of effort, especially if these models are modified and nicative Acts as derived from speech acts [11], Adjacency the user interface has to be adapted manually. Pairs adopted from Conversation Analysis [5], and RST rela- tions inherited from Rhetorical Structure Theory (RST) [6]. In a specific category of interactive systems, such as product Communicative Acts are semi-structured messages carrying recommendation systems, reservation systems or shopping the intention (e.g., asking a question or issuing a request) applications, the underlying (semantic) model may strongly and represent basic units of language communication. Ad- influence the behavior of the systems and, therefore, also the jacency Pairs are sequences of talk-turns that are specific to interactions to be implemented through the user interfaces. human (oral) communication, e.g., a question should have a For this category of interactive systems, we address the se- related answer. RST relations specify relationships among mantic gap between underlying ontologies and user inter- text portions and associated constraints and effects, and are faces. We make use of our discourse models [1, 3] for bridg- organized in a tree structure. In our work, we use RST for ing this gap. In this course, a discourse model and a domain- linking Adjacency Pairs of Communicative Acts and further of-discourse model together serve as a high-level interaction structures made up of RST relations. We have also included model and, as such, as a kind of “intermediate language” be- procedural constructs, to provide means to express a partic- tween the ontology and the user interface. In addition, such a ular order during discourse execution, to specify repetitions model can even be used for the (semi-)automatic generation or conditional execution of different discourse parts. Since of a user interface [3, 10]. such discourses cast the communication between a human and the computer on a high level, abstracting from technical details, they may even be created without any programming knowledge and experience. Instead of our discourse models, ConcurTaskTrees from Pa- terno et al. [7] may be used for bridging the semantic gap be- tween ontologies and user interfaces. ConcurTaskTrees fa- cilitate modeling tasks, that are being transformed into a user interface. Our discourse models focus more on the commu- IUI SEMAIS Workshop 2011. Copyright is held by the owner/author(s) 1 Domain-Specific Domain-Specific Constraints for Constraints for Discourse Domain-of- Model Discourse Model ONTOLOGY COMMUNICATION MODEL Annotated Generic Discourse Model Ontology Discourse Model Rule-Based Rule-Based User Transformation Transformation (without content Interface description) Step 1 Set of Individuals Domain-of-Discourse Model Rule-Based Transformation Step 2 Figure 1. Transformation from ontology to communication model. nication, and they can be used for machine-machine com- tains the knowledge of the given domain, the annotations munication as well [9]. According to our best knowledge, contain meta-knowledge, e.g., the priority of a given piece this has not been done with ConcurTaskTrees. We are also of knowledge with respect to the Product Advisor to be im- not aware of any approach for generating ConcurTaskTrees plemented. The result is a communication model consisting out of ontologies. of a discourse model and a domain-of-discourse model. UsiXML [4] is an XML-based specification language for We use a GoodRelation2 ontology for digital cameras as a user interface design. It allows describing a user interface at basis. In Figure 2, we depict selected parts from the Digi- different levels of abstraction, from high-level task models cam GoodRelation ontology. The top concept Thing is spe- to the concrete code of a user interface. So, it provides an al- cialized by the concepts ProductOrService and DomainSeg- ternative approach to ConcurTaskTrees. Also for UsiXML, ment. ProductOrService is further specialized by the Digi- we are not aware of any approach for generating UsiXML cam concept. DomainSegment groups together properties of models out of ontologies. a ProductOrService that have a semantic relation with each other. Paulheim and Probst [8] present a survey about ontology- enhanced user interfaces. They point out that ontologies Our ontology contains additional annotations that describe can be used to improve interaction possibilities, and our ap- characteristics of certain datatype and object properties with proach addresses such a possibility. respect to the intended Product Advisor. For example, the annotation priority specifies how important for the Product FROM ONTOLOGIES TO INTERACTION MODELS Advisor a specific object or datatype property is compared Now let us present our approach to automatically transform- to other properties. These priorities are a distinguishing fea- ing an ontology to a high-level interaction model in the form ture when the transformation process applies the transfor- of a specific communication model by using model trans- mation rules. The priority is an integer value between 0 (low formations. We focus on a small part of an ontology and priority) and 100 (high priority). The priorities allow the its corresponding transformations to generate the interaction transformation process to decide which datatype and object model of a Product Advisor for digital cameras as a running properties are of interest for the discourse and the domain- example. The Product Advisor is designed to ask questions of-discourse. about desired properties of a digital camera to be bought. In the first step, a set of model-transformation rules matches parts of the ontology (including its individuals) and trans- Overall Transformation Approach forms them automatically into corresponding parts of a dis- In Figure 1, we provide an overview of the transformation course model (see the middle part of Figure 1). These process, which consists of two steps for generating a com- transformations are subject to domain-specific constraints munication model from an annotated ontology. We start explained in detail below. The discourse model generated from such an ontology represented in OWL1 (illustrated in in this step represents only the generic communication flow the left part of Figure 1). While the ontology per se con- 1 2 Last visited on December 10, 2010: http://www.w3.org/ Last visited on December 10, 2010: http://www. TR/owl2-overview/ heppnetz.de/projects/goodrelations/ Copyright is held by the owner/author(s) 2 [1] [2] [1] [3] [3] Class example: [2] Instance example: Figure 2. Selected parts from Digicam ontology. and is incomplete since the content of its communicative acts mentClusterRule illustrated in Figure 3. It matches the con- does not (yet) refer to the content of the communication (the cept DomainSegment in the ontology, which has several in- domain-of-discourse model). dividuals. For example, the digital camera ontology has the domain segments EnergySupply, LensFeatures, Ports, In the second step, our model-transformation approach etc. All datatype and object properties in the ontology that transforms the individuals of the ontology and their con- are related to a DomainSegment via the object property be- crete datatypes and object property values into a domain-of- longsToDomainSegment are of interest for our transforma- discourse model. Here we apply domain-specific constraints tion process. for a domain-of-discourse model. In effect, this step defines the content of the communicative acts so that the discourse The rule DomainSegmentClusterRule creates a cluster of model refers to the domain-of-discourse model. The con- questions for all object and datatype properties that belong to tents of communicative acts in the case of a Product Advisor the same domain segment. A cluster groups questions that are concrete question and answer texts that link to elements hold a semantic relation (e.g., the ports USB and FireWire of the domain-of-discourse model. belong to the DomainSegment Ports). Such a definition of a cluster results in a Joint RST relation of a discourse (see the From Ontologies to Discourse Models right part of Figure 3). The datatype and object properties Our transformation approach applies several rules to create are transformed into question/answer pairs that are branches a discourse and a domain-of-discourse model out of the on- of the Joint relation. In addition to the rule presented above, tology. Each rule application can be constrained by domain- the following domain-specific constraint applies: Each prop- specific constraints, that are externally configured. We have erty needs a minimum priority value (e.g., 20) to be included a logical rule chain (by using Operational Query/View/- in a cluster. The minimum priority value is configured a pri- Transformation3 (QVT)) defining the application order of ori in the domain-specific constraints. the rules. In principle, a rule that is applied later in the trans- formation process can influence the outcome of a rule that is After the DomainSegmentClusterRule has been applied, all applied sooner. In the following, however, we describe two properties with a minimum priority are grouped in the dif- independent rules applied in the transformation process, the ferent clusters. For example, USB belongs to the segment DomainSegmentClusterRule and the SingleQuestionRule. Ports. Now a rule applies that combines all Boolean prop- erties (like USB) of one domain segment into one question, The first rule explained as an example is the DomainSeg- for optimizing the interaction with the Product Advisor. The 3 left part of Figure 4 shows the datatype property USB, which Last visited on December 10, 2010: http://wiki. represents the USB port of a digital camera having the prior- eclipse.org/M2M/Operational_QVT_Language_ %28QVTO%29 ity 85. For this USB property, the SingleQuestionRule takes Copyright is held by the owner/author(s) 3 Domain-Specific Constraint Domain-Specific Constraint Create cluster for domain segments that have properties with Create question/answer pair ONLY priority > 20 if priority of property > 70 Knowledge of Related Part in Property in Ontology Related Part in DomainSegments in Ontology Discourse Model Discourse Model Domain- Segment- SingleQuestion Cluster for each Rule Rule domain segment http://rdf4ecommerce.esolda. ... com/digitalcamera#USB Figure 3. Transformation with DomainSegmentCluster Rule. Figure 4. Transformation with SingleQuestion Rule. effect now. Due to its high priority value (and importance), where the Product Advisor asks the user about his or her USB becomes a single ClosedQuestion-Answer pair again. preferences (e.g., price range) regarding the camera price This adjacency pair, shown in the right part of Figure 4, be- (shown at the top of Figure 5). The Product Advisor should comes also part of the generated discourse model. only ask for relevant product properties and their values. For example, the prices of all cameras should be within the price As shown with these example rules and their applications, range offered for selection. So, the set of individuals of the for each selected property a corresponding question is be- products is used to generate the possible contents of the com- ing generated for the Product Advisor, since this property is municative acts, e.g., to determine their price range. considered important for the selection of a camera. While the ontology specifies what exists in the domain, the process So, for the content of each question in the discourse model, being implemented in the Product Advisor contains related a unique datatype representing the product property is gen- questions. This semantic gap is bridged by our approach in erated in the domain-of-discourse model. Figure 6 shows the context of the given application. a small excerpt of such a generated domain-of-discourse model. For product properties representing numbers (e.g., We show an excerpt of a yet incomplete discourse model in price), only the minimum and maximum values are relevant Figure 5, that is the result of the first transformation step de- for the Product Advisor (e.g., to generate a slider in the fi- picted in Figure 1. The contents of the communicative acts nal UI for selecting the preferred value between the mini- are URIs that refer to datatype or object properties in the on- mum and maximum). These values are stored together with tology. A Joint relation combines one Adjacency Pair and the generated datatype. The left part of Figure 6 shows the the Background relation connecting two more Adjacency datatype of the price property realized by a Float number. Pairs. In this example, these properties have a high enough The minimum and maximum values are displayed as an an- priority, so that they have to be grouped together in a spe- notation in a note below the datatype. For product properties cial cluster at the beginning of the recommendation process representing Boolean values, the concrete individuals do not of the Product Advisor. The first question gathers informa- have to be searched for possible values, of course. As an tion on the price range, defining the minimum and maximum example of such a Boolean datatype, the USB datatype is price that the user is potentially willing to pay. The second shown in the middle of Figure 6. For all other properties, an question elicits the interest for a USB port on the digital cam- Enumeration datatype is generated for storing all possible era. This Boolean question is modeled as a closed question. values. The right part of Figure 6 shows the Enumeration Moreover, there is an RST relation Background intended to type generated for the producer datatype. The values of the optionally inform the human user on additional details about enumeration are derived from the set of all camera producer the subject matter, e.g., more information on USB. individuals in the given ontology. From Ontology to Domain-of-Discourse Model The applications of these transformation rules can be in- The second step in the transformation from ontologies to fluenced by domain-specific constraints specific for the our communication models is to generate the domain-of- domain-of-discourse model. For example, if no values for discourse model. This model represents the content of the a specific property exist in the set of individuals in the on- communication, more precisely the content of the commu- tology or if all of them are same (e.g., if all cameras have nicative acts within our discourse models. For example, a USB interface), then the content of the question would be the digital camera’s property hasCurrencyValue (represent- empty, so that the whole question is deleted from the dis- ing the price of the camera) is the content of the question course model. Copyright is held by the owner/author(s) 4 REFERENCES 1. C. Bogdan, H. Kaindl, J. Falb, and R. Popp. Modeling of interaction design by end users through discourse modeling. In Proceedings of the 2008 ACM International Conference on Intelligent User Interfaces (IUI 2008), Maspalomas, Gran Canaria, Spain, 2008. ACM Press: New York, NY. 2. J. Falb, H. Kaindl, H. Horacek, C. Bogdan, R. Popp, and E. Arnautovic. A discourse model for interaction design based on theories of human communication. In Extended Abstracts on Human Factors in Computing Systems (CHI ’06), pages 754–759. ACM Press: New York, NY, 2006. 3. J. Falb, S. Kavaldjian, R. Popp, D. Raneburger, E. Arnautovic, and H. Kaindl. Fully automatic user interface generation from discourse models. In Proceedings of the 13th International Conference on Intelligent User Interfaces (IUI ’09), pages 475–476. ACM Press: New York, NY, 2009. 4. D. Faure and J. Vanderdonckt. User interface extensible markup language. In Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS ’10), pages 361–362. ACM Figure 5. A cluster of questions with high priority. Press: New York, NY, 2010. 5. P. Luff, D. Frohlich, and N. Gilbert. Computers and Conversation. Academic Press, London, UK, January 1990. 6. W. C. Mann and S. Thompson. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281, 1988. Figure 6. Excerpt of Domain-of-Discourse Model. 7. F. Paterno, C. Mancini, and S. Meniconi. ConcurTaskTrees: A diagrammatic notation for CONCLUSION specifying task models. In Proceedings of the IFIP To ease and speed up the development of ontology-based TC13 Sixth International Conference on interactive systems, the automatic generation of their user Human-Computer Interaction, pages 362–369, 1997. interfaces would be advantageous. However, due to differ- ent perspectives as well as technical and conceptual foci of 8. H. Paulheim and F. Probst. Ontology-enhanced user ontologies used in such systems, the generation of user in- interfaces: A survey. Int. J. Semantic Web Inf. Syst., terfaces directly from ontologies would be hard. We use a 6(2):36–59, 2010. high-level interaction model in the form of a communication 9. R. Popp. Defining communication in SOA based on model based on discourses as an intermediate language. In discourse models. In Proceeding of the 24th ACM this paper, we explain the automatic generation of such mod- SIGPLAN Conference Companion on Object Oriented els out of (annotated) ontologies, and taking application- Programming Systems Languages and Applications specific constraints into account. (OOPSLA ’09), pages 829–830. ACM Press: New York, NY, 2009. From such communication models, user interfaces can be generated (semi-)automatically, as we have previously 10. D. Raneburger, R. Popp, S. Kavaldjian, H. Kaindl, and shown already [3]. For small devices, even fully automatic J. Falb. Optimized GUI generation for small screens. In generation leads to usable interfaces through special opti- LNCS Volume on Models in Software Engineering: mizations of the use of the constrained space [10]. Workshops and Symposia at MoDELS 2010. Springer, 2011. Acknowledgment 11. J. R. Searle. Speech Acts: An Essay in the Philosophy This research has been carried out in the SOFAR project of Language. Cambridge University Press, Cambridge, (No. 825061), funded by the Austrian FIT-IT Program of England, 1969. the FFG and Smart Information Systems GmbH. Copyright is held by the owner/author(s) 5