36 A Multi-layered Context Modelling Approach for End Users, Expert Users and Programmers Sandra Trullemans Abstract WISE Lab Context awareness plays an important role in smart envi- Vrije Universiteit Brussel ronments and embedded interactions. In order to increase Pleinlaan 2 user satisfaction and acceptance, context-aware solutions 1050 Brussels, Belgium should be controllable by end users. Over the last few years strullem@vub.ac.be we have therefore witnessed an emerging trend of visual programming tools for context-aware applications based on simple “if this then that” rules. Unfortunately, existing solutions do not support the easy reuse of the “this” part Beat Signer in other rules. Further, the desired level of control varies WISE Lab among individuals. In order to let users choose the right Vrije Universiteit Brussel level of automation and control, we propose a multi-layered Pleinlaan 2 context modelling approach distinguishing between end 1050 Brussels, Belgium users, expert users and programmers. We report on our on- bsigner@vub.ac.be going development of the Context Modelling Toolkit (CMT) consisting of the necessary context modelling concepts as well as a rule-based context processing engine. We further discuss an initial design of the graphical user interface for the presented multi-layered context modelling approach. Author Keywords Context-aware systems; context modelling toolkit; end user. Copyright is held by the author/owner(s). AVI, June 07–10, 2016, Bari, Italy ACM Classification Keywords H.5.m [Information interfaces and presentation (e.g., HCI)]: Miscellaneous 37 Introduction ontology-based approaches are not ideal for dynamic con- It is well-accepted by the community that we have to achieve text modelling, given that changing the ontology at runtime a better synergy between context-aware systems and their introduces potential issues with conflicts and ontology in- users by providing control and intelligibility [8]. There is no tegrity. The Java Context Aware Framework (JCAF) [1] general rule about the necessary level of control since in- takes an object-oriented approach and forwards the rea- dividual users have varying preferences. As pointed out by soning of context rules to the application layer. This has the Hardian [6], there needs to be a balance between the level effect that context rules are spread over various applica- of control and automation. In order to provide end users tions which can lead to issues concerning conflict manage- some control, visual programming tools such as the Jigsaw ment and inconsistent application behaviour across client Editor [7] and iCAP [4] as well as commercial applications applications. In order to overcome this issue, JCOOLS [9] such as IFTTT1 have been developed for defining simple “if integrates JCAF with the Drools inference engine. Although situation then action” context rules. Note that in this case a JCOOLS is very promising, it lacks support for different ex- situation consists of one or more events such as the current pertise levels and reusable user-defined situations. Finally, temperature. Nevertheless, the complexity of context-aware the most well-known and used context-aware framework in systems is significantly increased in new forms of smart the Human-Computer Interaction (HCI) community is the homes. In such a complex environment, advanced mod- Context Toolkit by Dey et al. [2]. The framework applies elling support for end users is required since simple context a component-based software design where components rules cannot cover all desired smart home behaviour [10]. are represented by so-called widgets. Each widget pro- cesses information, from low-level data input to higher-level In this paper we explore the opportunities for advanced end abstract context information (i.e. detected situations). In user modelling of rule-based context-aware applications. order to foster intelligibility and offer control over context We propose a multi-layered context modelling approach rules, Dey and Newberger extended the Context Toolkit with which supports the seamless transition between different a middleware that includes the concept of situations [3]. levels of expertise. We further enable the reuse of situa- Whenever all the widgets of a given situation are active, tions defined by end users. Finally, we discuss an imple- the situation will invoke an action of the context services mentation of the Context Modelling Toolkit (CMT) enabling listening for it. While some user control is provided via the the practical use of our multi-layered context modelling ap- customisation of parameters, the reuse of situations in the proach. definition of other situations is not possible. Background We can conclude that there is a broad spectrum of context For decades context-aware frameworks have been devel- modelling tools ranging from context-aware development oped to facilitate the development of context-aware applica- frameworks to end-user tools. Nevertheless, existing con- tions. They often include support for context modelling and text modelling solutions do not foresee a seamless transi- reasoning. The SOCAM [5] framework is an example of an tion between the different levels of expertise of end users, ontology-based context-aware framework. Unfortunately, expert users and programmers and they do not support the 1 reuse of user-defined situations. https://ifttt.com 38 End User Multi-layered Context Modelling Approach fact or situation is only done when the template is used to (3) In order to facilitate the end-user context modelling process, create a concrete situation or context rule. In order to know Tool Rules we foresee the possibility to let end users create new situa- which object types are needed by the template, it contains tions such as ‘cooking’ by creating so-called situation rules a To Fill In part listing the required object types which which take the form of “if situations then new situation”. In are connected to the corresponding input variables of sit- (1) its simplest form, the IF side of a situation rule can just be uations or functions on the IF side. When a user fills in a a combination of context data as seen in existing visual template, they only have to provide the concrete instances Programmer programming tools, such as iCAP [4], in order to construct of the listed object types such as Person:Bob. Finally, the context rules. Note that this context data includes facts (real THEN side of a template defines the new situation. Since sit- world objects such as Person is Bob), events or situations. uations can have parameters, template designers can add Situations Rules (2) Besides simple situation rules, we also support the reuse these to the new situation by defining which properties of a of newly defined situations on the IF side. For example, template’s required object types have to be forwarded to the Facts Actions the situation rule “IF Person is Bob and Location is kitchen new situation. When a template is created, it can be reused THEN Bob is in the kitchen” takes the facts Person:Bob to define custom situations or context rules. For instance, and Location:kitchen as context data on the IF side the situation rule “IF Person is Bob and Location is kitchen Figure 1: Existing context to define a new situation. This new situation can then, for THEN Bob is in the kitchen” could be created by using the modelling approaches example, be reused in the situation rule “IF Bob is in the PersonIsAtLocation template by simply filling in the vari- kitchen AND stove is on THEN Bob is cooking” or in a con- ables Person:Bob and Room:kitchen and by naming the End User text rule such as “IF Bob is in the kitchen THEN turn radio new situation as “Bob is in the kitchen”. The new situation is Situation Filled in on”. In contrast to existing work, with our approach users do identified by its name. Similarly, the same template could be (7) Templates not have to re-define a situation each time they want to use used to define that Alice is in the living room. Rule (4) it in a different situation or context rule. The situation rules and templates were introduced to sup- Expert User We further make use of templates to simplify the definition port users with different levels of expertise as proposed by of situation and context rules. A template serves as a skele- Ur [10]. As illustrated in Figure 1, in existing systems facts Templates (8) ton for rules. The IF side can consist of multiple events, (e.g. Person is Bob), situations (e.g. cooking) and actions (6) situations or logical functions. Similar to programming con- (e.g. turning on the lights) (1) are usually predefined by a (5) Programmer cepts, a function evaluates some logical statements and programmer and can be applied by an end user to construct returns the result. In our multi-layered context modelling so- simple “IF situation THEN action” context rules (3). Fur- Situations Actions lution, functions only return boolean values. Furthermore, thermore, in context modelling solutions without end-user situations and functions can have a number of parameters. control, programmers also implement the desired context Facts Functions Rules For example, a function IsPersonInLocation evaluates rules (2). We extended the previous model with an inter- whether the given Person is in the specified Room and re- mediate layer for expert users and integrated the notions of Figure 2: Multi-layered context turns true or false. In contrast to the definition of a context custom and reusable situations as shown in Figure 2. Sim- modelling approach rule, a template does not specify a specific fact or situation ilar to existing systems, in our multi-layered context mod- for a parameter of a given type. The binding to a specific elling approach, programmers are responsible to provide 39 context data in the form of facts, situations and actions to Application & CMT Server Rule the end user (4). Besides the previously mentioned context Sensors Compiler Database (1,1) CMT-REST Interface hasOutput (1,*) data, they also have to provide functions (5) in order that compiled Observer to rule Template THEN-side expert users can create templates (6). End users might fill CMT Client Library in templates by using the provided context data. In addi- Translator Delegator db4o (1,1) (1,1) (1,*) X tion to the templates defined by expert users, end users can CMT-RMI Drools Com Interface also use some default templates such as an AND template Interface Event has Observer hasInput Situation hasAction to construct simple situation and context rules. The com- Listener Manager pleted templates define new situation or context rules (7). Drools (1,*) (0,*) (0,*) New situations are made available at the programmer layer IF-side Situation Action in order that they can be reused at the expert and end- user layer (8). Similarly, custom context rules flow from the Figure 4: CMT architecture (0,*) (0,*) end-user layer back to the programmer layer and can be hasInput used by programmers or other applications. This provides a currently support the Java Remote-Method Invocation (RMI) Event seamless transition between the three layers. and REST communication protocols. Since the two pro- has Function tocols require a different data exchange format, the CMT Context Modelling Toolkit server contains the CMT-RMI and CMT-REST components. (0,*) The proposed multi-layered context modelling approach has These components translate their input to CMT data model (0,*) Context (1,1) been implemented in the Context Modelling Toolkit (CMT). entities and generate the required output format when data Function Input CMT has been designed as a client-server architecture and is sent to the clients. Finally, the Java-based CMT Client (0,*) is implemented in Java. While the server side takes care of Library consists of a Translator component for trans- has (0,*) the context reasoning, clients such as sensors, third-party lating Java objects to CMT data model entities and to the Parameter Item applications and applications which allow end users to de- corresponding JSON formats, the Com Interface which fine new situations, provide the necessary context data to abstracts REST calls and the Listener component which the server. As shown in Figure 4, CMT consists of a db4o2 provides Java listeners for CMT. database backend and uses the non-persistent version of Property Fact the Drools 63 rule reasoning engine. In order to have a uni- Clients first have to register the type of situations or facts (0,*) (1,1) fied representation of the context data delivered by clients, they will send to CMT. Since Drools requires compiled Java has we have designed the conceptual CMT data model shown classes of the registered types, the Compiler component Property has Type FactType in Figure 3. The data model highlights the elements of the will compile these classes and add them to the classpath (0,*) previously described multi-layered modelling approach in at runtime. After registration, clients can add instances of (0,*) the form of an Entity-Relationship (ER) diagram. The dif- the registered types to the server. In case the instance is a (0,*) has Type Context Type ferent entities of the CMT data model serve as a common fact, the Delegator component sends it to the Database vocabulary used in any client-server communication. We Interface which inserts it into db4o. The Delegator 2 component further sends both, facts and situations, to the Figure 3: CMT data model http://www.mono-project.com/archived/db4o 3 http://docs.jboss.org/drools/release/6.1.0.Final Drools Interface where they are inserted in Drools and 40 Figure 5: Design of templates in the expert graphical user interface rules are re-evaluated. Furthermore, new templates are which again has been defined by a programmer and returns stored in the database. When a filled in template is re- true if there is no movement in a given room. Further, ceived, it is compiled to the Drools DRL rule format. The she adds a time constraint which indicates that it must be new rule is then inserted into the Drools knowledge base later than a specified time. When the logical statements are and all rules are re-evaluated. Note that our implementation defined, Alice has to add the object types to be provided of the rule compilation is a complex mechanism which takes when using the template. In our example, a Location and into account the full first order logic and provides in depth ItIsAfter type are added to the To Fill In part. Next, she error handling for client applications. When Drools detects connects the Location type to the inBed and noMovement some conflicting rules, the Event Manager forwards them parameters to indicate that the entered location has to be to the clients. In the future, we plan to further develop this passed to the two functions. Finally, she allocates the room component to provide some intelligibility. to the new situation on the THEN side by clicking the ar- row of the room field in the Location type. After saving the Graphical User Interfaces template, Alice can use the template to define “Sleeping” End users as well as expert users can interact with CMT situations such as that she or her son Bob sleep. via a graphical user interface (GUI). The expert GUI en- ables the design of templates as shown in Figure 5. In our The end user GUI shown in Figure 6 displays the created proof of concept scenario Alice designs a template to cre- template with a placeholder for the required Location in- ate “Sleeping” situations. First, she adds to the IF side the stance where Alice can enter Bob’s room. In addition, there inBed situation which has been defined by a programmer is a ItIsAfter placeholder. By clicking the ‘Enter Values’ and triggers when a pressure sensor in the bed detects button, Alice can enter the desired time (e.g. 20:00). She some pressure. Alice also adds the function noMovement then enters “Bob Sleeps” as the name for the new situation 41 on the Situation side. Similarly, she can enter her room and References specify 22:00 to define that she sleeps. Finally, the sleep- [1] Jakob Eyvind Bardram. 2005. The Java Context ing situations can be used to define context rules. Alice can Awareness Framework (JCAF) - A Service Infrastruc- add “Bob Sleeps” and “I Sleep” to the IF side of the AND ture and Programming Framework for Context-Aware template (i.e. default context rule template) and add the Applications. In Proc. Pervasive. action that the night light has to be turned on to the THEN [2] Anind Dey, Gregory Abowd, and Daniel Salber. 2001. side as shown in Figure 7. Note that similar to the creation A Conceptual Framework and a Toolkit for Supporting of templates for new situations, expert users can design the Rapid Prototyping of Context-Aware Applications. templates for advanced context rules. Human-Computer Interaction 16, 2 (2001). [3] Anind Dey and Alan Newberger. 2009. Support for Conclusion and Future Work Context-Aware Intelligibility and Control. In Proc. CHI. We are currently investigating the further design of the ex- [4] Anind Dey, Timothy Sohn, Sara Streng, and Justin Ko- Figure 6: End-user GUI to define new pert user and end user graphical user interfaces. Besides dama. 2006. iCAP: Interactive Prototyping of Context- situations usability and interaction enhancements, we are exploring Aware Applications. In Proc. PerCom. the integration of feedforward and intelligibility features. Fur- [5] Tao Gu, Xiao Hang Wang, Hung Keng Pung, and Da thermore, we plan to deploy CMT under a research license Qing Zhang. 2004. An Ontology-based Context Model in order that the HCI community can explore new opportuni- in Intelligent Environments. In Proc. CNDS. ties in designing user-centric context-aware applications. [6] Bob Hardian, Jadwiga Indulska, and Karen Henrick- sen. 2006. Balancing Autonomy and User Control in We have presented a multi-layered context modelling solu- Context-Aware Systems - a Survey. In Proc. PerCom. tion and the corresponding CMT data model as an innova- [7] Jan Humble, Andy Crabtree, Terry Hemmings, Karl- tive approach for end users to control their context-aware Petter Åkesson, Boriana Koleva, Tom Rodden, and applications. The presented solution contributes to the ex- Pär Hansson. 2003. "Playing with the Bits" User- isting body of work by enabling end users to define new configuration of Ubiquitous Domestic Environments. custom situations in the form of “if situations then new sit- In Proc. UbiComp. uation” at runtime and to reuse these situations in other [8] Brian Lim and Anind Dey. 2009. Assessing Demand situation and context rules. Ultimately, the proposed multi- for Intelligibility in Context-Aware Applications. In Proc. layered context modelling approach enables end users, UbiComp. Figure 7: End-user GUI to define new expert users as well as programmers to collaboratively de- [9] Jongmoon Park, Hong-Chang Lee, and Myung-Joon context rules by using the default AND template velop their context models. Lee. 2013. JCOOLS: A Toolkit for Generating Context- aware Applications with JCAF and DROOLS. Journal Acknowledgements of Systems Architecture 59, 9 (2013). The research of Sandra Trullemans is funded by the Agency [10] Blase Ur, Elyse McManus, Melwyn Pak Yong Ho, and for Innovation by Science and Technology in Flanders (IWT). Michael Littman. 2014. Practical Trigger-Action Pro- We would further like to thank Wouter Mensels and Brecht gramming in the Smart Home. In Proc. CHI. De Rooms for their work on an earlier version of the toolkit.