Towards the user confidence in sensor-rich interactive application environment Ilkka Niskanen Julia Kantorovitch Vildjiounaite Elena Software Architectures and Platforms Software Architectures and Platforms Context-awareness and service VTT - Technical Research Center of VTT - Technical Research Center of interaction Finland Finland VTT - Technical Research Center of Oulu, Finland Espoo, Finland Finland Ilkka.Niskanen@vtt.fi Julia.Kantorovitch@vtt.fi Oulu, Finland Elena.Vildjiounaite@vtt.fi ABSTRACT Cooking guide is an example application that is heavily depended The recent advances in sensor-rich, ambient computing on dynamic context information [17]. The Cooking Guide may environmets have led to a situation in which ordinary users may run in a touch-screen device, for example, and it helps the user express negative reactions when they feel that their behaviour is during meal preparation by providing detailed, step-by-step being monitored and analysed by technological systems which explanations. Cooking Guide adapts its behavior according to the they do not understand. Cooking guide is an example application context information (e.g. available smart appliances augmented by that is heavily depended on dynamic context information and various sensors, output devices, and user's cooking experience) adapts its behavior according to the context data. The thus each step can be potentially performed in a different way. VisuMonitor approach, described in this study, supports the users Cooking guide is a true effort towards the contextual rich dynamic of Cooking Guide by providing visualization views that show the proactive knowledge-based application. Proactive knowledge base proceeding of cooking processes and also explains the is built from the sensors augmenting the objects in use, functionality and behavior of the system during different cooking surrounding devices and user profiles. Sophisticated data mining activities, thus improving user awareness, technology acceptance algorithms, rule based mechanisms and user model learning and user education. VisuMonitor utilizes semantic technologies in techniques facilitate contextual awareness and adaptability the modeling of workflows, which facilitates data integration and towards the assistance and end user ambient support. enables more efficient work progress monitoring and The importance of explanation interfaces in providing system visualization. transparency and thus increasing user acceptance has been well recognized early in a number of fields such as expert systems [2], ACM Classification Keywords intelligent tutoring systems [3], office documents user assistance H.1.2 User/Machine Systems: Human factors. systems [18], data exploration systems [4], and recommendation systems [5][6][7]. In relation to ubicomp environment, the Author keywords necessity to support the features that aim at supporting user Context awareness, proactive knowledge, sensors, user education, acceptance by making system‟s reasoning process visible and semantic technologies, user education, data visualization insight of the system comprehendible has been acknowledged only recently [1][8][9], while prototyping of such feature is still in General Terms its infancy. For our knowledge only work by K.Cheverst [9] has Design, Human factors practically addressed the transparency and comprehensibility of the system leveraging the power of explanation user interfaces. 1. INTRODUCTION There the Intelligent Office System can learn a given user When evaluating the ideas of sensor-rich, ambient computing situation to use the inferred rules and support appropriate environments to ordinary users, non-technical people, in proactive behaviour such as e.g. turning on/off the fun or particular, express anxiety when they find themselves in opening/closing window under appropriate conditions. On the situations, where they feel that their behaviour is being monitored same time, the system enables the user to explicitly scrutinise and and analysed by technological systems which they do not override the „if-then‟ rules held in user model. If the user wishes understand [1]. Such negative reaction to applications which use to enquire why the system is performed in a certain way, the sensing technology sets a challenge which needs to be addressed. appropriate button can be pressed in order to view a window such Technology must be regarded as helpful rather than threatening. as the one shown in Figure 1. We believe that if users perceive themselves to understand and to have control over their personal application, they will be more likely to trust applications which use sensing data. Accordingly a knowledge-based system should be able to explain its reasoning, and rules used to justify its conclusions to be accepted by users. Figure 1. Scrutinising the rules behind the prompt me text Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA knowledge domain and are able to describe explicitly the content However manually acquired textual explanations may not be and semantics of heterogeneous data sources to support always sufficient especially in the cases where the context of the integration, processing and further new knowledge discovering application and user is rapidly varying such as in cooking which is tasks. The utilization of semantic technologies provides also an a creative process with continuously changing cooking situation, intelligent way to define and use rules that guide the behavior of appliances in use and products features. This sets the additional the application. challenges on the design of the user interface. Moreover, the The use of semantic technologies is especially pertinent with such purpose of the system plays an important role in defying of applications as the VisuMonitor where complex and respective elements that influence system acceptance. When heterogeneous data is gathered from multiple sources and it has to interacting with work and task-oriented systems, the perceived be presented to the users in a comprehensive way. The annotation usefulness is more important. In contrast when interacting with of the data using ontologies and concept taxonomies will allow hedonic systems that are aimed at fun and pleasure (as cooking users to better perceive the relationships between different guide mostly does) the perceived enjoyment is more desirable in concepts. Additionally, by utilizing reasoning mechanisms achieving user acceptance [10]. provided by semantic technologies, the data can be better clustered and targeted to the particular users. VisuMonitor supports the users of Cooking Guide in two ways: 2. VISUALIZATION showing practical information related to the cooking process itself Looking for the means to fulfil the above discussed requirements, (the proceeding of the cooking process from one step to another, we believe that visualization based aids which are intuitive and the information provided by different sensors, the usage of easily customizable, may help the user to link the complex different devices etc.) and providing explanations related to the contextual world of physical services residing in the environment, functionality and behavior of the cooking guide system (for reasoning of the system and human mind. Visualization of data example why the cooking guide application decided to change makes it possible to obtain insight into these data in an efficient from speech to textual guidance in some point of the cooking and effective way, thanks to the unique capabilities of the human process etc.). VisuMonitor may also educate the user by visual system, which enables us to detect interesting features and explaining why the particular recipe/ingredients are recommended patterns in a short time [11]. In particular with recent advances in e.g. due health reasons, diseases, dietary, recent blood test, etc. computer graphics, visualization is able to benefit the sense of Different cooking processes executed with Cooking Guide are wonder connected with the application presenting the content of modeled as workflow descriptions. Cooking Guide is tightly the data in a completely innovative and quickly comprehendible integrated with a Workflow engine tool, which manages the form. workflows that are executed in cooking processes. The executable Currently existing approaches to visualise the rules of the system workflows are described with an XML-based serialization format are targeting mainly application developers [12][19] or data known as XPDL [20] (XML Process Definition Language). exploitation professionals [13][14][15][16]. Accordingly common XPDL is a common format supported by a number of editing tools for the developed techniques is that they rather support the and process execution engines. XPDL workflow models are categorization, browsing and management of potentially complex standardized representations of one or more workflows. The rule bases, while the ground to the world of physical devices and workflow engine plans, checks and manages the execution and context attractiveness, fast assimilation and intuitive visualization states of workflows. If an activity is finished, it is e.g. responsible important for non-technical end user are left beyond. for checking outgoing conditions of transitions and deciding if the transitions should be activated or not. Workflow engine utilizes also context information extensively. Besides of information 3. VISUMONITOR – TOWARDS BETTER source, the engine uses context data to adapt to the situation, to trigger activity transitions and to influence the control flow. USER AWARENESS In this position paper we present a visual monitoring approach – VisuMonitor communicates with Workflow engine to retrieve the VisuMonitor, which is currently under development. VisuMonitor necessary information needed for workflow visualizations. In is directed for the end-users of different context-aware addition to static and dynamic workflow representations, applications and aims towards a better user awareness, technology VisuMonitor provides also other workflow related information to acceptance and user educating. The approach enhances the the users. It may show, for example, the different resources sharing of knowledge by integrating information from multiple, needed to complete a workflow activity or information related to heterogeneous sources and providing interactive views to this functionality and behavior of the cooking guide system. By data. To enable the integration of heterogeneous data sources, integrating the data acquired from Workflow engine and Cooking VisuMonitor utilizes semantic technologies and especially Guide, VisuMonitor is able to produce a global view of a cooking ontologies that facilitate shared and common understanding of process. Permission to make digital or hard copies of all or part of this work for personal or classrom use is granted without fee provided that copies are 3.1 Compositional structure not made or distributed for profit or commercial advantage and that The compositional structure of the VisuMonitor infrastructure is copies bear this notice and the full citation on the first page. To copy shown in Figure 2. otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Conference’10, Month 1–2, 2010, City, State, Country. Copyright 2010 ACM 1-58113-000-0/00/0010…$10.00. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA Figure 3. The dynamic workflow visualization Figure 2. The compositional structure According to the sequence diagram above, the user may first create a client in order to start monitoring workflows. (1) - Workflow visualization and monitoring, which is a VisuMonitor connects to Workflow engine and retrieves the core of the tool. This component provides mechanisms workflows that are currently hosted by the engine. The user may for visualizing workflows and other related information. then select the workflows that he/she wants to visualize and (2) - Semantic library represented by ontologies, which will monitor. Subsequently, the monitor communicates with Workflow contain the workflow related knowledge base. This engine and subscribes as a listener to the selected workflows. As a component contains semantically modelled workflow result, Workflow engine notifies the monitor each time something descriptions that are visualized with the tool. It may also noticeable happens in the execution of the selected workflows (i.e. contain other semantically modelled information, such a transition from one activity to another or some as context and sensor data, rules and other system exception/anomaly occurs during the execution). Each time functionality data and information about different VisuMonitor receives a change notification it updates the resources that are related to workflows. visualization view accordingly. VisuMonitor may also query some (3) - Ontology management tools, which allow to query and additional, workflow related information from the Cooking Guide update ontology instances. Some existing open source application. The monitor may acquire, for example, such software like Jena and OWL-API reasoners can be used information as the logical rules applied in a certain cooking for this purpose activity. (4) - Visualization libraries containing domain specific 3D icons that are used in workflow visualizations. 3.3 Semantic data integration (5) - System platform, which provides the necessary data As earlier discussed, VisuMonitor utilizes semantic technologies for workflow visualization. For example, the workflow to provide visually rich and informative workflow representations engine provides static information about workflows and to the users. For example, by using well defined ontology the Cooking guide allows to query such information as vocabularies and taxonomic hierarchies data gathered from the rules applied in the user interface adaptations. heterogeneous sources can be better integrated and semantically modeled. For example, when the monitor tool receives non- Device/hardware level: from laptop/PC to light device like semantic workflow descriptions, it saves them semantically and PDA/smart phone. annotates the data with descriptive metadata. Next VisuMonitor stores the workflow activities into an RDF data model and finally 3.2 Dynamic structure visualizes the workflows. Whenever additional information is While compositional structure provides the static layout of the queried from Cooking Guide application, it can be stored into the workflow monitoring architecture, the sequence diagram same RDF model and linked to the appropriate activities of the presented in Figure 3 highlights the way on how different workflow. components dynamically interact. The semantic modeling of workflows has many potential benefits. For example, more comprehensive diagnostics information about the work processes can be produced by discovering the hidden relationships and patterns that may exist in the data. The diagnostics information can include historical, real-time and predictive data. Additionally, the utilization of different reasoning Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA mechanisms may lead to proactive action recommendations, VisuMonitor addresses this requirement by providing illustrative which in turn enable more efficient fault prevention. Finally, the graphical explanations that makes the behavior of the cooking semantic modeling of data enables more efficient work progress guide system more transparent. VisuMonitor provides monitoring and visualization. An excerpt from an RDF- explanations, for example, about the logical rules that guide the description of semantically stored workflow data is presented in functionality of the Cooking Guide system during a certain Figure 4. cooking activity. As an example, a visualization presented in Figure 6 explains one of the rules that automatically turn the Cooking Guide‟s audio features off if music is detected during the last 20 seconds. Figure 4. Example RDF workflow data description Each of the activities contained by a workflow is defined as an individual, which has certain property and value descriptions. For example, the activity described above has a property „activityDefinitionId‟ with value „makeCoffee‟ and a property „state‟ with value „CLOSED.COMPLETED‟. 3.4 UI design mock-ups VisuMonitor tool is currently in a design phase and different specifications of the tool are being created. Since visualization and graphical user interface form such an important part of the approach several user interface mock-ups were decided to be created and evaluated before the actual implementation work is Figure 6. A rule visualization mock-up started. The purpose of the initial evaluations is to make sure that user perceive the created views and explanation dialogs as Although VisuMonitor is still on a design phase some of the informative and comprehensible. initial user interface mock-ups have been already evaluated in a UI design mock-up presented in Figure 5 shows an overall view user study performed for the Cooking Guide prototype [17]. The of the cooking process, in which the proceeding of the workflow results proved that VisuMonitor enhances the understanding of from one step to another is illustrated. The already finished application behavior and makes the functionality of Cooking activities are depicted with blue boxes, the current step of the Guide more appreciable for the user. cooking process is emphasized with red color and the green boxes represent the activities that have not yet been started. The user is able to acquire more detailed information about different activities 4. CONCLUSION AND FUTURE WORK by clicking the boxes representing the different steps. The purpose This paper has presented the VisuMonitor approach, which of this kind of overall view is to enhance the general addresses the problem of complex sensor-rich, ambient computing comprehension of cooking processes. environments causing negative reactions for ordinary users, as they feel they do not have control over their personal applications. VisuMonitor enhances the understanding of application behavior by applying interactive visualization techniques that enable users to observe, manipulate, search, navigate, explore, discover and filter data far more rapidly and far more effectively. VisuMonitor is tightly coupled with the Cooking Guide application, which provides step-by-step explanations for meal preparation and adapts its behavior according to the context information. VisuMonitor supports the users of Cooking Guide by providing visualization views that show the proceeding of the cooking process from one step to another and also explains the functionality and behavior of the system during different cooking activities. By utilizing different visualization methodologies it aims at improving user awareness, technology acceptance and user education. An important feature of chosen visualization approach is that it Figure 5. A workflow visualization mock-up semantically integrates heterogeneous data gathered from different sources. In this way all the workflow related data can be modeled and stored in a similar and structured way. The semantic As earlier discussed, a knowledge-based system should be able to representation of data facilitates also the discovering of hidden explain its reasoning and rules to justify its conclusions. relationships that may exist in the data. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA The development of VisuMonitor is currently in its initial stage. [8] Callaghan, V., Clarke, G. S., and Chin, S. J. Y. 2008. Some The work will continue by analyzing thoroughly the results gained socio-technical aspects of intelligent buildings and pervasive from the evaluation and applying this data in the implementation computing research, Intell. Build. Int‟l J. 1:1. phase. The construction process will be iterative by its nature and [9] Cheverst K., et al. 2005. Exploring Issues of User Model after each design and implementation cycle the approach will be Transparency and Proactive Behaviour in an Office evaluated with the end-users. Environment Control System, User Modeling and User- Although VisuMonitor is currently developed in a close Adapted Interaction 15:235-273 cooperation with the Cooking Guide application, we are looking [10] Cramer,H.S.M., Evers V., Van Someren, M., Ramlal, S., for more generic domain independent way to support application Rutledge, L., Stash, N., Aroyo, L., Wielinga, B. 2008. The users. Different application domain may set an additional research effects of transparency on perceived and actual competence challenge, for example on the visualization aspects like various of a content-based recommender, Semantic Web User visualization types might be used depends on the problem domain Interaction Workshop, CHI 2008, April 2008 and also on application features to be monitored and visualized. Additionally, the workflows describing semantic models will be [11] Wijk, J. 2005. The value of visualization. Proceedings of the improved by developing the data integration methods and using IEEE Visualization (VIS‟05), Minneapolis, MN, USA, 23.28 more sophisticated reasoning capabilities October 2005. [12] Hassanpour, S., O‟Connor, M.J. and Das, A.K. 2010. A Software Tool for Visualizing, Managing and Eliciting 5. ACKNOWLEDGMENTS SWRL Rules, ESWC 2010, Part II, LNCS 6089, pp. 381–385 This research was conducted within the SmartProducts EU project, grant number 231204. We would like to thank in [13] Blanchard J., et al. 2007. A 2D-3D visualization support for particular Philips for inspired and encouraging comments. human-centered rule-mining, Computer and Graphics 31, 3 (2007) 350-360 6. REFERENCES [14] Ma, Y., Liu B. and Wong, C.K. 2000. Web for data mining: [1] Feeney K. et al. 2008. Avoiding “Big Brother” Anxiety with organizing and interpreting the discovered rules using the Progressive Self-Management of Ubiquitous Computing Web. SIGKDD Explorations, ACM Press, vol. 2, num. 1, pp Services, MobiQuitous 2008, July 21-25, 2008, Dublin, 16{23} Ireland [15] Hofmann, H. and Wilhelm, A. 2001. Visual comparison of [2] Klein, D.A. and Shortliffe, E.H. 1994. A framework for association rules. Computational Statistics, Physica-Verlag, explaining decision-theoretic advice, Artificial Intelligence vol. 16, num. 3, pp 399{415} 67, 1994, 201-243. [16] Lehn, R. 2000. An interactive rule visualization system for [3] Sørmo, F. and Aamodt, A. 2002. Knowledge communication knowledge discovery in databases. PhD thesis, University of and CBR. In Proceedings of the ECCBR-02 Workshop on Nantes Case-Based Reasoning for Education and Training, 2002, [17] Vildjiounaite E., et al. 2011. Designing Socially Acceptable 487-59. Multimodal Interaction in Cooking Assistants. In [4] Carenini, G. and Moore, J. 1998. Multimedia explanations in proceedings of International Conference on Intelligent User IDEA decision support system. Working Notes of the AAAI Interfaces, Palo Alto, California, USA, February 2011. Spring Symposium on Interactive and Mixed-Initiative [18] Kohlhase, A., & Kohlhase, M. 2009. Semantic Transparency Decision Theoretic Systems. in User Assistance Systems. In Proceedings of the 27th [5] Chen, L. and Pu, P. 2005. Trust building in recommender annual ACM international conference on Design of agents. In Proceedings of the Workshop on Web Communication.Special Interest Group on Design of Personalization, Recommender Systems and Intelligent User Communication (SIGDOC-09), Bloomingtion,, IN, United Interfaces at the 2nd International Conference on E-Business States. ACM Press. and Telecommunication Networks (ICETE‟02). [19] Gribova, V. 2007. Automatic generation of context sensitive [6] McSherry, D. 2004. Explanation in recommender systems. In help using a user interface project. In proceedings of 8thh Workshop Proceedings of the 7th European Conference on International Conference "Knowledge-Dialogue-Solutions" – Case-Based Reasoning, 2004, 125-134. KDS 2007, July 2007, Varna, Bulgaria [7] O‟Donovan, J. and Smyth, B. 2005. Trust in recommender [20] WFMC 2002. Workflow Management CoalitionWorkflow systems. In Proceedings of the 10th International Conference Standard: Workflow Process Definition Interface – XML on Intelligent User Interfaces (IUI‟05), 167 - 174. Process Definition Language (XPDL) (WFMC-TC-1025). Technical report, Workflow Management Coalition, Lighthouse Point, Florida, USA. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA