Towards the user confidence in sensor-rich interactive application environment

Abstract

The recent advances in sensor-rich, ambient computing environmets have led to a situation in which ordinary users may express negative reactions when they feel that their behaviour is being monitored and analysed by technological systems which they do not understand. Cooking guide is an example application that is heavily depended on dynamic context information and adapts its behavior according to the context data. The VisuMonitor approach, described in this study, supports the users of Cooking Guide by providing visualization views that show the proceeding of cooking processes and also explains the functionality and behavior of the system during different cooking activities, thus improving user awareness, technology acceptance and user education. VisuMonitor utilizes semantic technologies in the modeling of workflows, which facilitates data integration and enables more efficient work progress monitoring and visualization.

Author keywords

Context awareness, proactive knowledge, sensors, user education, semantic technologies, user education, data visualization

1. INTRODUCTION

When evaluating the ideas of sensor-rich, ambient computing environments to ordinary users, non-technical people, in particular, express anxiety when they find themselves in situations, where they feel that their behaviour is being monitored and analysed by technological systems which they do not understand [ 1 ]. Such negative reaction to applications which use sensing technology sets a challenge which needs to be addressed. Technology must be regarded as helpful rather than threatening. We believe that if users perceive themselves to understand and to have control over their personal application, they will be more likely to trust applications which use sensing data. Accordingly a knowledge-based system should be able to explain its reasoning, and rules used to justify its conclusions to be accepted by users. Cooking guide is an example application that is heavily depended on dynamic context information [ 17 ]. The Cooking Guide may run in a touch-screen device, for example, and it helps the user during meal preparation by providing detailed, step-by-step explanations. Cooking Guide adapts its behavior according to the context information (e.g. available smart appliances augmented by various sensors, output devices, and user's cooking experience) thus each step can be potentially performed in a different way. Cooking guide is a true effort towards the contextual rich dynamic proactive knowledge-based application. Proactive knowledge base is built from the sensors augmenting the objects in use, surrounding devices and user profiles. Sophisticated data mining algorithms, rule based mechanisms and user model learning techniques facilitate contextual awareness and adaptability towards the assistance and end user ambient support.

The importance of explanation interfaces in providing system transparency and thus increasing user acceptance has been well recognized early in a number of fields such as expert systems [ 2 ], intelligent tutoring systems [ 3 ], office documents user assistance systems [ 18 ], data exploration systems [ 4 ], and recommendation systems [ 5 ][ 6 ][ 7 ]. In relation to ubicomp environment, the necessity to support the features that aim at supporting user acceptance by making system‟s reasoning process visible and insight of the system comprehendible has been acknowledged only recently [ 1 ][ 8 ][ 9 ], while prototyping of such feature is still in its infancy. For our knowledge only work by K.Cheverst [ 9 ] has practically addressed the transparency and comprehensibility of the system leveraging the power of explanation user interfaces. There the Intelligent Office System can learn a given user situation to use the inferred rules and support appropriate proactive behaviour such as e.g. turning on/off the fun or opening/closing window under appropriate conditions. On the same time, the system enables the user to explicitly scrutinise and override the „if-then‟ rules held in user model. If the user wishes to enquire why the system is performed in a certain way, the appropriate button can be pressed in order to view a window such as the one shown in Figure 1. However manually acquired textual explanations may not be always sufficient especially in the cases where the context of the application and user is rapidly varying such as in cooking which is a creative process with continuously changing cooking situation, appliances in use and products features. This sets the additional challenges on the design of the user interface. Moreover, the purpose of the system plays an important role in defying of respective elements that influence system acceptance. When interacting with work and task-oriented systems, the perceived usefulness is more important. In contrast when interacting with hedonic systems that are aimed at fun and pleasure (as cooking guide mostly does) the perceived enjoyment is more desirable in achieving user acceptance [ 10 ].

2. VISUALIZATION

Looking for the means to fulfil the above discussed requirements, we believe that visualization based aids which are intuitive and easily customizable, may help the user to link the complex contextual world of physical services residing in the environment, reasoning of the system and human mind. Visualization of data makes it possible to obtain insight into these data in an efficient and effective way, thanks to the unique capabilities of the human visual system, which enables us to detect interesting features and patterns in a short time [ 11 ]. In particular with recent advances in computer graphics, visualization is able to benefit the sense of wonder connected with the application presenting the content of the data in a completely innovative and quickly comprehendible form.

Currently existing approaches to visualise the rules of the system are targeting mainly application developers [ 12 ][ 19 ] or data exploitation professionals [ 13 ][ 14 ][ 15 ][ 16 ]. Accordingly common for the developed techniques is that they rather support the categorization, browsing and management of potentially complex rule bases, while the ground to the world of physical devices and context attractiveness, fast assimilation and intuitive visualization important for non-technical end user are left beyond.

3. VISUMONITOR – TOWARDS BETTER

USER AWARENESS

In this position paper we present a visual monitoring approach – VisuMonitor, which is currently under development. VisuMonitor is directed for the end-users of different context-aware applications and aims towards a better user awareness, technology acceptance and user educating. The approach enhances the sharing of knowledge by integrating information from multiple, heterogeneous sources and providing interactive views to this data. To enable the integration of heterogeneous data sources, VisuMonitor utilizes semantic technologies and especially ontologies that facilitate shared and common understanding of Permission to make digital or hard copies of all or part of this work for personal or classrom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

Conference’10, Month 1–2, 2010, City, State, Country.

Copyright 2010 ACM 1-58113-000-0/00/0010…$10.00.

Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA knowledge domain and are able to describe explicitly the content and semantics of heterogeneous data sources to support integration, processing and further new knowledge discovering tasks. The utilization of semantic technologies provides also an intelligent way to define and use rules that guide the behavior of the application.

The use of semantic technologies is especially pertinent with such applications as the VisuMonitor where complex and heterogeneous data is gathered from multiple sources and it has to be presented to the users in a comprehensive way. The annotation of the data using ontologies and concept taxonomies will allow users to better perceive the relationships between different concepts. Additionally, by utilizing reasoning mechanisms provided by semantic technologies, the data can be better clustered and targeted to the particular users.

VisuMonitor supports the users of Cooking Guide in two ways: showing practical information related to the cooking process itself (the proceeding of the cooking process from one step to another, the information provided by different sensors, the usage of different devices etc.) and providing explanations related to the functionality and behavior of the cooking guide system (for example why the cooking guide application decided to change from speech to textual guidance in some point of the cooking process etc.). VisuMonitor may also educate the user by explaining why the particular recipe/ingredients are recommended e.g. due health reasons, diseases, dietary, recent blood test, etc. Different cooking processes executed with Cooking Guide are modeled as workflow descriptions. Cooking Guide is tightly integrated with a Workflow engine tool, which manages the workflows that are executed in cooking processes. The executable workflows are described with an XML-based serialization format known as XPDL [ 20 ] (XML Process Definition Language). XPDL is a common format supported by a number of editing tools and process execution engines. XPDL workflow models are standardized representations of one or more workflows. The workflow engine plans, checks and manages the execution and states of workflows. If an activity is finished, it is e.g. responsible for checking outgoing conditions of transitions and deciding if the transitions should be activated or not. Workflow engine utilizes also context information extensively. Besides of information source, the engine uses context data to adapt to the situation, to trigger activity transitions and to influence the control flow. VisuMonitor communicates with Workflow engine to retrieve the necessary information needed for workflow visualizations. In addition to static and dynamic workflow representations, VisuMonitor provides also other workflow related information to the users. It may show, for example, the different resources needed to complete a workflow activity or information related to functionality and behavior of the cooking guide system. By integrating the data acquired from Workflow engine and Cooking Guide, VisuMonitor is able to produce a global view of a cooking process.

3.1 Compositional structure

The compositional structure of the VisuMonitor infrastructure is shown in Figure 2. (1) - Workflow visualization and monitoring, which is a core of the tool. This component provides mechanisms for visualizing workflows and other related information. (2) - Semantic library represented by ontologies, which will contain the workflow related knowledge base. This component contains semantically modelled workflow descriptions that are visualized with the tool. It may also contain other semantically modelled information, such as context and sensor data, rules and other system functionality data and information about different resources that are related to workflows. (3) - Ontology management tools, which allow to query and update ontology instances. Some existing open source software like Jena and OWL-API reasoners can be used for this purpose (4) - Visualization libraries containing domain specific 3D icons that are used in workflow visualizations. (5) - System platform, which provides the necessary data for workflow visualization. For example, the workflow engine provides static information about workflows and the Cooking guide allows to query such information as the rules applied in the user interface adaptations.

Device/hardware level: from laptop/PC to light device like PDA/smart phone.

3.2 Dynamic structure

While compositional structure provides the static layout of the workflow monitoring architecture, the sequence diagram presented in Figure 3 highlights the way on how different components dynamically interact. According to the sequence diagram above, the user may first create a client in order to start monitoring workflows. VisuMonitor connects to Workflow engine and retrieves the workflows that are currently hosted by the engine. The user may then select the workflows that he/she wants to visualize and monitor. Subsequently, the monitor communicates with Workflow engine and subscribes as a listener to the selected workflows. As a result, Workflow engine notifies the monitor each time something noticeable happens in the execution of the selected workflows (i.e. a transition from one activity to another or some exception/anomaly occurs during the execution). Each time VisuMonitor receives a change notification it updates the visualization view accordingly. VisuMonitor may also query some additional, workflow related information from the Cooking Guide application. The monitor may acquire, for example, such information as the logical rules applied in a certain cooking activity.

3.3 Semantic data integration

As earlier discussed, VisuMonitor utilizes semantic technologies to provide visually rich and informative workflow representations to the users. For example, by using well defined ontology vocabularies and taxonomic hierarchies data gathered from heterogeneous sources can be better integrated and semantically modeled. For example, when the monitor tool receives nonsemantic workflow descriptions, it saves them semantically and annotates the data with descriptive metadata. Next VisuMonitor stores the workflow activities into an RDF data model and finally visualizes the workflows. Whenever additional information is queried from Cooking Guide application, it can be stored into the same RDF model and linked to the appropriate activities of the workflow.

The semantic modeling of workflows has many potential benefits. For example, more comprehensive diagnostics information about the work processes can be produced by discovering the hidden relationships and patterns that may exist in the data. The diagnostics information can include historical, real-time and predictive data. Additionally, the utilization of different reasoning mechanisms may lead to proactive action recommendations, which in turn enable more efficient fault prevention. Finally, the semantic modeling of data enables more efficient work progress monitoring and visualization. An excerpt from an RDFdescription of semantically stored workflow data is presented in Figure 4. Each of the activities contained by a workflow is defined as an individual, which has certain property and value descriptions. For example, the activity described above has a property „activityDefinitionId‟ with value „makeCoffee‟ and a property „state‟ with value „CLOSED.COMPLETED‟.

3.4 UI design mock-ups

VisuMonitor tool is currently in a design phase and different specifications of the tool are being created. Since visualization and graphical user interface form such an important part of the approach several user interface mock-ups were decided to be created and evaluated before the actual implementation work is started. The purpose of the initial evaluations is to make sure that user perceive the created views and explanation dialogs as informative and comprehensible.

UI design mock-up presented in Figure 5 shows an overall view of the cooking process, in which the proceeding of the workflow from one step to another is illustrated. The already finished activities are depicted with blue boxes, the current step of the cooking process is emphasized with red color and the green boxes represent the activities that have not yet been started. The user is able to acquire more detailed information about different activities by clicking the boxes representing the different steps. The purpose of this kind of overall view is to enhance the general comprehension of cooking processes. As earlier discussed, a knowledge-based system should be able to explain its reasoning and rules to justify its conclusions. VisuMonitor addresses this requirement by providing illustrative graphical explanations that makes the behavior of the cooking guide system more transparent. VisuMonitor provides explanations, for example, about the logical rules that guide the functionality of the Cooking Guide system during a certain cooking activity. As an example, a visualization presented in Figure 6 explains one of the rules that automatically turn the Cooking Guide‟s audio features off if music is detected during the last 20 seconds. Although VisuMonitor is still on a design phase some of the initial user interface mock-ups have been already evaluated in a user study performed for the Cooking Guide prototype [ 17 ]. The results proved that VisuMonitor enhances the understanding of application behavior and makes the functionality of Cooking Guide more appreciable for the user.

4. CONCLUSION AND FUTURE WORK

This paper has presented the VisuMonitor approach, which addresses the problem of complex sensor-rich, ambient computing environments causing negative reactions for ordinary users, as they feel they do not have control over their personal applications. VisuMonitor enhances the understanding of application behavior by applying interactive visualization techniques that enable users to observe, manipulate, search, navigate, explore, discover and filter data far more rapidly and far more effectively.

VisuMonitor is tightly coupled with the Cooking Guide application, which provides step-by-step explanations for meal preparation and adapts its behavior according to the context information. VisuMonitor supports the users of Cooking Guide by providing visualization views that show the proceeding of the cooking process from one step to another and also explains the functionality and behavior of the system during different cooking activities. By utilizing different visualization methodologies it aims at improving user awareness, technology acceptance and user education.

An important feature of chosen visualization approach is that it semantically integrates heterogeneous data gathered from different sources. In this way all the workflow related data can be modeled and stored in a similar and structured way. The semantic representation of data facilitates also the discovering of hidden relationships that may exist in the data.

The development of VisuMonitor is currently in its initial stage. The work will continue by analyzing thoroughly the results gained from the evaluation and applying this data in the implementation phase. The construction process will be iterative by its nature and after each design and implementation cycle the approach will be evaluated with the end-users.

Although VisuMonitor is currently developed in a close cooperation with the Cooking Guide application, we are looking for more generic domain independent way to support application users. Different application domain may set an additional research challenge, for example on the visualization aspects like various visualization types might be used depends on the problem domain and also on application features to be monitored and visualized. Additionally, the workflows describing semantic models will be improved by developing the data integration methods and using more sophisticated reasoning capabilities

5. ACKNOWLEDGMENTS

This research was conducted within the SmartProducts EU project, grant number 231204. We would like to thank in particular Philips for inspired and encouraging comments.

References

[1] Feeney K. et al. 2008 . Avoiding “Big Brother” Anxiety with Progressive Self-Management of Ubiquitous Computing Services , MobiQuitous 2008, July 21-25 , 2008 , Dublin, Ireland

[2] Klein , D.A. and Shortliffe , E.H. 1994 . A framework for explaining decision-theoretic advice , Artificial Intelligence 67 , 1994 , 201 - 243 .

[3] Sørmo , F. and Aamodt , A. 2002 . Knowledge communication and CBR . In Proceedings of the ECCBR-02 Workshop on Case-Based Reasoning for Education and Training , 2002 , 487 - 59 .

[4] Carenini , G. and Moore , J. 1998 . Multimedia explanations in IDEA decision support system . Working Notes of the AAAI Spring Symposium on Interactive and Mixed-Initiative Decision Theoretic Systems.

[5] Chen , L. and Pu , P. 2005 . Trust building in recommender agents . In Proceedings of the Workshop on Web Personalization, Recommender Systems and Intelligent User Interfaces at the 2nd International Conference on E-Business and Telecommunication Networks (ICETE‟02).

[6] McSherry , D. 2004 . Explanation in recommender systems . In Workshop Proceedings of the 7th European Conference on Case-Based Reasoning , 2004 , 125 - 134 .

[7] O ‟Donovan , J. and Smyth , B. 2005 . Trust in recommender systems . In Proceedings of the 10th International Conference on Intelligent User Interfaces (IUI‟05) , 167 - 174 .

[8] Callaghan , V. , Clarke , G. S. , and Chin , S. J. Y. 2008 . Some socio-technical aspects of intelligent buildings and pervasive computing research , Intell. Build. Int‟l J . 1 : 1 .

[9] Cheverst K. , et al. 2005 . Exploring Issues of User Model Transparency and Proactive Behaviour in an Office Environment Control System, User Modeling and UserAdapted Interaction 15 : 235 - 273

[10] Cramer , H.S.M. , Evers V. , Van Someren , M. , Ramlal , S. , Rutledge , L. , Stash , N. , Aroyo , L. , Wielinga , B. 2008 . The effects of transparency on perceived and actual competence of a content-based recommender , Semantic Web User Interaction Workshop , CHI 2008, April 2008

[11] Wijk , J. 2005 . The value of visualization . Proceedings of the IEEE Visualization (VIS‟05) , Minneapolis, MN, USA, 23 .28 October 2005 .

[12] Hassanpour , S. , O ‟Connor, M.J. and Das , A.K. 2010 . A Software Tool for Visualizing, Managing and Eliciting SWRL Rules , ESWC 2010 , Part II , LNCS 6089, pp. 381 - 385

[13] Blanchard J. , et al. 2007 . A 2D-3D visualization support for human-centered rule-mining , Computer and Graphics 31 , 3 ( 2007 ) 350 - 360

[14] Ma , Y. , Liu B. and Wong , C.K. 2000 . Web for data mining: organizing and interpreting the discovered rules using the Web . SIGKDD Explorations , ACM Press, vol. 2 , num . 1, pp 16 { 23 }

[15] Hofmann , H. and Wilhelm , A. 2001 . Visual comparison of association rules . Computational Statistics , Physica-Verlag, vol. 16 , num . 3, pp 399 { 415 }

[16] Lehn , R. 2000 . An interactive rule visualization system for knowledge discovery in databases . PhD thesis , University of Nantes

[17] Vildjiounaite E. , et al. 2011 . Designing Socially Acceptable Multimodal Interaction in Cooking Assistants . In proceedings of International Conference on Intelligent User Interfaces , Palo Alto, California, USA, February 2011 .

[18] Kohlhase , A. , & Kohlhase , M. 2009 . Semantic Transparency in User Assistance Systems . In Proceedings of the 27th annual ACM international conference on Design of Communication .Special Interest Group on Design of Communication (SIGDOC-09), Bloomingtion,, IN, United States. ACM Press.

[19] Gribova , V. 2007 . Automatic generation of context sensitive help using a user interface project . In proceedings of 8thh International Conference "Knowledge-Dialogue-Solutions" - KDS 2007 , July 2007 , Varna, Bulgaria

[20] WFMC 2002 . Workflow Management CoalitionWorkflow Standard: Workflow Process Definition Interface - XML Process Definition Language (XPDL) (WFMC-TC-1025) . Technical report , Workflow Management Coalition, Lighthouse Point, Florida, USA.