An Experiment and Analysis System Framework for the Evaluation of Contextual Relationships Ralf Bierig Michael Cole Jacek Gwizdka Rutgers University Rutgers University Rutgers University bierig@rci.rutgers.edu m.cole@rutgers.edu cirse10@gwizdka.com Nicholas J. Belkin Jingjing Liu Chang Liu Rutgers University Rutgers University Rutgers University belkin@rutgers.edu jingjing@eden.rutgers.edu imliuc@gmail.com Jun Zhang Xiangmin Zhang Rutgers University Rutgers University zhangj@eden.rutgers.edu xiangminz@gmail.com ABSTRACT for the evaluation of particular contextual attributes and This paper presents an experiment and analysis system frame- 2) integrate data and analyze results to better understand work that allows researchers to design and conduct inter- contextual relationships. The framework promotes an inter- active experiments and analyze data for the evaluation of active and task-oriented viewpoint that is supported by a contextual relationships. wide range of logging tools. Categories and Subject Descriptors Section 2 reviews the system architecture consisting of the H.4 [Information Systems Applications]: Miscellaneous experiment and the analysis system. Section 3 describes how the architecture supports researchers to investigate and evaluate contextual relationships. Section 4 discusses the Keywords current state of the system and future plans for its dissemi- Context Evaluation, Information System, Data Analysis nation. 1. INTRODUCTION AND BACKGROUND In the last decade, context-aware computing has made much 2. SYSTEM FRAMEWORK effort to formalize context[3], describe general context mod- The system framework is part of a project deliverable1 that els[7] and develop systems that apply such models in dif- aims to investigate ways to improve users’ ability to find in- ferent application domains [5] – such as mobile computing formation in search environments such as digital libraries. In (e.g. tourism and recreation [16, 11]). There is, however, particular we analyze various interacting contextual factors only limited research about the experimental evaluation of that are involved in such online search activities. Despite context, particularly about the effects of various contextual our focus, results are expected to contribute to a much wider attributes and their interaction. This gap is beginning to be range of application environments such as mobile search and addressed with several workshops and conferences [9, 8, 12, recommender systems. 2]. 2.1 Overview Rigorous experimentation in this domain presents challenges The overall aim of our framework is to reduce the complex- in that such experiments are generally difficult to adminis- ity of designing and conducting experiments and integrating ter and demanding in resources [6, 10]. Although software and analysing results from experiments for the evaluation frameworks for contextual enrichment of applications exist of contextual relationships as usually expressed in user and [13, 4] there is generally little system-related support for context models. Such experiments usually require a complex comprehensive evaluation of context attributes and models. arrangement of system components (e.g. GUI, user manage- This paper presents a system framework that provides re- ment and persistent data storage). Our framework enables searchers with a tool to: 1) design and conduct experiments researchers to focus on research related issues (e.g. task and questionnaire design and the selection of experiment vari- ables) rather than the creation of the experiment logic and the transformation, integration and the processing of data and results after the experiment has been completed. This helps to reduce the overall time and effort that is needed to design and conduct experiments and to get valuable results about contextual relationships from experiment data. As Appears in the Proceedings of The 2nd International Workshop on Contex- shown in figure 1, the system framework consists of two parts CIRSE ’10 MiltonAccess, tual Information Keynes,Seeking UK and Retrieval Evaluation (CIRSE 2010), 1 March 28, 2010, Milton Keynes, UK. http://comminfo.rutgers.edu/imls/poodle/ http://www.irit.fr/CIRSE/ Copyright owned by the authors. Figure 1: Components of the experiment and analysis system framework – 1) an experiment system that allows researchers to design tools that cover additional contextual information from the and conduct interactive experiments in close-to-operational user or the user’s environment. Examples may include lo- application environments and 2) an analysis system that en- cation information (e.g. geographic position or proximity to ables them to integrate and analyze results obtained from points of interest) or physiological states of the user (e.g. such experiments. heart rate or Galvanic skin response). Logging information is either stored in an experiment database through the DB 2.1.1 Experiment System Interface or in application-specific log files. The experiment system, described in more detail in [1], in- cludes a number of components. 2.1.2 Analysis System The analysis system serves as an extension of the experiment The GUI provides authenticated login for participants, their system with additional features to integrate experiment data assignment to one or more experiments and basic naviga- into a unified data structure. Researchers can inspect and tional support during an experiment. The Experimenter explore these data sets and segment and model results to controls and coordinates an Extensible Task Framework that gain a better understanding of contextual relationships. The offers researchers a set of reusable tasks that can be used for analysis system consists of the following components: creating experiments (e.g. standard open web search tasks). Own tasks can also be added to this collection. Furthermore, the Experimenter manages Task Progress and Control that • The Event Representation integrates experiment data balances task sequences, monitors the progress of partici- through the Event Reader Interface into a unified event pants including the safe recovery of interrupted sessions. In data structure. This data structure is extensible and addition, the Interaction Logger with Remote Logging pro- the collection of event readers mirror the logging tools vides a mechanism for tasks to log contextual information provided with the experiment system as described in internally at specific points during an experiment task and the previous section. An extensible set of event types to call external logging applications on the client. This al- ensures that researchers can adapt and extend the anal- lows creating more effective experiments that may include ysis framework to process data from a variety of ex- different kinds of contextual data logging on both server periments under a single platform. This ensures that and the client side. Whereas the server has a central log- additional logging tools can be introduced through the ging facility, the client consists of a flexible and expandable experiment system to capture additional types of user array of independent loggers. Currently, these loggers ob- context either through the logging of high-level user serve the most commonly known user behaviours – keyboard behaviour or through the application of low-level sen- and mouse activities, web navigation, usability information sors as described in [14]. from Morae2 and eye-tracking data from Tobii3 . This list • Event Reader Import Rules can be used to configure can easily be expanded with other (existing or new) logging event readers and therefore adapt the data import pro- 2 http://www.techsmith.com cess. Such rules can for example be applied to add 3 http://www.tobii.com additional filters for event readers (e.g. excluding web events with certain URLs) or providing standard vali- segmentation, as a tool for data categorization and dation (e.g. tagging certain events as problematic thus conditioning, and through modelling to investigate and flagging results for manual inspection). discover contextual relationships. • Data Segmentation divides experiment data into se- • Extensibility: As an extensible framework with respect mantic units guided by research hypotheses. The sys- to contextual logging tools (in the experiment sys- tem framework provides a standard minimal segmen- tem) and readers, rules, segmentations and models (in tation by distinguishing data based on experiments, the analysis system) the framework offers researchers users and tasks. The research can add additional lev- ways to adapt and extend it to their own require- els of data segmentation to structure data in smaller ments and research agendas. These extensions how- logical units. A segmentation can for example differ- ever require additional, customizing implementation entiate interaction data based on users’ current stage work by the user of the system framework; for example in a search task (e.g. distinguishing users’ task stages adding another logging tool to measure a new contex- of query formulation, result page inspection and con- tual aspect from the user also requires implementing tent page viewing) or, more generally, data can be the corresponding event representation and an addi- segmented along low-level decision points (e.g. mouse tional reader to import the new data log. Such proce- clicks and/or key strokes). dures, however, are guided through the application of programming interfaces and supported with examples • Model Representation processes (segmented) event se- that are available in open source as part of the project. quences to test specific research hypotheses i.e. ver- This is not much different from other extensible soft- ifying effects of context attributes and relationships ware frameworks such as WEKA [15]. between them (e.g. identifying users’ perceived useful- ness of content and determining reading behaviour). • Separation between data and modelling: Data (in the Other data segmentations and model representations form of low-level event representations) is separated can be added by researchers to further specialize the from its interpretation (in the form of high-level seg- system framework for particular types of analysis. mentations and models). Thus, it is possible to gener- ate multiple, alternative context models from the same • The Web-based User Interface extends the system to underlying events that can each be evaluated in isola- an online service where researchers can generate, in- tion. This also allows user and context models to be spect and share event representations, data segmen- reused for different data segments from one or across tations and models within one or across multiple ex- multiple experiments. periment data sets. These are stored through a DB Interface that persists both event and model repre- • Collaboration is central to the design and has been sup- sentations into separate databases for later reuse. The ported in both parts of the system framework. The ex- user interface supports authenticated login to allow the periment system allows researchers to implement and system to be used as part of a collaborative research share experiment tasks thus building a collaborative platform. repository (e.g. internet search tasks, tag cloud search, standard questionnaires for language understanding and various cognitive tests). Likewise, configurations for 3. CONTEXT EVALUATION WITH THE behavioural and contextual logging tools can be cre- SYSTEM FRAMEWORK - BENEFITS ated and reused across different experiments and shared AND LIMITATIONS between researchers. The analysis system offers a meet- ing platform through its web-based user interface. Data, The system design incorporates many aspects useful for the segmentations and models can be configured, integrated evaluation of contextual relationships from data obtained in and shared between researchers allowing collaborat- interactive and task-based experiments. This section sum- ing with data and ideas and forming virtual research marizes these aspects, shows how they relate to the sys- groups. Researchers can create and exchange inte- tem framework, points out how they can help researchers to grated event data sets from experiments specific to evaluate context, and expresses limitations that should be the needs of individuals or groups (e.g. event data considered. limited to a subset of experiment participants, exper- iment tasks or types of context such as web activity • Modularity: Context models may cover a wide range or eye movement). Shared data sets can then be ap- of attributes based on dimensions such as the applica- plied for further data segmentation (e.g. selecting only tion environment (e.g. library or mobile environment) particular user activities or contextual states, such as and the intended user group (e.g. professional jour- query input or reading behaviour). An extensible pool nalists or online web searchers) as well as others. The of models can be applied to such segments and ac- system framework supports this requirement in a num- cessed collaboratively. Basic summary visualizations ber of ways. First, a modular and multi-dimensional are available and findings can be exported allowing re- logging framework within the experiment system can searchers to further process data with third-party tools record behavioural data from the user and sensory data and apply results (e.g. integrating a learned context from the user’s environment. Second, these multi- model in a personalized desktop search application). dimensional data streams can be integrated into a uni- fied stream of events within the analysis system. Third, this event stream can be treated holistically through 4. CURRENT STATE AND FUTURE PLANS [8] P. Ingwersen, K. Jaervelin, and N. Belkin. Workshop A prototype of the experiment system has has been designed on information retrieval in context. In 28th Annual and developed with active work on improving logging com- International ACM SIGIR Conference on Research prehensiveness (especially for contextual, sensor-based log- and Development in Information Retrieval, Salvador, ging) and scalability. The experiment system has already Brazil, 2005. Royal School of Library and Information been applied to design and conduct four experiments each Science, Copenhagen, Denmark. with distinctive design and goals for our research project. In [9] P. Ingwersen, K. van Rijsbergen, and N. Belkin. those experiments we have collected rich contextual informa- Workshop on information retrieval in context (irix). In tion for the basic investigation of relationships between use 27th Annual International ACM SIGIR Conference on behaviour and various user context attributes such as cog- Research and Development in Information Retrieval, nitive abilities and individual differences, reading and scan- Sheffield, UK, 2004. ning behaviour and perception of usefulness during online [10] J. Kjeldskov, C. Graham, S. Pedell, F. Vetere, search. The analysis system has been designed and the mod- S. Howard, S. Balbo, and J. Davies. Evaluating the elling and user interface is in active development. The exper- useability of a mobile guide: The influence of location, iment system framework has been released as open source4 . participants and resources. Behaviour and Information The analysis system will be released as open source when it Technology, 24(1):51–65, 2005. is feature complete and stable. Both of these systems can [11] D. M. Mountain and A. MacFarlane. Geographic benefit the research community by allowing for collabora- information retrieval in a mobile environment: tion between researchers and enabling additional improve- Evaluating the needs of mobile individuals. Journal of ments and extensions to better serve the needs of context Information Science, 33(5):515–530, 2007. researchers. [12] I. Ruthven, P. Borlund, P. Ingwersen, N. Belkin, A. Tombros, and P. Vakkari, editors. Information Acknowledgements: This work is supported by IMLS Interaction in Context. 1st International Symposium grant LM-06-07-0105-07. on Information Interaction in Context, IIiX 2006. ACM Press, Copenhagen, Denmark, 2006. 5. REFERENCES [13] D. Salber, A. K. Dey, and G. D. Adowd. The context [1] R. Bierig, J. Gwizdka, and M. Cole. A user-centered toolkit: Aiding the development of context-enabled experiment and logging framework for interactive applications. In Conference on Human Factors in information retrieval. In N. J. Belkin, R. Bierig, Computing Systems (CHI), pages 434–441, Pittsburgh, G. Buscher, L. v. Elst, J. Gwizdka, J. Jose, and PA, USA, 1999. ACM Press. J. Teevan, editors, SIGIR 2009 Workshop on [14] A. Schmidt. Ubiquitous Computing - Computing in Understanding the user - Logging and interpreting Context. Phd thesis, Lancaster University, 2002. user interactions in IR, Boston, MA, 2009. [15] I. H. Witten and E. Frank. Data Mining: Practical [2] P. Borlund, J. W. Schneider, M. Lalmas, A. Tombros, Machine Learning Tools and Techniques. Morgan J. Feather, D. Kelly, A. de Vries, and L. Azzopardi. Kaufmann, 2nd edition, 2005. Second symposium on information interaction in [16] A. Zipf. User-adaptive maps for location-based context (iiix). London, UK, 2008. ACM Press. services (lbs) for tourism. In 9th Int. Conf. for [3] A. K. Dey, G. Kortuem, D. Morse, and A. Schmidt. Information and Communication Technologies in Special issue on situatuated interaction and Tourism (ENTER 2002), pages 329–337, Innsbruck, context-aware computing. Personal and Ubiquitous Austria, 2002. Springer Verlag. Computing, 5(1), 2001. [4] P. Fahy and S. Clarke. Cass - middleware for mobile context-aware applications. In 2nd International Conference on Mobile Systems, Applications, and Services (MobiSys 2004), Workshop on Context-Awareness, Bosten, MA, USA, 2004. [5] A. Göker, H. I. Myrhaug, and R. Bierig. Context and information retrieval. In A. Göker and J. Davies, editors, Information Retrieval: Searching in the 21st Century. John Wiley and Sons, Ltd, Chichester, UK, 2009. [6] J. Goodman, S. Brewster, and P. Gray. Using field experiments to evaluate mobile guides. In 6th International Symposium on Human Computer Interaction with Mobile Devices and Services (Mobile HCI), International Workshop on HCI with Mobile Guides, Glasgow, UK, 2004. [7] J. Indulska and D. D. Roure. Workshop on advanced context modelling, reasoning and management. In 6th International Conference on Ubiquitous Computing (UbiComp), Nottingham, UK, 2004. 4 http://sourceforge.net/projects/piirexs/