Catching the User - User Context through Live Logging in DAFFODIL Claus-Peter Klas Matthias Hemmje FernUniversität in Hagen FernUniversität in Hagen claus-peter.klas@fernuni-hagen.de matthias.hemmje@fernuni-hagen.de ABSTRACT DAFFODIL consists of several tools to support the users This demo will present the logging facilities to capture the search tasks. We would like to demonstrate the DAFFODIL 1 user context within the Daffodil framework during a live framework as an experimental system for the evaluation of search in computer science data sources. We propose to use the interaction in information search and retrieval. DAF- the Daffodil system as an experimental framework for the FODIL is a virtual digital library system providing access evaluation and research of interactive IR. The system al- to many sources from the domain of computer science, and ready provides a rich set of working services and available targeted at strategic support of users during the informa- information sources. These services and sources can be used tion seeking and retrieval process. It provides basic and as a foundation for further research going beyond basic func- high-level search functions for exploring and managing in- tionalities. In addition, the system can easily be extended formation objects including annotations over a federation regarding both services and sources. Daffodil’s highly flex- of heterogeneous digital libraries (DLs) based on a service- ible and extensible agent-based architecture allows for easy oriented architecture. For structuring the functionality, we integration of additional components and access to all ex- employ the concept of high-level search activities for strate- isting services. Finally, the system provides a user-friendly gic support. A comprehensive evaluation in [2] showed that graphical interface and facilitating services for log genera- the system supported most of the information seeking and tion and analysis. The experimental framework can serve as retrieval aspects needed for a computer scientist’s daily rou- a joint theoretical and practical platform for the evaluation tine. of DLs, with the long-term goal of creating a community centered on interactive IR and DL evaluation. Additionally DAFFODIL incorporates the concepts of adap- tivity [3], collaboration, recommendation and awareness. In Categories and Subject Descriptors order to enable adaptivity and recommendations DAFFODIL H.3.3 [Information Search and Retrieval]: User Inter- collects implicit and explicit user interactions and system face, Logging; H.3.7 [Digital Libraries]: User Issues actions as described in previous publications [1]. This in- teraction can be examined and captured at various levels of abstraction, starting at the system/hardware level and General Terms covering the complete spectrum of user-system interaction. information retrieval, visualization, interactive systems 2. LEVELS OF LOGGING 1. INTRODUCTION When using transaction logs for evaluation, the primary lev- It is our intention to demonstrate the Daffodil system with els surveyed are the user, the system, and the content that regard to capturing the user context by logging all user ini- is being searched, read, manipulated, or created. Because tiated actions. Due to the rich functionality of the DAF- interaction between the system and the user can be captured FODIL system the user is able to explore the search domain at various levels of abstraction we focus on three levels of in a comprehensive and sustainable way. During the inter- evaluation: action of the user with the system we capture the behavior through logging the actions. We categorized these actions User behavior level: Data about users and their behavior into ten different conceptual events described in [1]. are located at this level. Each user has a task to accomplish, within a certain social environment, and brings her individ- 1.1 Daffodil-Framework ual knowledge to that task. Concept level for comparative evaluations: The con- cept level captures data about generalized events generated by the DL user. By logging these events, user evaluation can be backed up with statistical data and a comparative evalu- ation of different users, systems and system content can be undertaken. System level: System events happen on the computer or in the computer network where DL services are executed. This Copyright is held by the author/owner(s). SIGIR’09, July 19-23, 2009, 1 Boston, USA. http://www.daffodil.de level aggregates specific information concerning the state of the DL (e.g., database conditions, server load, or amount of network traffic) and its response (e.g., response time). Through logging events by level, we have a horizontal view that tracks a sequence of events dealing with a single aspect of the DL. For example, by focusing on events that occur on the concept level, we can identify the user’s moves and tactics as she works her way through the document space. In contrast, a vertical view across levels gives us information about the impact of a specific event across the DL system, from information about user behavior on the highest level to system specific data on the lowest level. 3. EVENTS ON THE CONCEPT LEVEL At the concept level, we have identified several general event types that support comparative evaluation across DLs. Our focus on the concept level represents the centrality of these events for log analysis and interpretation: events that occur Figure 1: top-down tree visualization on the concept level indicate critical aspects of the user’s interaction with the DL system and supply valuable data Based on a given task several search and browse tools will for rich interpretation of user behavior. As is highlighted be presented and a analysis, e.g. for relevance feedback will in other DL logging studies [4], current approaches are of- be given. In figure 1 a small search session if graphically ten inadequate for capturing complex or abstract actions by presented as tree visualization. The colors of the nodes in the user and are therefore unable to elicit meaningful con- the figure correspond to the concept level.events for easier clusions. By logging data about general event types at the recognition. concept level, we provide a basis for comparative evaluation across DLs and still gain insides into the users behavior. This captured information represents the basis to further un- derstand and support the user. It of course does not excuse The event types and event properties that we have iden- from running a real user evaluation. Such support could be tified are neither fixed nor complete and should be viewed done through recommendation via implicit relevance feed- as recommendations. that can also serve as discussion points back as well as collaborative recommendations through other in the community. users in a similar situation. We think, that given the con- text model within the Daffodil-Framework, we are able to We have identified the following events on the concept level: understand and categories user behavior and provide solid Search: The user formulates a query or filter condition that data to support system oriented IR evaluation, e.g. based is to be processed by a given DL service against a collection. on user simulation. Navigate: The user initiates an event by selecting from a set of possible moves from one point to another. Inspect: The user accesses the details of a single object. 5. ACKNOWLEDGMENTS This work is funded via the German Science Foundation Display: The display event describes a specific visualization (DFG) in the project LACOSTIR. of the information presented to the user. Browse: The user selects an event for viewing a set of DL objects (e.g., viewing a result list following a search). 6. REFERENCES Store: The user files an object for later reuse, either at a [1] C.-P. Klas, H. Albrechtsen, N. Fuhr, P. Hansen, generic location (e.g., on a clipboard) or at a specific location S. Kapidakis, L. Kovács, S. Kriewel, A. Micsik, (e.g., in a specific folder). C. Papatheodorou, G. Tsakonas, and E. Jacob. A Annotate: The user adds information to an existing DL logging scheme for comparative digital library object, either by marking specific parts of it, by linking it to evaluation. In ECDL 2006, Lecture Notes in Computer other digital library objects, or by adding inline or external Science, pages 267–278, Heidelberg et al., September comments. 2006. Springer. Author: The user creates a new DL object or edits an [2] C.-P. Klas, N. Fuhr, and A. Schaefer. Evaluating existing object such as a document or annotation. strategic support for information access in the Help: The user requests help or information. The help DAFFODIL system. In ECDL 2004, Lecture Notes in event may be general or context-specific and can include Computer Science, Heidelberg et al., 2004. Springer. introductory overviews or tutorials about the DL system. [3] C.-P. Klas, S. Kriewel, and M. Hemmje. An Communicate: Users collaborate through communication, experimental system for adaptive services in either by posing a simple question or through the use of information retrieval. In Proceedings of the 2nd specific tools or services. International Workshop on Adaptive Information Retrieval (AIR 2008), October 2008. 4. DEMONSTRATION [4] B. Pan. Capturing users behavior in the national In the demonstration we will present the DAFFODIL sys- science digital library (nsdl). Technical report, NSDL, tem with a specific focus on the live logging of user events. 2003. http://dlist.sir.arizona.edu/848/.