Evaluation of Digital Library Services Using Complementary Logs∗ Maristella Agostii Franco Crivellari Giorgio Maria Di Nunzio University of Padua University of Padua University of Padua Via Gradenigo 6/a, 35131 Via Gradenigo 6/a, 35131 Via Gradenigo 6/a, 35131 Padua, Italy Padua, Italy Padua, Italy agosti@dei.unipd.it crive@dei.unipd.it dinunzio@dei.unipd.it ABSTRACT Log is a concept commonly used in computer science; in In recent years, the importance of log analysis has grown, log fact, log data are collected by programs to make a permanent data constitute a relevant aspect in the evaluation process of record of events during their usage. The log data can be used the quality of a digital library system. In this paper, we ad- to study the usage of a specific application, and to better dress the problem of log analysis for complex systems such adapt it to the objectives the users were expecting to reach. as digital library systems, and how the analysis of search In the context of the Web, the storage and the analysis of query logs or Web logs is not sufficient to study users and Web log files are mainly used to gain knowledge on the users interpret their preferences. In fact the combination of im- and improve the services offered by a Web portal, without plicitly and explicitly collected data improves understanding the need to bother the users with the explicit collection of of behavior with respect to the understanding that can be information. gained by analyzing the sets of data separately. When research addresses the problem of studying log data Categories and Subject Descriptors in digital libraries, which are very complex systems, differ- H.3.7 [Digital Libraries]: User Issues; H.3.3 [Information ent characteristics regarding library automation systems and Storage and Retrieval]: Information Search and Retrieval— digital library systems need to be taken into account. In fact, Search process; H.3.4 [Systems and Software]: User pro- for all the different categories of users of a digital library sys- files and alert services tem, the quality of services and documents the digital library supplies are very important. Log data constitute a relevant aspect in the evaluation process of the quality of a digital li- General Terms brary system and of the quality of interoperability of digital Algorithms, Design, Experimentation library services [2, 18]. With this concept in mind, it is also possible to think about new different logging formats which Keywords reflect how a generic DL system behaves [14]. Web Log, Search Log, User Study This paper deals with the study of complementary types of logs in complex systems with the aim of finding new ways 1. INTRODUCTION of using them to evaluate and personalize digital library ser- The interaction between the user and an information access vices for the final users. The paper is organized as follows: system can be analyzed and studied to gather user prefer- Section 2 presents previous related work, Section 3 analyzes ences and to “learn” what the user likes the most, and to use and presents different facets of the study and use of logs of this information to personalize the presentation of results. complex systems, Section 4 presents the findings of the case User preferences can be learned explicitly, for example ask- study conducted in the context of the TELplus project1 for ing the user to fill-in questionnaires, or implicitly, by study- the evaluation and personalization of the services of The Eu- ing the actions of the user which are recorded in the search ropean Library, and lastly Section 5 draws conclusions and log of a system. The second choice is certainly less intrusive indicates directions for the continuation of the work. but requires more effort to reconstruct each search session a user made in order to learn his preferences. ∗Copyright is held by the author/owner(s). 2. RELATED WORK In the last decade, log analysis has become one of the main SIGIR’09, July 19-23, 2009, Boston, USA. threads of research for understanding users of search engines as shown by the works presented at three major relevant conferences and that have been analyzed by us2 . Those works study logs in different ways and for different 1 http://www.theeuropeanlibrary.org/telplus/ 2 The three analyzed major conferences are: SIGIR - http://www.sigir.org/ WWW - http://www.iw3c2.org/ JCDL - http://www.jcdl.org/ purposes, but they can be divided into two main classes: by content in the same manner as information retrieval sys- studies about search query logs, and studies about Web tems and search engines [1]. In all other types of searches, server logs. Since most of these research papers concern either the digital library system makes use of authority data search engines, the focus of their research is more on improv- to respond to final users in a more consistent and coherent ing queries and results and less on surfing the Web. The few way through a search system that is a sort of a new gener- exceptions to this classification will be analyzed later in the ation of online public access catalogue (OPAC) system, or paper. the system supports the full content search with a service that gives the final users the facilities of a search engine. Query search logs can be used for: building knowledge, such as automatically building a search thesaurus [10], or ac- Search query logs or Web logs alone give only a partial view quiring ontological knowledge [24]; refining and expanding of the stream of information that users produce. [28] show queries by means of analysis of search logs [4], or by means of how to combine two different streams of data, search query correlations between query terms and document terms based logs and click-streams, in order to analyze re-finding behav- on search query logs [11]; comparing of query extension tech- ior of a group of users under observation for a period of one niques with pseudo-relevance feedback techniques [30]; orga- year. nizing search results [29]; studying temporal changes and re- lationships, such as changes of queries on hourly basis in or- Moreover, log analysis can be supported and validated by der to understand how user preferences change over time [5], user studies which are a valuable method for understanding analysis of multitasking user searches [6], issues related to user behavior in different situations. User studies require ambiguity and freshness of queries [22], studies of causal a significant amount of time and effort, so an accurate de- relations between queries [27]; mining queries for extracting sign of the process has to be carried out. In general, user news-related queries [20], and association rules to discover studies and logs are used in a separate way, since they are related queries [25], or fast query recommendations [32]. adopted with different aims in mind. Ingwersen and Järvelin report in [17] that it seems more scientifically informative to Web logs can be used for: improving rank of results by re- combine logs together with observation in naturalistic set- placing the adjacency matrix of the HITS algorithm with a tings. Pharo and Järvelin in [23] suggest systematic use of link matrix which weights connections between nodes based the triangulation of different data collection techniques as on the usage data from Web server log traffic [21]; matching a general approach in order to get better knowledge of the website organization with visitor expectations by means of Web information search process. An example of this type of Web log analysis [26]; finding user navigational patterns [9]; combined studies is [15], where that authors claim that fully agents’ detection [7]. understanding user satisfaction and user intent requires a depth of data unavailable in search query logs but possible There is also a recent emerging research activity about log to acquire from other sources of data, such as one-on-one analysis which tackles cross-lingual issues: [13] extends the studies or instrumented panels. notion of query suggestion to cross-lingual query suggestion studying search query logs; [16] leverages click-through data The combination of implicitly and explicitly collected data to extract query translation pairs. The interest in multilin- improves understanding of behavior with respect to the un- gual log analysis is also confirmed by initiatives promoted derstanding that can be gained by analyzing the sets of data by the TrebleCLEF3 coordination action which supports the separately. In particular for digital libraries, where the eval- development and consolidation of expertise in the multidis- uation of the different services is difficult if logs are used ciplinary research area of multilingual information access alone, the combined sets of data provide the opportunity (MLIA). of reaching insights towards user personalization of digital library services. 3. LOGS OF COMPLEX SYSTEMS From this starting point we have developed a method for col- Present digital library systems are complex software sys- lecting data derived from the user interaction log, “implicit” tems, often based on a service-oriented architecture, able to data, and data collected from user questionnaires, “explicit” manage complex and diversified collections of digital objects. data, for analyzing the interaction between users and digital One significant aspect that still relates present systems to libraries. This means that the conceived method is based on the old ones is that the representation of the content of the the combination and analysis of the following data sources: digital objects that constitute the collection of interest is HTTP log which contains the HTTP requests sent by the still done by professionals. This means that the manage- Web client to the Web server during a user browsing session; ment of metadata can still be based on the use of authority search log which contains the actions performed by the user control rules in describing author, place names and other rel- during a search; questionnaire data which are collected at evant catalogue data. A digital library system can exploit the end of a user browsing and searching session. authority data that keep lists of preferred or accepted forms of names and all other relevant headings. This is a dra- The possibility of studying and correlating different sources matic difference between digital library systems and search of data was envisaged during the study of the Web portal of engines, and it is usually overcome with the analysis of log The European Library4 , which provides a vast virtual col- data. In fact a search engine often becomes a specific com- lection of material from all disciplines and offers interested ponent of a digital library system, when the digital library visitors simple access to European cultural heritage. system faces the management and search of digital objects 3 4 http://www.trebleclef.eu/ http://www.theeuropeanlibrary.org/ 4. RESULTS OF THE CASE STUDY The European Library is a free service that offers access to Table 1: Summary of statistics for the time of a the resources of 48 national libraries of Europe in 20 lan- user session in minutes calculated in the search guages with about 150 million entries across Europe. The logs (between brackets registered user only), HTTP European Library provides a vast virtual collection of mate- logs (between brackets user who participated in the rial from all disciplines and offers interested visitors simple study), and the time for filling-in the questionnaire. access to European cultural heritage. Search log HTTP log Questionnaire Median 2.0 (4.0) 1.3 (30.25) 31.0 To validate the proposed method, a study was conducted in Mean 6.0 (8.0) 4.7 (31.80) 33.0 a controlled setting at the end of 2007 – beginning of 2008, in the computer laboratories of different faculties of the Uni- versity of Padua, Italy, where students were requested to session. One of the outcomes of the questionnaire was the conduct a free navigation and search for information on The disorientation of the user upon entering The European Library European Library portal and to fill in a questionnaire specif- portal for the first time, in particular it seems not to be clear ically designed to harvest the data that can be used to ex- what kind of information can be accessed through this por- tract information on users satisfaction on the use of different tal. Users are in general ready to search in a Google-like parts of the portal. A total of 155 students participated in fashion and obtain documents, in terms of links to pages the study, mostly Italians, equally distributed between males or documents online, in the case of The European Library and females, and with an age range typical of students of they are essentially in front of an online public access cat- Bachelor and Master Degree (in most cases between 19 and alogue which retrieves bibliographic records. Obtaining li- 25 years old). brary catalogue records after a search is a source of confusion which leaves the user unhappy and willing to leave the portal The analysis of the results was done in the following order: quickly. the analysis of each stream of data - i.e. HTTP log, search query log, questionnaires - was first conducted, while the Questionnaires also show that images in particular seem to analysis of possible interrelation among these sources was be very appealing for users; both the “treasures” section, a conducted later. The description of the analysis of each section which shows high resolution images of ancient doc- single stream is reported in [3], here we concentrate on the uments, and the “exhibition” section, a section which shows aspects which emerge from the correlation of the different pictures of the national libraries buildings, were thoroughly sources of information. browsed by users even before making any query in the por- tal. This is an important clue which may suggest that there Table 1 summarizes one of the important features when do- should be more linking from the images to the catalogue ing log analysis: session length. In particular, the table records. The interrelation among the information about shows how different these lengths are according to the source users who prefer images and the HTTP log and searches that is analyzed. The “Search log” column shows the statis- log is still under investigation. In fact, we would like to tics of the times, in minutes, of sessions found in the search see if this willingness expressed in the questionnaire is also logs, and between brackets the times of sessions of users reflected in user actions: for example, a user who is inter- who registered to the portal. This shows that logging on ested in images clicks more frequently on images or search is a clear intention of users who are willing to spend time for documents like maps or paintings; or a user expresses in the portal and search more, compared to random users. this interest in images but actually does not perform any The “HTTP log” column shows the times of sessions found action in the portal which confirms this interest. in the HTTP logs computed in October 2007, and between brackets the times of the sessions of users who participated 5. CONCLUSIONS in the user study at the University of Padua. In this case, The insights gained by analyzing log data together with data there is a strong bias of the students of the user study due from controlled studies are more informative than the results to the time slot which was about 30/45 minutes. The times that can be derived by separately analyzing the groups of of random users are comparable to those found in the search data. Our studies on logs combined with interviews have logs. The last column shows the times of sessions for filling- shown that the results are more scientifically informative in the questionnaires, which are obviously very similar to than those obtained when the two types of studies are con- the times of HTTP sessions of the user study. There is one ducted alone. This encouraging result constitutes the ground important aspect which emerges from the data: sessions are on which we are generalizing and formalizing starting from very short, browsing and searching activity lasts less than 2 the obtained results. A crucial feature in the future will be minutes in 50% of the cases. This particular situation can be making active use also of the information on metadata that explained only by studying the answers of the users to the are present in the log, because until now no active way of questionnaire where there are clear indications about some using them has been incorporated in the proposed method. difficulties they found in understanding how to read the list of the results, and how to use some functions of the inter- 6. ACKNOWLEDGEMENTS face. These are also the reasons why they would have left The work has been partially supported by the TELplus Tar- the portal sooner if they had not been asked to stay and fill geted Project for digital libraries, as part of the eContentplus in the questionnaire. Program of the EC, and by the TrebleCLEF Coordination Action, as part of the 7FP of the EC. An important interrelation was found among questionnaires and log data which may explain the short length of a user 7. REFERENCES [1] M. Agosti, editor. Information access through search Q. Yang. Web query translation via web log mining. engines and digital libraries. Springer, Berlin, In S.-H. Myaeng, D. W. Oard, F. Sebastiani, T.-S. Germany, 2008. Chua, and M.-K. Leong, editors, SIGIR, pages [2] M. Agosti. Log data in digital libraries. In M. Agosti, 749–750. ACM, 2008. F. Esposito, and C. Thanos, editors, IRCDL, pages [17] P. Ingwersen and K. Järvelin. The Turn. Springer, 115–122. DELOS: an Association for Digital Libraries, The Netherlands, 2005. 2008. [18] T. Koch, A. Ardö, and K. Golub. Browsing and [3] M. Agosti, F. Crivellari, and G. M. Di Nunzio. A searching behavior in the renardus web service a study method for combining and analyzing implicit based on log analysis. In H. Chen, H. D. Wactlar, interaction data and explicit preferences of users. C. chih Chen, E.-P. Lim, and M. G. Christel, editors, Workshop on Contextual Information Access, Seeking JCDL, page 378. ACM, 2004. and Retrieval Evaluation (ECIR 2009), April 2009. [19] W. Kraaij, A. P. de Vries, C. L. A. Clarke, N. Fuhr, [4] P. G. Anick. Using terminological feedback for web and N. Kando, editors. SIGIR 2007: Proceedings of search refinement: a log-based study. In SIGIR, pages the 30th Annual International ACM SIGIR 88–95. ACM, 2003. Conference on Research and Development in [5] S. M. Beitzel, E. C. Jensen, A. Chowdhury, D. A. Information Retrieval, Amsterdam, The Netherlands, Grossman, and O. Frieder. Hourly analysis of a very July 23-27, 2007. ACM, 2007. large topically categorized web query log. In [20] M. Maslov, A. Golovko, I. Segalovich, and M. Sanderson, K. Järvelin, J. Allan, and P. Bruza, P. Braslavski. Extracting news-related queries from editors, SIGIR, pages 321–328. ACM, 2004. web query log. In Carr et al. [8], pages 931–932. [6] N. Buzikashvili. An exploratory web log study of [21] J. C. Miller, G. Rae, and F. Schaefer. Modifications of multitasking. In Efthimiadis et al. [12], pages 623–624. kleinberg’s hits algorithm using matrix exponentiation [7] N. Buzikashvili. Sliding window technique for the web and weblog records. In W. B. Croft, D. J. Harper, log analysis. In Williamson et al. [31], pages D. H. Kraft, and J. Zobel, editors, SIGIR, pages 1213–1214. 444–445. ACM, 2001. [8] L. Carr, D. D. Roure, A. Iyengar, C. A. Goble, and [22] J. Parikh and S. Kapur. Unity: relevance feedback M. Dahlin, editors. Proceedings of the 15th using user query logs. In Efthimiadis et al. [12], pages international conference on World Wide Web, WWW 689–690. 2006, Edinburgh, Scotland, UK, May 23-26, 2006. [23] N. Pharo and K. Järvelin. The SST method: a tool for ACM, 2006. analysing Web information search processes. [9] J. Chen and T. Cook. Mining contiguous sequential Information Processing & Management, patterns from web logs. In Williamson et al. [31], 40(4):633–654, July 2004. pages 1177–1178. [24] S. Sekine and H. Suzuki. Acquiring ontological [10] S.-L. Chuang, H.-T. Pu, W.-H. Lu, and L.-F. Chien. knowledge from query logs. In Williamson et al. [31], Auto-construction of a live thesaurus from search term pages 1223–1224. logs for interactive web search. In SIGIR, pages [25] X. Shi and C. C. Yang. Mining related queries from 334–336, 2000. search engine query logs. In Carr et al. [8], pages [11] H. Cui, J.-R. Wen, J.-Y. Nie, and W.-Y. Ma. 943–944. Probabilistic query expansion using query logs. In [26] R. Srikant and Y. Yang. Mining web logs to improve WWW 2002, pages 325–332, 2002. website organization. In WWW 2001, pages 430–437, [12] E. N. Efthimiadis, S. T. Dumais, D. Hawking, and 2001. K. Järvelin, editors. SIGIR 2006: Proceedings of the [27] Y. Sun, K. Xie, N. Liu, S. Yan, B. Zhang, and 29th Annual International ACM SIGIR Conference on Z. Chen. Causal relation of queries from temporal Research and Development in Information Retrieval, logs. In Williamson et al. [31], pages 1141–1142. Seattle, Washington, USA, 2006. ACM, 2006. [28] J. Teevan, E. Adar, R. Jones, and M. A. S. Potts. [13] W. Gao, C. Niu, J.-Y. Nie, M. Zhou, J. Hu, K.-F. Information re-retrieval: repeat queries in yahoo’s Wong, and H.-W. Hon. Cross-lingual query suggestion logs. In Kraaij et al. [19], pages 151–158. using query logs of different languages. In Kraaij et al. [29] X. Wang and C. Zhai. Learn from web search logs to [19], pages 463–470. organize search results. In Kraaij et al. [19], pages [14] M. A. Gonçalves, G. Panchanathan, 87–94. U. Ravindranathan, A. Krowne, E. A. Fox, [30] R. W. White, C. L. A. Clarke, and S. Cucerzan. F. Jagodzinski, and L. N. Cassel. The xml log Comparing query logs and pseudo-relevance standard for digital libraries: Analysis, evolution, and feedbackfor web-search query refinement. In Kraaij deployment. In JCDL, pages 312–314. IEEE et al. [19], pages 831–832. Computer Society, 2003. [31] C. L. Williamson, M. E. Zurko, P. F. Patel-Schneider, [15] C. Grimes, D. Tang, and D. M. Russell. Query logs and P. J. Shenoy, editors. Proceedings of the 16th alone are not enough. In E. Amitay and C. G. M. J. International Conference on World Wide Web, WWW Teevan, editors, Query Log Analysis: Social And 2007, Banff, Alberta, Canada, 2007. ACM, 2007. Technological Challenges. A workshop at the 16th [32] Z. Zhang and O. Nasraoui. Mining search engine International World Wide Web Conference (WWW query logs for query recommendation. In Carr et al. 2007), May 2007. [8], pages 1039–1040. [16] R. Hu, W. Chen, P. Bai, Y. Lu, Z. Chen, and