In recent years, the importance of log analysis has grown, log data constitute a relevant aspect in the evaluation process of the quality of a digital library system. In this paper, we address the problem of log analysis for complex systems such as digital library systems, and how the analysis of search query logs or Web logs is not su±cient to study users and interpret their preferences. In fact the combination of implicitly and explicitly collected data improves understanding of behavior with respect to the understanding that can be gained by analyzing the sets of data separately.
The interaction between the user and an information access system can be analyzed and studied to gather user preferences and to \learn" what the user likes the most, and to use this information to personalize the presentation of results. User preferences can be learned explicitly, for example asking the user to ¯ll-in questionnaires, or implicitly, by studying the actions of the user which are recorded in the search log of a system. The second choice is certainly less intrusive but requires more e®ort to reconstruct each search session a user made in order to learn his preferences. ¤Copyright is held by the author/owner(s).
SIGIR'09, July 19-23, 2009, Boston, USA.
Log is a concept commonly used in computer science; in fact, log data are collected by programs to make a permanent record of events during their usage. The log data can be used to study the usage of a speci¯c application, and to better adapt it to the objectives the users were expecting to reach. In the context of the Web, the storage and the analysis of Web log ¯les are mainly used to gain knowledge on the users and improve the services o®ered by a Web portal, without the need to bother the users with the explicit collection of information.
When research addresses the problem of studying log data
in digital libraries, which are very complex systems,
di®erent characteristics regarding library automation systems and
digital library systems need to be taken into account. In fact,
for all the di®erent categories of users of a digital library
system, the quality of services and documents the digital library
supplies are very important. Log data constitute a relevant
aspect in the evaluation process of the quality of a digital
library system and of the quality of interoperability of digital
library services [
This paper deals with the study of complementary types of logs in complex systems with the aim of ¯nding new ways of using them to evaluate and personalize digital library services for the ¯nal users. The paper is organized as follows: Section 2 presents previous related work, Section 3 analyzes and presents di®erent facets of the study and use of logs of complex systems, Section 4 presents the ¯ndings of the case study conducted in the context of the TELplus project1 for the evaluation and personalization of the services of The European Library, and lastly Section 5 draws conclusions and indicates directions for the continuation of the work.
In the last decade, log analysis has become one of the main threads of research for understanding users of search engines as shown by the works presented at three major relevant conferences and that have been analyzed by us2. Those works study logs in di®erent ways and for di®erent
2The three analyzed major conferences are: SIGIR - http://www.sigir.org/ WWW - http://www.iw3c2.org/ JCDL - http://www.jcdl.org/ purposes, but they can be divided into two main classes: studies about search query logs, and studies about Web server logs. Since most of these research papers concern search engines, the focus of their research is more on improving queries and results and less on sur¯ng the Web. The few exceptions to this classi¯cation will be analyzed later in the paper.
Query search logs can be used for: building knowledge, such
as automatically building a search thesaurus [
There is also a recent emerging research activity about log
analysis which tackles cross-lingual issues: [
Present digital library systems are complex software
systems, often based on a service-oriented architecture, able to
manage complex and diversi¯ed collections of digital objects.
One signi¯cant aspect that still relates present systems to
the old ones is that the representation of the content of the
digital objects that constitute the collection of interest is
still done by professionals. This means that the
management of metadata can still be based on the use of authority
control rules in describing author, place names and other
relevant catalogue data. A digital library system can exploit
authority data that keep lists of preferred or accepted forms
of names and all other relevant headings. This is a
dramatic di®erence between digital library systems and search
engines, and it is usually overcome with the analysis of log
data. In fact a search engine often becomes a speci¯c
component of a digital library system, when the digital library
system faces the management and search of digital objects
3http://www.trebleclef.eu/
by content in the same manner as information retrieval
systems and search engines [
Moreover, log analysis can be supported and validated by
user studies which are a valuable method for understanding
user behavior in di®erent situations. User studies require
a signi¯cant amount of time and e®ort, so an accurate
design of the process has to be carried out. In general, user
studies and logs are used in a separate way, since they are
adopted with di®erent aims in mind. Ingwersen and JaÄrvelin
report in [
The combination of implicitly and explicitly collected data improves understanding of behavior with respect to the understanding that can be gained by analyzing the sets of data separately. In particular for digital libraries, where the evaluation of the di®erent services is di±cult if logs are used alone, the combined sets of data provide the opportunity of reaching insights towards user personalization of digital library services.
From this starting point we have developed a method for collecting data derived from the user interaction log, \implicit" data, and data collected from user questionnaires, \explicit" data, for analyzing the interaction between users and digital libraries. This means that the conceived method is based on the combination and analysis of the following data sources: HTTP log which contains the HTTP requests sent by the Web client to the Web server during a user browsing session; search log which contains the actions performed by the user during a search; questionnaire data which are collected at the end of a user browsing and searching session. The possibility of studying and correlating di®erent sources of data was envisaged during the study of the Web portal of The European Library4, which provides a vast virtual collection of material from all disciplines and o®ers interested visitors simple access to European cultural heritage.
The European Library is a free service that o®ers access to the resources of 48 national libraries of Europe in 20 languages with about 150 million entries across Europe. The European Library provides a vast virtual collection of material from all disciplines and o®ers interested visitors simple access to European cultural heritage.
To validate the proposed method, a study was conducted in a controlled setting at the end of 2007 { beginning of 2008, in the computer laboratories of di®erent faculties of the University of Padua, Italy, where students were requested to conduct a free navigation and search for information on The European Library portal and to ¯ll in a questionnaire specifically designed to harvest the data that can be used to extract information on users satisfaction on the use of di®erent parts of the portal. A total of 155 students participated in the study, mostly Italians, equally distributed between males and females, and with an age range typical of students of Bachelor and Master Degree (in most cases between 19 and 25 years old).
The analysis of the results was done in the following order:
the analysis of each stream of data - i.e. HTTP log, search
query log, questionnaires - was ¯rst conducted, while the
analysis of possible interrelation among these sources was
conducted later. The description of the analysis of each
single stream is reported in [
Table 1 summarizes one of the important features when doing log analysis: session length. In particular, the table shows how di®erent these lengths are according to the source that is analyzed. The \Search log" column shows the statistics of the times, in minutes, of sessions found in the search logs, and between brackets the times of sessions of users who registered to the portal. This shows that logging on is a clear intention of users who are willing to spend time in the portal and search more, compared to random users. The \HTTP log" column shows the times of sessions found in the HTTP logs computed in October 2007, and between brackets the times of the sessions of users who participated in the user study at the University of Padua. In this case, there is a strong bias of the students of the user study due to the time slot which was about 30/45 minutes. The times of random users are comparable to those found in the search logs. The last column shows the times of sessions for ¯llingin the questionnaires, which are obviously very similar to the times of HTTP sessions of the user study. There is one important aspect which emerges from the data: sessions are very short, browsing and searching activity lasts less than 2 minutes in 50% of the cases. This particular situation can be explained only by studying the answers of the users to the questionnaire where there are clear indications about some di±culties they found in understanding how to read the list of the results, and how to use some functions of the interface. These are also the reasons why they would have left the portal sooner if they had not been asked to stay and ¯ll in the questionnaire.
An important interrelation was found among questionnaires and log data which may explain the short length of a user session. One of the outcomes of the questionnaire was the disorientation of the user upon entering The European Library portal for the ¯rst time, in particular it seems not to be clear what kind of information can be accessed through this portal. Users are in general ready to search in a Google-like fashion and obtain documents, in terms of links to pages or documents online, in the case of The European Library they are essentially in front of an online public access catalogue which retrieves bibliographic records. Obtaining library catalogue records after a search is a source of confusion which leaves the user unhappy and willing to leave the portal quickly.
Questionnaires also show that images in particular seem to be very appealing for users; both the \treasures" section, a section which shows high resolution images of ancient documents, and the \exhibition" section, a section which shows pictures of the national libraries buildings, were thoroughly browsed by users even before making any query in the portal. This is an important clue which may suggest that there should be more linking from the images to the catalogue records. The interrelation among the information about users who prefer images and the HTTP log and searches log is still under investigation. In fact, we would like to see if this willingness expressed in the questionnaire is also re°ected in user actions: for example, a user who is interested in images clicks more frequently on images or search for documents like maps or paintings; or a user expresses this interest in images but actually does not perform any action in the portal which con¯rms this interest.
The insights gained by analyzing log data together with data from controlled studies are more informative than the results that can be derived by separately analyzing the groups of data. Our studies on logs combined with interviews have shown that the results are more scienti¯cally informative than those obtained when the two types of studies are conducted alone. This encouraging result constitutes the ground on which we are generalizing and formalizing starting from the obtained results. A crucial feature in the future will be making active use also of the information on metadata that are present in the log, because until now no active way of using them has been incorporated in the proposed method.
The work has been partially supported by the TELplus Targeted Project for digital libraries, as part of the eContentplus Program of the EC, and by the TrebleCLEF Coordination Action, as part of the 7FP of the EC. 7.