Log data constitutes a relevant aspect in the evaluation process of multilingual search services. Activity logs allow to study the usage of search engines and to better adapt them to the needs of their users. The study of multilingual log analysis was promoted by the Cross Language Evaluation Forum (CLEF). For the second time, the track LogCLEF was conducted. As is 2009, large log files were obtained from information providers. One log covers 30 months of activities on the website of The European Library (TEL) and the second log shows user activities of users on the German EduServer. Seven groups explored the data using a variety of approaches. They analyzed languages of queries, activities within sessions and success of searches. The data for the track, the evaluation methodology and results are presented and discussed.
Web Search Engines deal with the representation, storage, organization of, and access to information items which are essentially Web pages. The characterization of the user information need is not simple, and this problem can roughly be divided into three aspects: how the user poses his request to the search engine, how the user interacts with the search engine, and how the search engine organizes the results.
Log data constitute a relevant aspect in the evaluation process of the quality of a
search engine and the quality of a multilingual search service; log data can be used to
study the usage of a search engine, and to better adapt it to the objectives the users
were expecting to reach [
The interest in multilingual log analysis was promoted by the Cross Language Evaluation Forum (CLEF)1 in the track LogCLEF2 which was conducted for the first
2 http://www.uni-hildesheim.de/logclef/
time in 2009 [
LogCLEF differs from other evaluation tracks since its goal is not the production of a gold standard for a specific task, but to create a forum for the creative exploration of user behavior based on logs.
The data sets used in 2010 were activity logs derived from the The European Library (TEL) Web site3 and the German EduServer4 -- Deutscher Bildungsserver (DBS) -- maintained by the DIPF, the Leibniz Institute for Educational Research and Educational Information. The task definition, the data for the track, the evaluation methodology and some results of submitted experiments are presented in this overview paper.
The main question behind the task definition comes from search service providers who wonder how they can improve their services. Ultimately, researchers need to better understand user behavior in order to reach that high level goal. Two objectives of the analysis of the logs are proposed, one for each set of log files:
TEL: Investigate language of queries with respect to successful search sessions. A successful search could be defined as one of the following actions listed in the right hand box of the TEL interface when an item of the result clicked is listed. + Services:
Availability at the library, Link to other services, collection homepage + Options:
Save in favorites,
Send by email.
1. language identification for the queries 2. initial language vs country IP address 3. subsequent languages used on same search 4. country of the library vs language of the query vs
language of the interface
DBS: The objective of the analysis of the DBS logs is the exploration of the relation between query and viewed content. The analysis can explore formal issues of
4 http://www.eduserver.de/ the query and content as well as the distribution of words within both.
Potential research issues for DBS: 1. Are query terms related to the content viewed and/or paths taken within the system? 2. Can query modifications be explained by the content viewed? 3. Develop metrics to identify successful searches
The data for LogCLEF 2010 collection consists of two large log files from information providers: • The European Library (TEL) logs: As in 2009, a large log of activities from The European Library are provided. This service provides access to several national libraries of Europe. Users and content come from many languages. • German EduServer (Deutscher Bildungsserver, DBS) logs: The "Deutscher Bildungsserver" is a quality controlled internet directory for educational resources. A raw server log representing three months of activities on the portal is made available. The size of all files is 5 GB.
The following table gives an overview on the log resources which were been made available at CLEF over the last years. TEL is a free service that offers access to the resources of 48 national libraries of Europe in 35 languages, it aims to provide a vast virtual collection of material from all disciplines and offers interested visitors simple access to European cultural heritage. Resources can be both digital (e.g. books, posters, maps, sound recordings, videos) and bibliographical and the quality and reliability of the documents are guaranteed by the 48 collaborating national libraries of Europe.
The data used for this task are search logs and Web server logs of The European Library portal.
Search logs are usually named “action logs” in the context of TEL activities. In TEL portal’s home page, a user can initiate a simple keyword search with a default predefined collection list presenting catalogues from national libraries. From the same page, a user may perform an advanced search with Boolean operators and/or limit search to specific fields like author, language, and ISBN. It is also possible to change the searched collection by checking the theme categories below the search box. After the search button is clicked, the result page appears, where results are classified by collections and the results of the top collection in the list are presented with brief descriptions. Subsequently, a user may choose to see result lists of other collections or move to the next page of records of current collection’s results. While viewing a result list page a user may also click on a specific record to see detailed information about the specific record. Additional services may be available according to the record selected.
All these type of actions and choices are logged and stored by TEL in a relational
table, where each record represents a user action [
The Web server log files of TEL cover the same period of the first data set of action logs, from January 2007 until June 2008. These log files are saved in 18 text files date: year-month-day. time: hour:minute:second.
HTTP method: for example GET, HEAD, POST, etc.
URI stem: the path of the requested file.
URI query: the string of the query in the URL, if any.
IP address: the address of the client, (obfuscated, e.g. 127.0).
User agent: the user agent of the client.
Cookie: the cookie sent to/by the client.
Referrer: the URL of the resource which linked the client to TEL.
The Cookie field is divided into subfields by semi-colons “;”. Some of the subfields are (some of them are ignored for this task): • cTargets: the identifiers of the collections selected by the user; • TELSESSID: the identifier of the session. It is the same identifier recorded in the acion logs under the name “sesid”. This is an important field to crossanalyze action logs to Web server logs. Figure 1, shows an example of how a user session may be stored in the two different logs. The quality controlled "Deutscher Bildungsserver" is a clearinghouse for educational resources on the Web5. It also contains content provided by the DIPF as well as 5 http://www.bildungsserver.de/start_e.html descriptions and reviews on Web sites on education. The Internet resources (web sites) are described, checked for their quality, manually indexed and classified. The logs were collected in the time between September and November of 2009. The logs are server logs in standards format in which the searches and the results viewed can be observed. An excerpt is shown in table 2. The logs have been anonymized by partially obscuring the IP addresses of users.
The two upper levels of server names or IP addresses have been hashed. This allows the reconstruction of sessions within the data. Note that accesses by search engine bots are still within the logs. The logs allow to observe two types of user queries: • • queries in search engines (in the referrer when DBS files were found using a search engine) queries within the DBS (see query parameters in metasuche/qsuche) The logs also allow so observe the browsing behavior within the DBS server structure. The following pages are of most interest: • the descriptions of the educational web sites within DBS (mlesen) thematic lists of educational web sites (zeigen, anzeigen, fachlist, listen) a newspaper documentation on articles about education (zeitdok) the descriptions of the educational web sites within DBS The logs allow to access two types of content and compare them to the queries. • the content of the educational web sites themselves (which might have changed since the logs have been collected) in those cases where the user might have accessed them
The two following sections shows the participants of LogCLEF 2010 and presents some results. For more detailed results, the reader is referred to the papers by the participants which describe the approaches and findings in more detail. 4.1
As shown in Table 4, a total of 7 groups submitted results for LogCLEF. Of the 15 registered groups, only less than 50% managed to obtain results. The results of the participating groups are reported in the following section and elaborated in the papers of the participants. All groups analyzed the TEL logs and none worked with the DBS logs. This might be due to the nature of a raw web server log which requires much pre-processing. LogCLEF could not provide a pre-processed version due to the lack of funding for LogCLEF.
DAEDALUS
4.2
A large variety of approaches was taken to analyze the TEL log files. This can be considered as a success of the open definition of the task which encouraged creative exploration of the data.
Two groups contrasted user behavior at a quality search service like TEL to
common Web search behavior. A group from The Netherlands under the leadership of
the University of Nijmegen contrasted frequent queries and number of queries per
session in the TEL log with data from an MSN log [
The NII from Tokio also compared the TEL logs to web search logs and theories
developed by exploiting web search [
The DAEDALUS group formally defined success for sessions and queries. They calculated that only 6% of the queries and 10% of the sessions could be labeled as successful.
Three groups focused on language issues. The SINAI group showed that most of the sessions are in English. They also conclude that 50.000 of the sessions exhibit only one action. More than 80% of the sessions have 10 or fewer actions.
The CELI research institute tried to identify the language of search queries [
The difficulties of language identification were elaborated by a group from Berlin
[
A group from Dublin [
Studies on log files are limited by privacy issues. For the first time, LogCLEF provided evaluation resources for log file analysis which can be used for comparative system evaluation. The second year of LogCLEF obtained more attention by participants. It is intended to encourage and facilitate the exchange of resources and tools generated within the participation at LogCLEF.
In the future, log analysis should be the basis for other evaluation tasks. Logs can
show how users behave and what they need. One example could be the selection of
topics for retrieval evaluation or for questions answering systems [