This paper describes the collaborative participation of Dublin City University and Trinity College Dublin in LogCLEF 2010. Two sets of experiments were conducted. First, di erent aspects of the TEL query logs were analysed after extracting user sessions of consecutive queries on a topic. The relation between the queries and their length (number of terms) and position ( rst query or further reformulations) was examined in a session with respect to query performance estimators such as query scope, IDF-based measures, simpli ed query clarity score, and average inverse document collection frequency. Results of this analysis suggest that only some estimator values show a correlation with query length or position in the TEL logs (e.g. similarity score between collection and query). Second, the relation between three attributes was investigated: the user's country (detected from IP address), the query language, and the interface language. The investigation aimed to explore the in uence of the three attributes on the user's collection selection. Moreover, the investigation involved assigning di erent weights to the three attributes in a scoring function that was used to re-rank the collections displayed to the user according to the language and country. The results of the collection re-ranking show a signi cant improvement in Mean Average Precision (MAP) over the original collection ranking of TEL. The results also indicate that the query language and interface language have more in uence than the user's country on the collections selected by the users.
LogCLEF at the Conference on Multilingual and Multimodal Information Access Evaluation 2010 is an initiative to analyse search logs and discover patterns of multilingual search behaviour. The logs come from user interactions with the European Library (TEL)3, which is a portal forming a single interface for searching across the content of many European national libraries.
LogCLEF started as a task of the CLEF 2009 evaluation campaign [
In our LogCLEF 2010 participation, two sets of experiments were carried out on action logs from TEL. The rst set of experiments investigated the assumption that query performance predictors not only can be used to estimate performance, but can also be utilised to analyse real user behaviour. For example, users' query modi cations generally aim at improving the results and should re ect the fact that the users' queries improve over successive reformulations. This should also result in di erent values for performance estimates.
The second set of experiments were concerned with studying three attributes
related to the users and the queries they submit, namely: the user's country (i.e.
the location from which the query was submitted), the language of the
submitted query, and the speci ed interface language by the user. The experiments
investigated the in uence of the three attributes on the collections (libraries or
online resources) selected by the user. In other words, the experiments aimed
at answering the question of whether or not the users' choice of collections is
in uenced by their location and language. The experiments involved re-ranking
the list of collections displayed to the user. Typically, result re-ranking involves
re-ranking a list of documents. However, in this study, the re-ranking process
was applied to the list of collections displayed to the user by TEL. A scoring
function that comprises the three attributes was used to determine the relevance
of each collection to the current search in terms of country and language.
Reranking alternatives were investigated by assigning di erent weights to the three
attributes in the scoring function. The results of the collection re-ranking
experiments showed a 27.4% improvement in Mean Average Precision (MAP) [
The rest of this paper is organised as follows: Section 2 introduces related work; Section 3 describes preprocessing of the log les for session reconstruction; Section 4 introduces query performance predictors and details the analysis carried out with these measures on the query logs; Section 5 describes the collection re-ranking experiments and their results; and nally, conclusions and outlook on future work are given in Section 6.
Query performance estimators have been widely used to improve retrieval e
ectiveness by predicting query performance [6{9], or to reduce long queries [
Typically, these estimators are applied to user queries for automatic query modi cation, either before retrieval (i.e. pre-retrieval), based on information available at indexing time, or after an initial retrieval (i.e. post-retrieval), in which case additional information such as relevance scores for the documents can be employed as features for the performance estimators. The main objective is to improve e ectiveness of information retrieval systems in general or for speci c topics. However, these estimators have { to the best of the authors' knowledge { not yet been applied to real user queries to investigate if real query reformulations actually improve the search results. 2.2
Result re-ranking is one of the well-known techniques used for search personalisation. [12{14]. Result re-ranking adapts to the user's needs in that it brings results that are more relevant to him/her to higher positions in the result list. It takes place after an initial set of results have been retrieved by the system, where an additional ranking phase is performed to re-order the results based on various adaptation aspects (e.g. user's language, knowledge, or interests).
Typically, the re-ranking process is applied to the list of retrieved documents. However this study investigates applying the re-ranking process to the list of collections displayed to the user, not the list of documents within a collection. 3
Two datasets (action log les) were provided at LogCLEF: the rst one (L1) contained action logs (queries and user interactions) from January 2007 to June 2008 from the TEL web portal (approximately 1.8 million records), and the second one (L2) contained action logs from January 2009 to December 2009 (approximately 76 thousand records). In addition, a third dataset (HTTP logs) was provided, which contained more details of the search sessions and HTTP connections, including the list of collections searched for each submitted query.
For the rst set of experiments, queries from the two datasets of action logs
were used. The logs were pre-processed following the approach in our previous
experiments [
Preliminary experiments on the TEL data showed that only a few URLs
from the logs were functional. Retrieving result pages for queries via their URL
results in error codes of "not found" (HTTP response code 404). Furthermore,
referrer URLs seem to be truncated to a certain number of characters, yielding
malformed URLs. Thus, an external resource with reasonable coverage was used
to compute query statistics (e.g. term frequencies and collection frequencies).
The DBpedia4 collection of Wikipedia article abstracts was indexed and the
queries were evaluated against this index. The Lucene toolkit5 was employed to
index the document collection, using the BM25 model [
For the second set of experiments, data from both the action logs and the HTTP logs of TEL 2007 was used. Due to time limitation and some technical errors in the recorded HTTP logs, the experiments were only conducted on a subset of the data. The selected subset comprised the submitted queries and clicked collections during the month of February 2007. This included approximately 1,800 queries from di erent languages. 4 4.1
Several pre-retrieval performance predictors were employed. Pre-retrieval means
that the computation does not rely on calculating relevance scores or other
postretrieval information. Performance predictors are described in more detail in [
Query length. For our experiments, the query length is de ned as the number of
unique query terms after stopword removal (see Equation 1). Zhai and La erty
[
QL = number of unique non-stopword terms (1) where q is the query.
Query scope (QS). The query scope tries to measure the generality of a query
by its result set size [
QS =
log(nq=N )
where nq is result set size and N is total number of documents in the collection.
IDF based score (IDFmm). IDF mm denotes the distribution of informative
amount of terms t in a query. This IDF -based feature is introduced as 2 by
He and Ounis [
log2(N + 1) IDF mm = idfmax idfmin where t is a term and Nt is the number of documents containing t. In the IDF mm formula (see Equation 4), idfmax and idfmin are the maximum and minimum IDF of terms from the query q.
SCQS =
X(1 + ln(tfcoll)) ln(1 + t2q
N Nt ) where Nt is the number of documents containing term t. (2) (3) (4) (5) (6) Simpli ed query clarity score (SCS). The query clarity is inversely proportional to the ambiguity of the query (i.e. clarity and ambiguity are contrary concepts).
This score measures ambiguity based on the analysis of the coherence of
language usage in documents whose models are likely to generate the query
[
SCS = Pt2q Pml(tjq) log2 PPmcoll(lt(jtq))
' Pt2q qqtlf log2 tfcollq=ttfo=kqelncoll
where Pml(tjq) is the maximum likelihood of the query model of term t in query
q and Pcoll(t) is the collection model. tfcoll is the collection frequency, tokencoll
is the number of tokens in the whole collection, qtf is the term frequency in the
query, and ql is the query length (the number of non-stopwords in the query).
Similarity score between collection and query (SCQS). Zhao, Scholer, et al. [
Average inverse document collection term frequency (avICT F ). Kwok [
The denominator ql is the reciprocal of the maximum likelihood of the query model of SCS in Equation 5. The use of avICT F is similar to measuring the divergence of a collection model (i.e. ICT F ) from a query model. Thus, avICT F and SCS should have similar query performance estimates. 4.2
For the rst part of our experiments on the TEL log data, the relation between real user queries and the query performance estimators are investigated. All searches from the reconstructed sessions from L1 and L2 were conducted on the indexed DBpedia abstract collection and the average values for all query performance estimators were calculated. Due to data sparsity, some results were grouped by query length (number of terms in a query) and by position (whether a query is the rst one submitted in a session or is a further reformulation) into bins. Moreover, session estimates were analysed by distinguishing between sessions beginning with a short query and those beginning with a long query. This was performed with the aim of investigating if the rst query in a session indicates the search behaviour during that session. For example, users may expand short queries and reduce long queries if their initial search is not successful.
For the experiments, short queries were de ned as queries containing one to
three terms and long queries were de ned as queries with more than three terms.
This value roughly corresponds to the well-known average number of terms in
web queries (2-3 terms) [
Comparing all six query performance estimators investigated (i.e. QL, QS, IDF mm, SCS, avICT F , and SCQS) to the query length, we nd that, unsurprisingly, there is a high correlation between the query length (non-stopword terms) and QL (unique non-stopword terms). Query scope (QS) increases slightly for queries containing more than 3 terms, but remains almost constant for longer queries. IDF mm slightly increases for longer queries. SCS has its maximum for the shortest queries (length 1-3 terms) decreases for medium queries, and increases again for most longer queries. Finally, avICT F also decreases for medium-length queries, with higher values for short and long queries (see Figure 1). All estimators show similar behaviour on the sessions of L1 and L2.
Next, the estimator values were calculated and compared across di erent positions in the sessions (see Figure 2). The positions correspond to users' query reformulations. The sessions from L1 are longer than sessions from L2 (14 queries maximum for L1 vs. 10 queries maximum for L2). With a few exceptions, most estimators have almost constant values for the di erent reformulations. One of the exceptions is avICT F , which drops after position 3 and has a sudden, but constant maximum after position 12. However, SCS increases after position 7 in both data sets. SCQS compared to the query position shows that with a higher number of queries in a session (i.e. more query reformulations), SCQS drops consistently (see Figure 4 left).
Finally, dividing the data into sessions starting with a short query (less than 4 terms) and sessions starting with longer queries reveals the most interesting results. QL, QS, and IDF mm are higher for longer queries in both data sets investigated, while SCS and avICT F are higher for shorter queries (see Figure 3). The highest di erence can be observed for SCQS (see Figure 4 right) which shows much lower values for sessions with initial long queries. 4.3
Most of the query performance predictors we experimented with the aim of measuring clarity, speci city, or ambiguity of query terms. For example, query length is presumed to be an indicator of result quality because the more terms a query contains, the more speci c the query is, i.e. the more precise the results should be. However, our experiments did not show that longer queries are consistently less ambiguous, or more speci c (Figure 1).
Our next analysis (Figure 2) was concerned with query reformulations in a single session. We investigated whether queries become more successful after reformulation, i.e. if a user's query reformulation will achieve better performance (estimates). We also have not observed a simple or direct relationship between the query position and the performance estimates. Instead, the extracted user sessions might contain data from users with di erent search strategies or data from users changing their search strategy in the middle of a session. This may be an explanation for sudden drops and increases in some performance estimators (see, for instance avICT F in Figure 2 on the L1 data on the left).
Finally, SCQS showed the most interesting behaviour. For longer queries, its value decreases consistently for both data sets. With higher query positions, its value typically decreases, and the di erences for sessions beginning with short and long queries are huge.
A small sample of sessions was manually investigated in more detail. Several users reformulate their queries by using the same search terms in di erent elds (for the advanced eld-based search) or to search in di erent indexes (e.g. "lima licinio" repeated for di erent elds). This type of modi cation might not actually improve the retrieved results. However, the current TEL portal already supports a catch-all eld ("any eld"), i.e. a eld covering a search in all indexes. Possibly this eld was not available at the time the query logs were collected.
Some users modify their original query, examine the results, and submit the original query again (e.g. "environmental comunication" is modi ed into "comunication" which is modi ed back to "environmental comunication" { note the spelling error in all queries). A similar behaviour can be observed for longer sessions. After several queries, the users seem to revert to previously submitted queries, probably because the TEL portal does not provide query suggestions and the users do not nd relevant or new information allowing them to change their query formulation.
Finally, some queries contain more than one change at once (e.g. \music manuscripts anton bruckner" is changed to \bruckner anton sinfonie"). Thus, changes in performance predictors might also incorporate several changes (i.e. increases and decreases in quality of results) at once. 5 5.1
When users submit a search to the TEL portal, they are presented with a list of collections on the left side of the screen and a list of results from the selected collection on the right side of the screen. A collection is either a library or an online resource that is associated with a certain country. The list of collections is presented to the user in alphabetical order of the countries' acronyms (two letters according to the ISO 3166-1 standard).
The users' queries and interactions with the portal were recorded in two datasets called: action logs and HTTP logs. This set of experiments involved studying di erent aspects regarding three attributes exhibited in the logs: user's country (location from which the query was submitted), query language, and interface language. In our experiments, the user's country was determined from the IP address recorded in the logs. The query language was detected using the Google AJAX Language API6, which returns a con dence level associated with
the detected language of a query. Only queries that had a minimum value of 10% con dence level were used in the experiments, which reduced the number of queries subject to experimentation to 566 queries (from the 1,800 queries of February 2007).
Furthermore, the experiments involved studying the list of collections that were searched for each query, and the collections that the user clicked on. The study investigated re-ranking the list of collections displayed to the user with the aim of increasing the retrieval precision (across the collections, not the actual result documents) by bringing collections that match the user's country or language to higher ranks in the list. In other words, the re-ranking process focused on studying if users have a higher tendency to click on collections that belong to their country or language. For this part of the experiments, languages were assigned to a collection based on the o cial languages that are spoken in the country associated with that collection. 5.2
The following statistics are drawn from the selected subset of the logs. Figure 5 shows the distribution of users' countries on the left (i.e. countries that the queries came from) and the language distribution of queries on the right. It is noted that a large percentage (50%) of the queries were in English although the total percentage of queries coming from English-speaking countries (United States, United Kingdom, and Canada) was only 16%. This suggests that many users from non-English countries do not submit queries in their native language.
Figure 6 shows the relation between user's country, query language, and
interface language for all queries (left) and for non-English queries (right).
Surprisingly, it is noted that only 24% of the queries coming from a country are
in languages associated with that country. This is not necessarily an inclination
towards using English in search, as similar results were observed when
studying non-English queries, where less than 30% of the queries were submitted in
a language that matched the country. Therefore, this suggests that the
country attribute may not be very reliable to base the collection re-ranking decision
upon. This was con rmed by further experiments (discussed below).
In order to evaluate the retrieval precision over the list of collections presented
to the user, we used the collections that the user clicked on as implicit relevance
judgements (i.e. binary relevance judgements where the clicked collections are
assumed to be the relevant ones, and non-clicked collections are assumed to be
irrelevant). This follows on the method adopted in [
Mean Average Precision (MAP) was used for evaluation as it is a common
evaluation metric in information retrieval that rewards relevant items being
ranked higher in the list [
A result (collection) scoring function was used to re-rank the list of collections. The function was based on matching the three attributes with the collection's country and language as follows (where matching=1 and non-matching=0): 1. Mc: matching the user's country with the collection's country. 2. Mq: matching the query's language with the collection's language (i.e. matching with any of the o cial languages spoken in the corresponding country). 3. Mi: matching the interface language with the collection's language.
Each of the above attributes is multiplied by a scalar weight (Wc, Wq, Wi respectively) so as to control (and test) the degree of contribution of each attribute in the function. Thus the collection scoring function becomes Equation 8 N ewScore = (Wc
Mc) + (Wq
Mq) + (Wi Mi) (8) Finally, the collections are re-ranked based on descending order of the new score.
Unlike other collections, collection #a0000 had no country associated with it (a virtual collection of various European digitised books, images, maps, etc.). This would have caused it to be permanently placed at the end of the re-ranked list (i.e. get a zero score from the scoring function) because of not matching any country or language. Therefore, an exception was made regarding this collection; it was placed at the top of the re-ranked list (similar to TEL's ranking). Table 1 shows the MAP value for some selected re-ranking runs with alternative combinations of weights that ranged from 0.0 to 1.0. The results showed a signi cant improvement of 27.4% in retrieval precision for the re-ranked collection lists (with weights: Wc=0.1, Wq=0.3, Wi=0.6) over the baseline ranking. This improvement is statistically signi cant as per the T-test (with p=0.01) and Wilcoxon test (with con dence=99%). The results also suggest that there is a relation between the query language and interface language on the one hand and the clicked collections on the other hand. Moreover, these two attributes seem to have more in uence than the user's country (location from which the query was submitted) on the selected collections by the users.
The experiments using query performance predictors show that SCQS has a high correlation with query length and query position. Distinguishing between sessions with an initial short or long query seems to indicate a di erent subsequent user behaviour. For example, users starting with a short query are more likely to expand their query to obtain more speci c results, while users starting with a longer query probably reduce their query because the retrieved result set is too small. To obtain more conclusive results, we want to perform more experiments on larger query logs. The results might also di er from results obtained from performing the same analysis on web search logs, because of the di erent application domain (bibliographic search in the TEL portal), the slightly longer queries and possibly longer sessions, which include more query reformulations. Query performance estimators have been developed for web search and may thus be optimised for shorter queries and fewer search iterations. In conclusion, more consistent results may be found if the methods are applied on a larger data set.
The experiments of collection re-ranking based on matching the query language and the interface language with the collection's language showed a signi cant improvement in precision. This suggests that there is opportunity for improving the user's experience with multilingual search in the TEL portal if the user's language is taken into consideration. Future work will involve using machine learning techniques to learn a ranking function that optimises the weights of the three attributes: country, query language, and interface language. Moreover, we will attempt to work around the technical errors present in the dataset of TEL HTTP logs, which will enable us to conduct the experiments on the full dataset.
This research is supported by the Science Foundation of Ireland (grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at Dublin City University and Trinity College Dublin.