In this paper, a sample set of 510 simple searches from the TEL action log 2009 is analyzed for query content and query language. More than half of the queries are for named entities, which has consequences for query language disambiguation. A manual identification of query language finds that often a definite language cannot be determined, because many named entities are not translated. Problems and challenges for query category and language identification are discussed. Further analysis shows that IP address and interface language are not very strong indicators for determining the query language.
One of the challenges in cross-lingual information retrieval systems is to identify the user‟s information need which is expressed in short and decontextualized queries in multiple languages. To be able to process the query adequately (e.g. stem or translate correctly), it is essential to determine the language of the query.
In this paper, we explore what the challenges for query language identification and classification are, especially in the context of digital libraries. We extracted 510 search queries from the TEL action log corpus 2009 and analyzed them on a conceptual and linguistic level. Of special interest was the identification of query characteristics from the corpus. We examine the ambiguity of query terms and the resulting challenges in determining the query language.
We also looked at other signals, which might help to determine the language of a query such as the IP address or the interface language. An analysis of the relationship between interface language, IP address and country of origin of users and the query language was carried out.
With our analysis we follow up on results generated from the TEL corpus in the
previous LogCLEF track in 20091, which is briefly summarized here. Data from The
European Library (TEL) and Tumba! were evaluated with the aim to analyze and
classify user queries in order to understand search behavior in multilingual contexts
and to improve search systems [
The paper is organized as follows: Section 2 briefly describes the TEL simple search log corpus and provides some general statistics. In Section 3 we present and discuss the analysis of our sample query corpus introducing categories for this sample of digital library queries. Section 4 deals with the problems in query category and language detection and provides examples for characteristic difficulties. We conclude the paper by investigating signals for language identification analysis that includes information about the interface language and the language information of the country the searcher is from.
For LogCLEF 2010, two corpora are provided. The first one contains logs from the Deutsche Bildungsserver, the second one logs from The European Library (TEL). We created a multilingual test corpus with TEL queries from the simple search interface. Log files from two different periods were available. In the first period from January 2007 to June 2008, action log files and server logs could be used for research. We analyzed the second set of data, which were action logs from the period of January to December 2009.
This one-year log data file contained 762,485 lines of log entries. We extracted queries, which either contained a simple search or an advanced search indicator (search_sim / search_adv) in the log file entry. Queries, which were entered on the result page or the full record view were not selected. Log entries, which contained a simple search made up 137,827 of the entries, advanced search 32,528 entries.
As the advanced search offers some categorizations of the query by adding certain facets like” title” or “author” we used only simple search queries for our sample
corpus as they do not give any context information regarding the intent of the user. Therefore, the query entered by the user and the information saved in the logs are the only signals the system can get. 2,guest,127.0,5E390977758E505C871AFB99E1342988,en,"(""t oto"")",search_sim,"/en/search/collections/a0268,a0365/ ",0,,,,2009-01-28 00:00:00.0 The query is shown in the sixth column field of the log entry (see Figure 1).
As we extracted a sample corpus from the simple search queries, which initiated a
search session, we also looked at the entire simple search queries to extract
information about the average length of the queries. One important aspect is the
length of the queries which can indicate the amount of textual information embedded
in a query. Early studies investigated the query length in web search engines, for a
comparison of the major studies see Jansen & Poosch [
A first analysis of queries in the cultural heritage domain was conducted by Jones
et al. [
We analyzed the words per query separated by white space in the simple search corpus. On average, the queries consist of 2.34 terms. Approximately, 43% of all simple search queries contain only one term, 27% two terms, 13% three terms, 7% four terms and only 10% of all queries contain 5 words and more. Our results validate again that query terms usually consist of a few keywords and therefore the correct language identification is very challenging.
To look at the most frequent queries of a retrieval system is an indication of trends and content people expect to find in the digital library. The most frequent queries in the TEL 2009 log files are “toto”, followed by “mozart” and “napoleon” (see table 1). From the 10 most frequent queries, 7 contain named entities and expressed a search for a person. These queries already indicate the challenges in determining the language of the query as they cannot be assigned to one particular language. Apparently, queries are not very often repeated since the most frequent one appears only 400 times.
From the simple searches, we extracted randomly 510 queries. The aim was to gain information about the query language, topic and intent of the queries. Another goal was to investigate the distribution of proper names and a categorization of different query types regarding their content.
In line with our findings about the entire simple search corpus our sample corpus showed an average in query length of 2.43 terms per query. More than ¾ of the queries consist of one term (41.5%), two terms (25%) or three terms (14.5%). 7% of the queries are composed of 4 terms, 12% have 5 and more terms. Ten queries contain only numbers such as ISBN /ISSN or dates and four queries consist of less than 3 characters.
Through a manual conceptual analysis of the extracted query terms we categorized the queries according to their content. We focused on flagging those queries, which might be problematic in terms of language identification and query translation. This includes proper names, uniform book titles and other entities requiring special recognition when processed during a cross-lingual search session. We subsumed these categories under the definition of named entities. Three different query types were defined in the context of named entities: 2 The queries “toto”, “a” and “test” are queries used for testing by the TEL office. It proves how important a thorough understanding of the data shown in log files is. To interpret user behavior by log file data correctly, it is necessary to exclude misleading log entries such as test data or search engine crawlers. This means that only 38% of our sample corpus queries could be readily translated with a dictionary-based translation approach - if the query language could be determined.
Previous studies have dealt with search engine query classification according to their intent [10], search goals [11] or topics [12].
A log file analysis of English Altavista queries showed that 20% of the queries are navigational, 48% are informational and the rest (30%) are transactional queries (excluding all sexual oriented queries) [10]. Rose and Levinson [11] created a hierarchy of search goals where the first level resembled Broder's taxonomy changing the transactional query to a search goal for resources. They found a greater proportion (around 61%) of informational queries and a smaller of navigational ones (around 15%).
Other studies focused on an automatic query classification [13]. The shortness of queries poses great challenges for automated query classification [14]. For our sample corpus we identified the following 6 named entity categories and two for non-named entities terms (table 3).
As shown in table 4, besides the topical searches such as “qualificações salário” users are mainly searching for persons: “dante”, followed by geo related topics, mainly countries or towns: “japan”, and titles: “social support and health status: a literature review (1997)”. Queries that can be assigned to more than one class are often a combination of author (person) and work: “all the russias by e. c. phillips” and counted for multiple categories (see table 4).
Table 5 shows the languages in which queries were expressed more than 10 times. It is striking that most of the queries are ambiguous terms where it was not possible to identify the language. This was mainly the case for named entities such as persons or geographic terms. In different languages they have normally no spelling variants e.g. “paris”. Several queries are not named entities but still ambiguous across languages e.g. “administration” or “culture”. Besides the languages listed we found queries in 13 other languages. Additionally, 4 Latin terms appeared in the corpus which expressed a very specific information need, e.g. “neuroptera myrmeleontidae”. We compared our manual language identification with an automated process using the Google Translate language detector4. The manual analysis showed that 39% of all queries cannot be assigned to a special language, which complicates the automatic language detection. In contrast to our analysis, Google did not detect any ambiguous queries with respect to language. With the Google API, more than 60% of the queries were detected to be English while our manual analysis identified 31% as English terms. The significant difference can be explained with a probable English language detection of the ambiguous queries because of an English language bias in the training or general Web data that Google uses.
It is a well known fact that the language identification of search engine queries is challenging but very important especially for multilingual information access. The correct language detection is necessary for further processing of the query such as stemming, spell checking, disambiguation or translation and the decision in which language the result list should be presented.
Web search queries are normally very short. Due to the large number of named entity queries – especially in the cultural domain - the automatic language detection has to deal with ambiguous terms or even terms that are not easily assigned to a certain language such as “Franz Kafka”. In our sample corpus 61.96% of the queries contain named entities and 54.70% of the queries consist only of named entities. As table 6 shows, from the 279 named entity queries we determined 167 as ambiguous. These are terms that occur in many languages such as: “Paris” or “Madonna”. Of course there are also named entities that can be assigned to the different languages such as: “Eiffelturm” (German), “Tour Eiffel” (France).
It is also shown, that queries which contain a named entity and another word are less ambiguous than those that only contain a named entity.
The named entity recognition is a very important aspect concerning the identification of a query language. For example, the query term “barber” can either refer to the English word for “hair dresser” or to the composer “Samual Barber”. In this case the correct detection of language alone does not ensure the identification of the user information need or intent.
For search engines, there are also cases where correct language detection does not necessarily imply that the user wants to see the results in the same language. For example, although the identification of the language for the query ”candida stellata” is Latin, a user entering this query from Germany, would most probably want to see German web pages, rather than web pages in Latin.
Table 6 shows the ratio of ambiguous terms in the sets of queries containing named entities and not containing named entities. This shows that many queries where the language cannot be clearly determined are expressing a search for a named entity. The 23 ambiguous terms which were not categorized as named entities are numbers and terms existing in several languages such as “culture” or “administration” or characters such as “a”. It is also worth to look at the ambiguity of different categories as shown in table 7. In the person category, the proportion of queries where the language cannot be identified is much higher than for geographic entities or titles of work. This is mainly due to the fact that names of persons do not change across languages, but it is fundamental that a CLIR system recognizes these entities. Standardized name authority files such as PND (Personennamendatei) or ULAN (Union List of Artist Names) are essential to fulfill this task.
As demonstrated before, language identification is very hard to implement correctly in an automated manner.
It is therefore reasonable to incorporate other aspects in the language detection that could hint at the language the user is searching in. Especially the correlation between the query language, the corresponding IP address and the interface language is of interest. Of course, the IP address might not be reliable in every case. The same user may use several IP addresses or several users can share one IP address. Furthermore, it is possible to hide the true location by using proxies. We are also aware of the fact that users rarely switch the interface language and that many of them work with the default English interface.
Interface language as a signal to detect the query language was also analyzed during
the last LogCLEF track by Oakes & Xu [
Since we also flagged terms, which could not be assigned to a particular language a slightly different dataset resulted. Looking at the number of queries which were entered under the same interface language as the query language the proportion of the total number of queries in these languages is very small. This is probably due to the fact that not many users switch their interface language.
For the shortened IP addresses, which were given in the log files, the respective country was identified. To make a statement about the relationship regarding country of origin and query language we determined the official language in these countries. The rows of table 10 show the languages spoken in the countries where queries originated from. This signal seems to be less strong than looking at the interface language. Looking for example at the 50 queries originating from German speaking countries5 like Germany or Austria, only 7 were German whereas the other languages of these queries were English (17), Spanish (3), Polish (1), other languages (2) and 20 queries, which could not be assigned to a single language. 5 Included countries are: Germany, Austria and Liechtenstein. We excluded countries with several spoken languages such as Switzerland and Belgium. The correct query language identification is decisive for language-dependent retrieval, the disambiguation and translation of query terms. It is also used in many retrieval systems to determine the language of the results presented. Our analysis shows that search query language identification and named entity recognition need to come together especially within a cultural heritage context. Many queries are expressing a search for proper names of persons, geographic entities or titles of work. Most of these queries cannot be assigned to a certain language. We also showed that signals commonly assumed to give indications about a user„s preferred language are not as strong as expected.
The retrieval system, however, should be able to identify named entities and language preferences and be able to present the users results in a language they can understand and enable them to judge the relevance of the documents. More research is therefore needed not only on language detection, a problem that might not be solved entirely – but also on named entities and their presentation in the search process. 12.Jansen, B. J., Spink, A., Bateman, J., Saracevic, T.: Real Life Information Retrieval: A Study of User Queries on the Web. SIGIR Forum: A Publication of the Special Interest Group on Information Retrieval, 32, 5-18 (1998) 13.Baeza-Yates, R. A., Calderón-Benavides,L., González-Caro, C.N.: The Intention Behind Web Queries. In: Lecture Notes in Computer Science. Vol. 4209, String Processing and Information Retrieval. 13th International Conference, SPIRE 2006, Glasgow, UK, October 11 - 13, 2006, 98‐109 (2006) 14.Kang, I., Kim, G.: Query Type Classification for Web Document Retrieval. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '03. ACM, New York, NY, 64-71 (2003)