This is the first time HIT2Lab has participated in CLEF. The Adhoc TEL task used a collection of electronic card catalog records from British Library, it is different from all the text retrieval tasks we've ever involved. In our runs, we used language modeling approach for retrieval model and incorporated pseudo-feedback, which have been proved effective in previous experiments. For bilingual tasks, we used Google translation service for query translation. We also adopted methods that have showed good effects in TEL 2009 task, including a stopword list provided by UniNE and partial fields indexing.
TEL@CLEF 2009 involves the British Library collection, which is consisted of electronic card catalog
records[
It is convenient to treat the whole text in a document as unstructured text, however, some have noticed that
removal of several useless fields can help to improve the retrieval performance[
We also used porter stemmer to normalize the terms. To eliminate the influence of very high frequent words, we used the stopword list released by UniNE last year.
Language modeling approach has been used widely in information retrieval researches. KL-divergence is a
language model for information retrieval [
x q(x)
Given the two models generated from document and query, we can model the risk of returning a document d
as relevant to a query q by KL-divergence between their respective language models[
P(t | Mq ) t P(t | Md ) where d is a document, q is a query. Md and Md are the language models for documents and queries language model respectively. P(t|Mq) and P(t|Md) are the probabilities that term t appears in Mq and Md.
R(d;q) = KL(Md || Mq ) =
P(t | Mq )log
Pseudo feedback is a technique to improve information retrieval performance, it has proven to be an effective
strategy for improving retrieval accuracy in all retrieval models. The method is to do normal retrieval to find an
initial set of most relevant documents, then assume that the top k ranked documents are relevant, and finally to do
relevance feedback as before under this assumption. The feedback model we used in this year’s TEL task is as
follows[
P(c | R)
P(c,q1...qk )
P(q1...qk ) where qi is an arbitrary query term, and c could be any possible representation concept. P(c, q1...qk) is calculated as follows:
P(c,q1...qk )=
P(c | D)P(q1...qk | D)P(D)
D
Cross-language information retrieval is a subfield of information retrieval dealing with retrieving
information written in a language different from the language of the user's query. For example, a user may pose
their query in Chinese but retrieve relevant documents written in English. One commonly used method is to
translate the query into document language, called query translation. Online translation service based query
translation has been proved to be an effective resource for cross-lingual information retrieval [
Experimental results showed that most queries have been translated well. For example, the title of the first query is translated as “Arctic animals”, which is the same as the English version; and its description is translated as “Find the Arctic fauna species and of related documents.” while the original one is “Find documents about arctic fauna species”. However there were still incorrectly translated topics. the title and description of topic 41 are “Sailing for Beginners” and “Find publications suitable for beginners or non-experts that provide information on any kind of sailing or boating”, and after it is translated into Chinese manually and translated back into English by Google translation, the contents of the two fields became “Portal navigation” and “Suitable for beginners or find information about non-professionals of any type of sailing or rowing motion information publications”. Portal navigation and sailing for beginners are definitely two different concepts, and so it caused the average precision of this topic to be 0%.
We submitted both monolingual and bilingual runs. Since we are familiar with English collection retrieval tasks, all our runs used only the English version dataset which is called TEL corpus from Britain Library as mentioned above. 50 topics are released for monolingual retrieval subtask. Since we are the only one who asked for Chinese version topics in bilingual retrieval subtask, we were responsible for translating the topics into Chinese. An undergraduate student who did not take part in the task took the charge of this job, and we required an associate professor from Harbin Institute of Technology to proof the translated topics.
For monolingual runs, we experimented with KL-divergence method and pseudo feedback. We compared partial fields indexing with full indexing, the effectiveness of stopword list provided by UniNE was also tested. For bilingual runs, we experimented with query translation based on Google translation. We also tried a corpus based method which takes a Chinese-English parallel corpus for converting Chinese topics to English version, however it did not show good performance on TEL@CLEF 2008 data, and we analyzed that the bad performance was due to the limit of the corpus we used. That is, the parallel corpus we used was too small and couldn’t reveal the true performance of the method, so we did not use the method in this year’s task.
Monolingual results without pseudo are listed below. All the experiments took KL-divergence as retrieval model. Run tag including “u” means running with UniNE stopword list, “p” means partial indexing, “t” means title field of the topic is used, “d” means description field is used, and “td” means title and description field are used.
Method kl-t kl-d kl-td kl-u-t kl-u-td kl-p-t kl-p-td kl-u-p-t kl-u-p-td
From the results we see that title combining with description can improve retrieval performance, especially after applying UniNE stopword list which effectively cut down meaningless terms in topics. Partial indexing also gets positive effect, and so suggests a potential research direction for the task.
Bilingual results are as follows.
Method kl-u-p-t kl-u-p-td
The retrieved documents set we submitted used pseudo feedback. The top 10 documents returned for each topic were considered to be relevant. We submitted 4 runs for monolingual subtask and 3 runs for bilingual subtask, the evaluation results of these runs are listed below. The meanings of “u”, “p”, “t”, “d”, “td” are as mentioned above, and “ft” means how many terms were contained in the expanded query generated from pseudo relevant documents.
Method kl-u-p-td kl-u-p-td-ft20 kl-u-p-td-ft40 kl-u-p-t-ft40 kl-u-p-td-ft20 kl-u-p-td-ft40 kl-u-p-t-ft40 monolingual monolingual monolingual monolingual bilingual bilingual bilingual
While classic language model coupled with pseudo feedback is suitable for the TEL task, particular features of the TEL corpus can also benefit to the retrieval performance. Different fields in TEL collection have different affect, and removal of some fields that have no relation with the document content can help to improve retrieval performance. Combining title and description field in a topic also works, especially after applying a stopword list.
Query translation based on online translation service works well for cross-lingual information retrieval. Online translation service is able to translate the most queries correctly, although there are a small portion of the queries be translated as absolutely different meaning from their real meaning.
There are still many issues to consider, including document fields weighting, the collection’s multilinguality and the document contents’ insufficiency. We will explore such issues in future work.