We present the participation of the University of Ottawa in the Cross-Language Spoken Document Retrieval task at CLEF 2005. In order to translate the queries, we combined the results of several online Machine Translation tools. For the Information Retrieval component we used the SMART system, with several weighting schemes for indexing the documents and the queries. One scheme in particular lead to better results than other combinations. We present the results of the submitted runs and of many unofficial runs. We compare the effect of several translations from each language. We present results on phonetic transcripts of the collection and queries and on the combination of text and phonetic transcripts. We also include the results when the manual summaries and keywords are indexed.
This paper presents the first participation of the University of Ottawa group in CLEF, the Cross-Language Spoken Retrieval (CL-SR) track. We briefly describe the task. Then, we present our system, followed by results for the submitted runs and for many unofficial runs. We experiment with many possible weighting schemes for indexing the documents and the queries. We compare the effect of several translations of the queries and of combining the translations. We look at using phonetic transcriptions of the queries and documents instead of the original ASR-produced text, and at combining the phonetic transcripts with the text. At the end we present the best results when all available information in the collection is used.
The CLEF-2005 CL-SR test collection includes 8104 segments, 75 topics (queries), and 12359
Relevance Judgments to facilitate information retrieval experiments. Segments are the unit of retrieval in the
CLEF CL-SR evaluation. Interviews with survivors of the Holocaust were manually segmented to form topically
coherent segments by subject matter experts at the Survivors of the Shoah Visual History Foundation. See
As a baseline, indexing the ASRTEXT2004A field using Okapi BM 25 term weights and searching with Title queries yielded an uninterpolated mean average precision of 0.0551 when run at the University of Maryland with the freely available PSE vector space search engine. This was evaluated on the 38 training topics, not on the 25 test topics for which we report results in this paper.
The topics were created in English from actual user requests and then translated into Czech, German, French, and Spanish by native speakers. An example topic in English is the following: <top> <num>1159 <title>Child survivors in Sweden <desc>Describe survival mechanisms of children born in 1930-1933 who spend the war in concentration camps or in hiding and who presently live in Sweden. <narr>The relevant material should describe the circumstances and inner resources of the surviving children. The relevant material also describes how the wartime experience affected their post-war adult life. </top>
The Spanish, German, and Czech topics provided by the CLEF organizers contained translations of all the fields (title, description, and narrative). For French the narrative field was not translated, due to lack of time. The French topic equivalent to the above English example is the following: <top> <num>1159 <title>Les enfants survivants en Suède <desc>Descriptions des mécanismes de survie des enfants nés entre 1930 et 1933 qui ont passé la guerre en camps de concentration ou cachés et qui vivent actuellement en Suède. </top> 2.
The University of Ottawa Cross-Language IR system was built with off-the-shelf components. For translating the queries from French, Spanish, and German into English, several free online machine translation tools were used. Their output was merged in order to allow for variety in lexical choices. All the translations of a title made the title of the translated query; the same was done for the fields description and narrative. For the retrieval part, the SMART IR system (Buckley et al., 2000) was tested with many different weighting schemes for indexing the collection and the queries. The weighting schemes are combinations of term frequency, collection frequency, and length normalization components. One scheme in particular was used in the submissions because it proved to have much better performance than other combinations. For weighting document terms we used term frequency normalized by the maximum value and probabilistic collection frequency weighting with cosine normalization. For queries we used non-normalized term frequency and inverse document frequency weighting. For all languages involved in the task, this combination worked very well when all the fields of the query were used (title, description, and narrative); it worked well with title plus description, and not as well with title only. 3.
For translating the topics into English we use several online MT tools. The idea behind using multiple translations is that they might provide more variety of words and phrases, therefore improving the retrieval performance. The seven online MT systems that we used for translating from Spanish, French, and German were: 1. 2. 3. 4. 5. 6. 7. For the Czech language topics we were able to find only one online MT system: http://intertran.tranexp.com/Translate/result.shtml
We combined their outputs by simply concatenating all the translations. All seven translations of a title made the title of the translated query; the same was done for the description and narrative fields. An example of combined output, for the French example used above, is: <top> <num> 1159 <title> surviving children in Sweden surviving children in Sweden The children survivors in Sweden surviving children in Sweden surviving children in Sweden The surviving children in Sweden surviving children in Sweden <desc> Descriptions of the mechanisms of survival of the children born between 1930 and 1933 who passed the war in concentration camps or hidden and who currently live in Sweden. Descriptions of the mechanisms of survival of the children born between 1930 and 1933 who passed the war in concentration camps or hidden and who currently live in Sweden.
Descriptions of the survival mechanisms of the born children between 1930 and 1933 that passed the war in co ncentration camps or hidden and that live currently in Sweden.
Descriptions of the mechanisms of survival of the children born between 1930 and 1933 who passed the war in concentration camps or hidden and who currently live in Sweden.
Descriptions of the mechanisms of survival of the children born between 1930 and 1933 who passed the war in concentration camps or hidden and who currently live in Sweden.
Descriptions of the mechanisms of survival of the children been born between 1930 and 1933 which crossed war in concentration camps or hidden and that live in Sweden nowadays.
Descriptions of the mechanisms of survival of the children born between 1930 and 1933 who passed the war in concentration camps or hidden and who currently live in Sweden. <narr> </top>
We used the SMART Information Retrieval (IR) system, originally developed at Cornell University in the 1960s.
SMART is based on the vector space model of information retrieval
The newest version of SMART (version 11) offers many state-of-the-art options for weighting the terms
in the vectors. Each term-weighting scheme is described as a combination of term frequency, collection
frequency, and length normalization components
tf max_ tf augmented normalized (a): new _ tf = 0.5 + 0.5 * ( tf max_ tf
) where max_ tf is the largest tf value in the vector. log (l): new _ tf = ln(tf ) + 1.0 square (s): new _ tf = tf 2 • Merging of collection frequency component Let num_docs, coll_freq_of_term, and coll _freq denote the number of documents in the collection, the number of documents in which term t occurs, and the total number of occurrences of the term t in the collection, respectively; then new_wt is defined as follows: none (n): new _ wt = new _ tf inverse document frequency weight (t): new _ wt = new _ tf * log( probabilistic (p): new _ wt = new _ tf * log( squared (s): new _ wt = new _ tf * (log( coll _ freq num _ docs coll _ freq _ of _ term )) 2 ) • Merging of vector normalization Let m denote the number of entries in the vector, then norm_wt is defined as follows: coll _ freq _ of _ term num _ docs _ coll _ freq num _ docs ) none (n): norm _ wt = new _ wt sum (s): norm _ wt =
tf ∑ new _ wt
m cosine (c): norm _ wt =
tf ∑ new _ wt 2
m
In this paper we employ the notation used in SMART to describe the combined schemes: xxx / xxx. The first three characters refer to the weighting scheme used to index the document collection and the last three characters refer to the weighting scheme used to index the query fields. For example, lpc/atc means that lpc was used for documents and atc for queries. lpc would apply log term frequency weighting (l) and probabilistic collection frequency weighting (p) with cosine normalization to the document collection (n). atc would apply augmented normalized term frequency (a) , inverse document frequency weight (t) with cosine normalization (c).
Table 2 presents results for various weighting schemes document/topics. There are 3600 possible combinations of weighting schemes: 60 schemes (5 x 4 x 3) for documents and 60 for queries. We tried 240 combinations and we present in the table the results for 15 combinations (the ones, plus some other ones to show the diversity of the results). mpc/ntn is still the best, but there are a few other weighting schemes that achieve similar performance. Some of the weighting schemes perform best when indexing all the fields of the queries (TDN), some on TD, and some on title only (T). npn/ntn was best for TD and lsn/ntn and lsn/atn are best for T.
In all the presented experiments we use stemming when indexing the collection and translated topics (except Section 5.3). Pseudo-relevance feedback was enables in the SMART system. We don’t present the results here, but when we tried using an English lemmatizer (to produce base forms of inflected words) instead of a stemmer, the results were slightly worse for all settings; when using no-stemming during indexing the performance was much worse.
Weighting scheme
TDN map
TD map bpref bpref
T map bpref
In Table 4 we present results for an experiment where the text of the collection and the queries were transcribed into phonetic form and split into n-grams (groups of n sounds, n = 4 in our case) that we used for indexing (without stemming). The phonetic n-grams were produced by University of Waterloo’s group. See their CLEF 2005 paper for more details.
The phonetic form might help compensate for the speech recognition errors made when the collection was produced. When the fields TD were indexed, the results are interesting. The map scores are higher than the previous results on the text form of the documents and queries (up to 28% for the translations from French), while the bpref scores are lower. When combining phonetic and text forms (by simply indexing both phonetic ngrams and text), the result are only slightly improved.
We obtained the best retrieval results among the seven teams that participated in this track. We believe that the improved performance is due to the choice of the weighting scheme used for indexing the document and query terms. Table 2 shows that performance varies a lot with the weighting scheme; it can be much lower for the some of the classic indexing schemes.
The idea of using multiple translations proves to be good. More variety in the translations would be beneficial. The online MT systems that we used are rule-based system. Adding translations by statistical MT tools might help, since they produce radically different translations.
On the manual data, the best map score we obtained is 46%, for English topics. On automatic data the best result is 21% map score. This difference shows that the poor quality of the ASR transcripts severely hurts the performance of IR systems on this collection. In future work we plan to investigate methods of removing or correcting some of the speech recognition errors in the ASR transcripts using the method of Inkpen and Désilets (2005).