The Dublin City University participated in the CLEF 2007 CL-SR English task. For CLEF 2007 we concentrated primarily on the issues of topic translation, combining this with search ¯eld combination and pseudo relevance feedback methods used for our CLEF 2006 submissions. Topics were translated into English using the Yahoo! BabelFish free online translation service combined with domain-speci¯c translation lexicons gathered automatically from Wikipedia. We explored alternative translations methods with document retrieval based the combination of the multiple document ¯elds using the BM25F ¯eld combination model. Our results indicate that extending machine translation tools using automatically generated domain-speci¯c translation dictionaries can provide improved CLIR e®ectiveness.
The Dublin City University participation in the CLEF 2007 CL-SR task focussed on extending our
CLEF 2006 system to investigate combinations of general and domain-speci¯c topic translation
resources. Our 2006 participation in the CL-SR task concentrated on the combination of the
multiple ¯elds associated with the speech documents. Our study was based on using the document ¯eld
combination extended version of BM25 termed BM25F introduced in [
The remainder of this paper is structured as follows: Section 2 summarises the motivation and implementation of the BM25F retrieval model, Section 3 overviews our basic retrieval system and describes our sentence boundary creation technique, Section 4 describes our topic translation methods, Section 5 presents the results of our experimental investigations, and Section 6 concludes the paper with a discussion of our results. 2
The English collection comprises 8104 \documents" that are manually-determined topically-coherent
segments taken from 272 interviews with Holocaust survivors, witnesses and rescuers, totaling 589
hours of speech. The spoken documents are provided with a rich set of data ¯elds, full details of
these are given in [
Two standard methods of combining multiple document ¯elds in retrieval are: ² to simply merge all the ¯elds into a single document representation and apply standard single document ¯eld information retrieval methods, ² to index the ¯elds separately, perform individual retrieval runs for each ¯eld and then merge the resulting ranked lists by summing in a process of data fusion.
The topic of ¯eld combination for this type of task with ranked information retrieval schemes
is explored in [
The BM25F combination approach uses a simple weighted summation of the multiple ¯elds of the documents to form a single ¯eld for each document in the usual way. The importance of each document ¯eld for retrieval can be determined empirically in separate runs, the count of each term appearing in each ¯eld is multiplied by a scalar constant representing the importance of this ¯eld, and the components of all ¯elds are then summed to form the overall single ¯eld document representation for indexing. Once the ¯elds have been combined in a weighted sum, standard single ¯eld IR methods can be applied. 3 3.1
The basis of our experimental system is the City University research distribution version of the
Okapi system [
Document terms were weighted using the Okapi BM25 weighting scheme shown as follows, tf (i; j) £ (k1 + 1) cw(i; j) = cf w(i) £ k1 £ ((1 ¡ b) + (b £ ndl(j))) + tf (i; j) cf w(i) = log µ (rload + 0:5)(N ¡ n(i) ¡ bigrload + rload + 0:5) ¶ (n(i) ¡ rload + 0:5)(bigrload ¡ rload + 0:5 ; ndl(j) = dl(j) agvdl where cw(i; j) n(i) N tf (i; j) ndl(j) dl(j) avgdl k1 and b represents the weight of term i in document j; is the total number of documents containing term i; is the total number of documents in the collection; is the within document term frequency; is the normalized document length; is the length of j; is the average document length in the collection; are empirically selected tuning constants for a particular collection.
The matching score for each document is computed by summing the weights of terms appearing in the query and the document. The BM25 k1 and b values used for our submitted runs were tuned using the 63 CLEF 2007 CL-SR English training topics. rload and bigrload take the default parameters of 4 and 5 respectively. 3.2
Query expansion by pseudo relevance feedback (PRF) is a well-established procedure in both
monolingual and cross-lingual IR, potentially providing some improvement in retrieval
e®ectiveness. The method used here is based on our work originally described in [
In our modi¯ed version of RSV, the top t potential expansion terms are selected from the augmented summaries of the top d1 ranked documents, but ranked using statistics from a larger number d2 of assumed relevant ranked documents from the initial run.
The summary-based PRF method operates by selecting topical-related expansion terms from
document summaries. However, since the ASR transcriptions of the conversational speech
documents do not contain punctuation, we developed a method of selecting signi¯cant document
segments to identify documents \summaries". Our approach is derived from Luhn's word cluster
hypothesis. Luhns hypothesis states that signi¯cant words separated by up to ¯ve non-signi¯cant
words maximum are likely to be strongly related. Clusters of these strongly related word were
identi¯ed in the running document transcription by searching for word groups separated by not
more than ¯ve insigni¯cant words. Words appearing between clusters are not included in clusters,
and thus can be ignored for the purposes of query expansion since they are by de¯nition stop
words. The clusters were then awarded a signi¯cance score based on the following two measures:
Luhn's Keyword cluster method Luhns method assigns a sentence score LS for the highest
scoring cluster within a sentence [
SW 2 T W where SW is the number of bracketed signi¯cant words, and T W is the total number of bracketed words.
Query-biasd method This method assigns a score QS to each sentence based on the number of query terms in the sentence as follows:
QS =
T Q2 N Q where T Q is the number of query terms occurring in the sentence, and N Q is the total number of terms in a query.
For each sentence (cluster), the overall sentence score SS is calculated using SS = LS + QS. The top s sentences (clusters) with the highest SS are then selected as the document summary. 4
Machine Translation (MT) based query translation uses an existing MT system to provide
automatic translation. This approach has been widely used in cross-language information retrieval
with good average performance when such an MT system is available for the language pair of the
topic and document. In our experiments, topics were translated into English using the Yahoo!
BabelFish powered by SYSTRAN [
As a multilingual hypertext medium, Wikipedia1 has been proved to be a valuable new source
of translation information [
Additionally, we observed that multiple English wikipedia URLs en.wikipedia.org/wiki/World_ War_II, en.wikipedia.org/wiki/World_War_2, en.wikipedia.org/wiki/WW2, and en.wikipedia.org/ wiki/Second_world_war are redirected to the same wikipedia page and the URL basenames \World War II", \World War 2", \WW2", and \Second world war" are synonyms. Using all these English terms during query translation is a straightforward approach to the automatic post-translation query expansion.
To utilize the multilingual linkage and the link redirection features, we implement a threestage automatic process to extract German, French, and Spanish to English translations from Wikipedia: 1. An English vocabulary for the domain of the test collection was constructed by performing a limited crawl of the English wikipedia2, Category:World War II. This category is more likely to contain links to pages and subcategories concerning events, persons, places, and organizations pertaining to war crimes or crimes against humanity especially during the second world war. In total, we collected 7431 English web pages. 2. For each English page obtained, we extracted the hyperlinks to each of the query languages.
This provided a total of 4446, 3338, and 4062 hyperlinks to German, Spanish, and French, respectively. 1http://www.wikipedia.org/ 2http://en.wikipedia.org
French Query: Les marches de la mort
| Les | marches de la mort |
Domain-specific lexicon marches de la mort → Death marches marches de la mort → Death marches Holocaust
| Death marches | Death marches Holocaust |
English Translation: Steps of death, Death marches, Death marches Holocaust 3. We then selected the basenames of each pair of hyperlinks (German{English, French{English, and Spanish{English) as translations and added into our domain-speci¯c lexicons. The nonEnglish multi-word terms were added into the phrase dictionary for each query language.
These phrase dictionaries are later used for phrase identi¯cation during query pre-processing. 4.2
As shown in Figure 1, our query translation process is performed in the following manner: 1. Query pre-processing: We used the phrase dictionary with the maximum forward matching algorithm to segment each query Q into a list of terms fq1; q2; q3; :::; qng. 2. Domain-speci¯c lexicon lookup: For each query term qi (where i 2 (1; n)), we obtained all its English translations fei1; ei2; ei3; :::; eimg via a domain-speci¯c lexicon look-up. 3. BabelFish translation: we then translated the original query Q into the English query E using the Yahoo! BabelFish. 4. Translation results merging: For each English term eij (where i 2 (1; n) and j 2 (1; m)) obtained in Step 2, we appended it to the end of the translated English query E. 5
In this section we report results for our experimental runs for the CLEF 2007 English CL-SR task. Results are shown for combinations of manual only ¯elds, automatic only ¯elds and combining both manual and automatic ¯elds. For monolingual retrieval results show precision at cuto® ranks of 5, 10 and 30, standard TREC mean average precision (MAP) and recall in terms of the total number of relevant documents retrieved for the test topic set. For CLIR results compare alternative topic translations resources showing MAP and precision at rank 10. Our submitted runs for the CLEF 2007 are indicated by a ¤ in the tables. 5.1
Our retrieval system requires a number of parameters to be set for the term weighting, ¯eld combination, and PRF components. All parameter values were set empirically using the 63 CLEF 2007 training topics.
Baseline ¤PRF Baseline ¤PRF ¤PRF Manual ¯eld combination (MK£1+SUM£1, k1 = 1:0, b = 0:5)
TDN 1850 0:2773 TDN 1903 ² For the manual data ¯eld combination, Okapi parameters k1 = 1:0 and b = 0:5 give the best results when the document ¯elds are weighted as MK£1, and SUM£1; ² For the automatic data ¯eld combination, k1 = 8:0 and b = 0:5 perform the best when the document ¯elds are weighted as A1K£1, AK2£1, and ASR06B£2; and ² For the manual and automatic data ¯eld combination, k1 = 3:0 and b = 0:6 produce the best results when the document ¯elds are weighted as MK£4, SUM£4, and ASR06B£1. PRF For all our PRF runs, the top d1 ranked documents were assumed relevant for term selection and document summaries comprised the best scoring s clusters. The RSV values to rank the potential expansion terms were estimated based on the top d2 ranked assumed relevant documents. The top t ranked expansion terms taken from the clusters were added to the original query in each case. The original topic terms are up-weighted by a factor ® relative to the expansion terms. Our PRF query expansion thus involves ¯ve parameters as follows: t s d1 d2 ® is the number of the expansion terms selected from the summary; is the number of sentences (clusters) selected as the document summary; is the number of documents used for sentence (cluster) selection; is the number of documents used for expansion terms ranking; is the up-weighting factor.
This set of parameters were again tuned using the CLEF 2007 CL-SR English training data. We note that PRF involves selection of parameter values that are not necessarily consistent from one collection (indexed using di®erent ¯eld combination methods) to another.
Our experiments showed that t = 60, s = 6, d1 = 3, d2 = 20, and ® = 3:0 give the best results for the manual data ¯eld combination and manual and automatic data ¯eld combination; t = 40, s = 6, d1 = 3, d2 = 20, and ® = 3:0 produce the best results for the automatic data ¯eld combination. 5.2
This section presents results for our ¯eld combination experiments for monolingual English retrieval. Table 1 shows results for both the baseline condition without application of PRF and with our summary-based PRF.
For the combination of the MK and SUM ¯elds we can ¯eld than application of PRF generally produces a small improvement in performance. Note that the topics here use all three topics ¯elds Title, Description and Narrative (TDN), and thus these results cannot be compared directly BabelFish baseline BabelFish+PRF BabelFish+LEX ¤BabelFish+LEX+PRF 0:0476 0:0501 to any other results shown here which use only Title and Description ¯elds (TD). Similarly for both the automatic only ¯elds runs combining AK1, AK2 and ASR2006B, and the combination of manual and automatic ¯elds using MK, SUM and ASR2006B, application of PRF produces a small improvement in average and high rank precision, although there appear to be some problems at lower ranks which we intend to investigate.
Yahoo! BabelFish combined with domain-speci¯c lexicons We then explore the combinations of the query translation and post-translation query expansion, and investigate the improvement contributed by each component in German, French, and Spanish to English CL-SR. The results of these experiments are shown in Table 2.
As shown in Table 2, in comparison to the standard BabelFish translation (BabelFish baseline), augmented translations from the domain-speci¯c lexicons (BabelFish+LEX) led to a signi¯cant improvement (27%) in French{English retrieval task, but only 3% and 4% in Spanish{English and German{English, respectively. This can be explained by the fact that the MAP values for the baseline runs of German and Spanish are much higher than the MAP for the French baseline. We noticed that the description ¯eld of German topics sometimes contains additional explanation enclosed by square brackets. The e®ect of this was often that more correct documents should be retrieved in the German{English task. We therefore believe that the BabelFish system gives a better translation from Spanish, rather French and German, to English.
At the individual query level (shown in Table 3), we observed that retrieval e®ectiveness sometimes slightly degraded when the query was augmented to contain translations from our domain-speci¯c lexicons, despite the fact that they are correct translations of the original query terms. This occurred mainly due to the fact that additional terms result in a decrease of relevant documents at ranks, because they are too general in the collection. For example, \war", \Europe", \Poland", \holocaust", \country", \Jewish", \people", \history", \concentration camp", etc. This problem may be solved if we down-weight the general-term translations during the retrieval process, so that when term frequency is used in calculating similarity, documents with many general terms will not be over-emphasized. We intend to explore this issue in further experiments.
We used the summary-based PRF to provide post-translation query expansion in all crosslingual retrieval runs (see BabelFish+PRF and BabelFish+LEX+PRF shown in Table 2). It gave improvements of 7% for the mono-lingual run, but only provided improvements of 5%, 1%, and 5% in French{English, Spanish{English, and German{English CL-SR e®ectiveness. 6
This paper has described results for our participation in the CLEF 2007 CL-SR track. In 2007 our experiments focussed on the combination of standard machine translation with domain-speci¯c translation resources. Our results indicate that combining domain-speci¯c translation derived from Wikipedia with the output of standard machine translation can produce substantial improvements in MAP. Further improvements can also be observed when combined with PRF. How4 2 t s u g u , d l a w n e h c u B , p
S S l e ® a t s z t u h c S , t s u a c o l o H s e am d h c n c a r
a on lo i
P m t ra ,t th t n a n e e e c m D con eov ,se
m ch ra d la ec ra W w n n a m ,e
C e r u t l u , y r t n u o n o i g e r e n i t s e l e a p P ro ,
e u n E it , s h s i l g n
M s e l l i e s r a , e l l i e s r a , y r o t s i H , y r F M t n e m n r e te p n o I r , u s E p , s am w c e J n
o , i e t
a r u o ab s l
w ed eJ c , r e o l F p o e r u t lu yn
C a ,r e s o P w ,t b h J la is d w le e su rem ts l , a
p r G au eo tn c
P ec c , o ts lo e e p l c J o o ro , e F sh P , i y t ew is h h l
o T
h n u o is o H a
H w C c e o e e , h J p T ,sh ca
m P J s , , y y n n e a a J r e J p , o r ew ,J s w , H , e tu eh tsu ie
T a h ca , c o , i l w o o ts l
u o e H a
H J en w n
o J it
a w tr A
A t n e m n r e t n I , s p m a c n o i t a r t n e c n o C , p m a c n o i t a r t n e c n o , S S l e ® a t
C s n o c i x e L m o r f s n o i t a l s n a r l a n o i t i d d T
F l e P b
a A B M x e L + h s i h s i F l e b a
B D h I s
i y r e m R ro S f A s + n 1 l s K n A a r + t 1
£ a £ tr K x 1 e A g , n n is i
o u ta f n o i
b m d ax le¯ E
c : it 3 a e m l o b t a u f o II s , I I , s e i l l B H ,
A p e ,
h S m T S a c ,
l t e n s o u a®
a t i t a c s o z tr l t n o u
H h
c e c , n
d S re co n ,
y u l
d o t a l r a e t i L e h w , n ia
um ,e a ls ra
u P o W n h p c c r u oo tso rou ra loo lo i ,t c l o l
s e i o ra tra e r u t t l s u u C ca , y lo tr o n H ou eh
C T t ,t , s s t u u su a a a c c lo lo co o o lo H H H e e , h h r T T a , , ts ts , u u ce a a n
W
W r a , s n a h 1 3 6 0 0 0 0 0 0 0 0 Q h 1 1 3 3 3 3 3 3 3 a 1 1 3 3 3 3 3 3 3 3 3 3 3 is 1 1 1 3 3 3 3 3 3 3 3 c m n n
r a e e rF 1 2 3 4 5 6 7 8 9 G 1 2 3 4 5 6 7 8 9 10 11 12 13 Sp 1 2 3 4 5 6 7 8 9 10 11 ever, these trends are not observed consistently in all cases, and further investigations will focus on understanding di®erences in behaviour more clearly and re¯ning our procedures for training domain-speci¯c translation resources.