We applied a PRF (Pseudo-Relevance Feedback) system, for both the monolingual task and the German(->English task. We focused on the effects of extracting a comparable corpus from the given newspaper data; our corpus doubled the average precision when used together with the provided parallel corpus. The PRF performance was lower for the queries with few relevant documents. We also examined the effects of the PRF first-step retrieval in the source language half of the parallel corpus vs. the entire document collection .
Times/German newspapers mismatch to be more pronounced. The results are even more encouraging when this mismatch is taken into account.
The algorithm for generating the comparable corpus 1) 2) 3)
Divide the German and English newspaper articles into overlapping N days long windows.
Initialize a dictionary (similarity thesaurus) D While the results are improving, a. For each window, i. Break the articles into fixed-size (P) paragraphs ii. Do a word-by-word translation of the paragraphs, using dictionary D and fertility F1 iii. Use each paragraph in one language as a query, and retrieve the top matching paragraph
among the ones in the other language iv. Repeat, switching languages v. If two paragraphs retrieved each other with a score above a certain threshold S, consider
them “mates” and add them to the comparable corpus C b. Extract a dictionary D’ from C using CHI-SQUARED (see below) as a similarity measure between words c. D = D’ The CHI-SQUARED statistic is measure we found useful in several contexts, which captures the crosslingual lexical associations based on co-occurrence in the training corpus.
CHI-SQUARED (t,s) = N(AD-BC)2/[(A+C)(A+B)(D+B)(D+C)] where t = term in the target language s = term in the source language N = number of document pairs A = number of documents where t occurs in the target language document and s occurs in the corresponding source language document B = number of documents where t occurs in the target language document and s DOES NOT occur in the corresponding source language document C = number of documents where t DOES NOT occur in the target language document and s occurs in the corresponding source language document D = N-A-B-C 4. Experimental Setup and Results
We used the Porter stemmer and the SMART stopword list for the English collection. For the German
collection, we used Morphix [
The PRF parameters were tuned using the CLEF-200 data. Good empirical values were 15-25 for K (number of top documents considered relevant), 200-300 for E (number of query words after expansion), and ltc and ntc term weighting. The same values proved to be best for the CLEF-2001 queries, with the exception of the term weighting scheme (ntc performed significantly worse than ltc on the new queries).
We used one week windows overlapping by half when generating the comparable corpus, because some of the newspapers were published weekly and a more fine-grained distinction was not needed. The best results were 1 The fertility is the number of words used to translate one word in the other language. This is different for every language pair. obtained when the starting dictionary was not initialized (i.e. the first retrieval step was based on names and cognates). The resulting corpus had cca. 20000 paragraph pairs. A paragraph size of 250 bytes (plus the bytes necessary to keep the entire last word) worked best.
The quality of the comparable corpus was fairly low (most paragraphs were far from being translations of each other). This is understandable given that the only thing linking the German and English articles was the time period; the culture, continent and the newspapers’ political goals and interpretations were different.
4.1 Official runs and results
CMUmll15e200td CMUmll5e300td CMUmll15e300tdn CMUbnn25e2td15 CMUbll25e3tdn25 CMUbnn25e2tdn15 There were 6 runs: 3 monolingual (DE->DE) and 3 bilingual (DE->EN). The results varied widely, and the term weighting was critical for the bilingual runs.
The monolingual runs were obtained by using the German (source language) half of the parallel corpus for the first retrieval step, in order to be consistent with the bilingual runs and provide a collection-dependent upper bound for them. Subsequent experiments revealed a significant difference between this procedure and the one using the entire collection (see figure below).
i 0.25 o ics 0.2 r 0.15 e .P 0.1 vg 0.05 A 0 ,01k= 03e= ,k15= e300= ,k25= e100= ,k25= e200= ,k25= e300= ,52k= 05e= ,k30= e300= ,5k= 003e= ,tcn ,25k= 0e30= Another important factor that affected our performance was the number of relevant documents. When a query only has one or two relevant documents, the basic assumption of the PRF idea is violated. Specifically, PRF assumes the first K documents to be relevant, which is false for at least 75% of the CLEF-2001 queries (if K>=20), even with a perfect search engine. 1.2 Our method’s sensitivity to the low number of relevant documents is illustrated in the figure above, where PRF is compared to the best method for each query. The correlation between the fictional “best” method and the number of relevant documents was practically non-existent (-0.07), while CMU-PRF was comparatively more affected by the low number of relevant documents (the correlation is 0.26). We do not know how the individual runs from which the best result was selected were affected by the low number of relevant documents.
After the release of the relevance judgments, we conducted several experiments to examine the effect different parameters had on the bilingual PRF system. The most impressive was the gain obtained from using the comparable corpus in addition to the given parallel corpus (the performance was doubled). n0.25 iso 0.2 ic0.15 e rP 0.1 .g0.05 vA 0 CMU PRF Best
Given Parallel ntc, k=25, e=300 5 10 15 25 30
The number of documents used for query expansion did not significantly affect the average precision, although the effects on individual queries remain to be examined.
Unlike the CLEF-2000 queries, this year’s queries were sensitive to the term weighting scheme. The difference in performance between the two schemes shown above is significant.
The fixed paragraph size of the comparable corpus was another important factor. Paragraphs that were too short produced unreliable co-occurrence statistics.
The most important finding in our experiments was the effect of the comparable corpus had on the average performance of the PRF method. Since the quality and quantity of parallel or comparable text is crucial to this method, we plan to gather more such data from the web. Preliminary results showed improvements over the official and unofficial CLEF runs and will be discussed elsewhere.
We are also planning to expand our CLEF toolkit to other methods previously implemented at CMU, such as
GVSM, LSI, EBT and LLSF [