CoLesIR at CLEF 2006: Rapid Prototyping of an N -gram-Based CLIR System

Abstract

In this our rst joint participation as the CoLesIR group, our team has participated in the Portuguese monolingual ad-hoc task and in all robust ad-hoc tasks |all monolingual tasks, the English-to-German bilingual task, and the multilingual task. We have developed an n-gram model inspired by the previous work of the Johns Hopkins University Applied Physics Lab. Our approach makes generalized use of freely available resources |such as the Europarl parallel corpus, the GIZA++ wordalignment toolkit, and the Terrier retrieval platform|, and employs a new n-gram direct translation technique. This new technique takes as input previously existing aligned word lists and obtains as output aligned n-gram lists. It can also handle word translation probabilities, as in the case of statistical word alignments. This new n-gram-based approach shares the main advantages of the original proposal. This solution avoids the need for word normalization during indexing or translation, and it can also deal with out-of-vocabulary words. Since it does not rely on language-speci c processing, it can be applied to very di erent languages, even when linguistic information and resources are scarce or unavailable. Our proposal adds to these characteristics a higher speed during the n-gram alignment process. Unfortunately, lack of time did not allow us to get our n-gram direct translation system ready on time. This way, we could submit only those initial results to be used as baselines in the future evaluation of our approach.

Keywords

• H
• 3
• 1 [Information Storage and Retrieval]
• Content Analysis and Indexing|Indexing methods
• Linguistic processing
• H
• 3
• 3 [Information Storage and Retrieval]
• Information Search and Retrieval|Query formulation
• J
• 5 [Arts and Humanities]
• Language translation

Introduction

CoLesIR group is an interuniversity research group created for joint participation in the CLEF competition. It is composed of members of the Compilers and Languages Research Group (CoLe)1 of the Universities of A Corun~a and Vigo (Spain), and members of the Information Retrieval Group2 of the University of Sunderland (United Kingdom). Although we have participated separately in CLEF [ 17, 16, 13, 12 ], this is our rst joint participation.

The Spanish CoLe group has been working for several years on the application of Natural Language Processing (NLP) techniques to Information Retrieval (IR) [ 18, 17, 16 ], and recently has entered into the eld of Machine Translation (MT) [ 3 ]. The possibility of applying its experience in these elds to the eld of Cross-Language Information Retrieval (CLIR), and the support given for this purpose by the Sunderland University IR Group, led to the birth of this joint group

So, this paper describes our rst research experiences in the eld of Cross-Language Information Retrieval. A CLIR system based in the employment of n-grams not only as indexing units, but also as translating units, is presented.

The article is outlined as follows. Section 2 presents previous work on the application of ngrams to CLIR systems. Next, section 3 describes our n-gram-based CLIR system. Section 4 shows the results obtained in our participation in both the Portuguese monolingual and robust tasks of the CLEF 2006 ad-hoc track. Unfortunately, lack of time did not allow us to get our ngram direct translation system ready on time. This way, we could only submit those initial results to be used as baselines for the future evaluation of our approach. Finally, Section 5 presents our conclusions and future work. 2

Previous Approaches on N -gram-Based Translation Our proposal has been inspired by the previous work of the Johns Hopkins University Applied Physics Lab (JHU/APL) about the employment of overlapping character n-grams for indexing documents [ 9, 8, 10, 11 ]. Their interest came from the possibilities that overlapping character n-grams may o er particularly in the case of non-English languages [ 8 ]: to provide a surrogate means to normalize word forms and to allow one to manage languages of very di erent natures without further processing, such as agglutinative languages as in the case of Turkish, or languages lacking word separator characters such as Japanese. Moreover, this knowledge-light approach does not rely on language-speci c processing, and it can be used even when linguistic information and resources are scarce or unavailable.

In the case of monolingual retrieval, the employment of n-grams is quite simple, since the documents to be indexed are just tokenized into overlapping n-grams instead of the usual words. This way, the word potato, for example, is split into its di erent overlapping compounding n-grams: -pot-, -ota-, -tat- and -ato-. These resulting n-grams are then indexed by the retrieval engine. The same tokenizing process will be made with queries, allowing matching between documents and queries.

In the case of translingual retrieval, the document indexing process remains the same, but two phases are now required during query processing: one for translation and another one for ngram splitting. In their initial cross-language experiments, JHU/APL rstly translated the source language query into the target language using Machine Translation (MT) techniques, parallel collections or bilingual dictionaries [ 9 ]. The resulting translated query was then split into n-grams, which were submitted to the retrieval engine.

Further experiments were made using a new n-gram-based translation approach. This so-called direct n-gram translation technique used n-grams instead of words as translation units. The objective pursued was to avoid some of the limitations of classical dictionary-based translation, such as the need for word normalization, the problems of translating multiple word expressions and the inability to handle out-of-vocabulary words [ 11 ]. This n-gram translation algorithm takes as input 1http://www.grupocole.org 2http://www.cet.sunderland.ac.uk/IR/ir.html a parallel corpus, aligned at the paragraph (or document) level and extracts candidate translations as follows [ 10 ]. Firstly, for each candidate n-gram term to be translated, paragraphs containing this term in the source language are identi ed. Next, their corresponding paragraphs in the target language are also identi ed and, using a statistical measure similar to mutual information, a translation score is calculated for each of the terms occurring in one of such target language texts. Finally, the target n-gram with the higher translation score is selected as the potential translation of the source n-gram. The whole process is quite slow: it is said that the process takes several days in the case of working with 5-grams, for example [ 10 ]. 3

Our approach

Taking as our model the system developed by JHU/APL, we have developed our own n-gram based retrieval system for testing our ideas. This system has been built using freely available resources when possible in order to make it more transparent and to minimize e ort.

This way, instead of the ad-hoc retrieval system employed by the original design [ 9 ], we have opted for using the open-source Terrier platform [ 1 ]. This decision was supported by the satisfactory results obtained with n-grams using other indexing engines [ 15 ].

The second point of di erence with respect to the original approach comes from the translation resources to be used. JHU/APL employed bilingual word-lists extracted from a huge parallel corpus of their own [ 10 ] created by mining the web of the O cial Journal of the European Union3. However, since our group has no access to such a large parallel corpus, we had to employ a smaller one, the well-known Europarl corpus [ 4 ]. This corpus was extracted from the proceedings of the European Parliament covering April 1996 to September 2003, containing up to 28 million words per language. It includes versions in 11 European languages: Romanic (French, Italian, Spanish, Portuguese), Germanic (English, Dutch, German, Danish, Swedish), Greek and Finnish.

Finally, with respect to the n-gram translation algorithm itself, since the alignment algorithm of the original approach was too slow for our purposes, we opted for a slightly di erent approach consisting of two phases. In the rst phase, the slowest one, word-level alignment of the text was made employing a statistical alignment model. For this purpose, the parallel corpus was processed through the well-known GIZA++ toolkit [ 14 ], obtaining the translation probabilities between the di erent source and target language words which have been aligned by the software tool. Next, prior to the second phase, several heuristics can be applied |if desired| for re ning or modifying such word-to-word translation scores. We can remove, for example, those candidate translations with a translation probability less than a previously established threshold, or we can combine the scores of bidirectional alignments [ 5 ] |source-target language and target-source language| instead of just the direct one |source-target language. Finally, in the second phase, n-gram translation scores are computed employing associative measures [ 7 ], taking as input the translation probabilities calculated by GIZA++.

At this point, and in order to illustrate accurately the process involved during this second phase, we will take as basis how associative measures are calculated and how they could be used for generating bilingual dictionaries automatically taking as input parallel collections aligned at paragraph level. In this illustrating context, given a word pair (wordu; wordv) |wordu standing for the source language word, and wordv for its candidate target language translation|, their cooccurrence frequency can be organized in a contingency table resulting from a cross-classi cation of their cooccurrences in the aligned corpus:

In this table, instances whose rst component belongs to type wordu |i.e., the number of aligned paragraphs where the source language paragraph contains wordu| are assigned to the rst row of the table, and tokens whose second component belongs to type wordv |i.e., the number of aligned paragraphs where the target language paragraph contains wordv| are assigned to the rst column. The cell counts of this contingency table are called the observed frequencies: O11: Number of aligned paragraphs where the source language paragraph contains wordu and the target language paragraph contains wordv.

O12: Number of aligned paragraphs where the source language paragraph contains wordu but the target language paragraph does not contain wordv.

O21: Number of aligned paragraphs where the source language paragraph does not contain wordu but the target language paragraph contains wordv.

O22: Number of aligned paragraphs where the source language paragraph does not contain wordu and the target language paragraph does not contain wordv either.

The sum of all these four observed frequencies |or sample size N | is equal to the total number of pairs of words considered. R1 and R2 are the row totals of the observed contingency table, while C1 and C2 are the corresponding column totals. Such row and column totals are also called marginal frequencies, and O11 is called the joint frequency. Equations for all association measures are given in terms of the observed frequencies, marginal frequencies, and the expected frequencies E11, ..., E22 (under the null hypothesis that wordu and wordv are statistically independent). The expected frequencies can easily be computed from the row and column totals:

Once the contingency table has been built, di erent association measures can be easily calculated for each pair of words. After this, the most promising pairs can be inserted into the automatically generated bilingual dictionary by selecting them from those with the highest association measures. In our case, two classical measures will be applied for this purpose: mutual information and Dice coe cient, de ned by equations 1 and 2, respectively:

At this point, we have described how to compute and employ association measures for the automatic generation of bilingual dictionaries from parallel corpora aligned at paragraph level. However, in our proposal, we do not have aligned paragraphs but aligned words |a source word and its candidate translation|, both composed by n-grams. Our rst idea could be just to adapt the contingency table to that context. Consequently, we can consider that we are now dealing with n-gram pairs (n-gramu; n-gramv) cooccurring at aligned words instead of word pairs (wordu; wordv) cooccurring at aligned paragraphs. So, contingency tables should be rede ned according to this new situation: O11, for example, should be re-formulated as the number of aligned words where the source language word contains n-gramu and the target language word contains n-gramv.

This rst solution seems logical, and it is intuitive and easy to understand. Nevertheless, we nd a problem. In the case of aligned paragraphs formed of words, we had real instances of word cooccurrences at the paragraphs aligned. However, in our proposal we do not have real instances of n-gram cooccurrences at aligned words |as it may be expected|, but just probable ones, since GIZA++ |the tool employed for the initial word-level alignment| is based on a statiscal alignment model which computes a translation probability for each cooccurring pair of words. So, the same word may appear as being aligned with several translation candidates, each one with its corresponding probability. For example, taking the English words milk and milky, and the Spanish words leche (milk), lechoso (milky) and tomate (tomato), a possible output alignment would be: source word candidate translation

probability milk milky milk leche lechoso tomate

This way, it may be considered that the source 4-gram -milk- does not really cooccur with the target 4-gram -lech-, since the alignment between its containing words milk and leche, and milky and lechoso is not certain. Nevertheless, it seems much more probable that the translation of the 4-gram -milk- was -lech- rather than -toma-, for example, since the probability of the alignment of their containing words |milk and tomato| is much smaller than that of the words containing -milk- and -lech- |the pairs milk and leche and milky and lechoso. Taking this idea as basis, our proposal consists of weighting the likelihood of a cooccurrence according to the probability of its corresponding alignment.

Taking again the previous milk-milky example, we can consider the overlapping 4-grams that compose each word. Thus, we would obtain an alignment like this: source word candidate translation

probability -milk- -lech- -eche-milk- -ilky -lech- -echo- -chos- -hoso-milk- -toma- -omat- -mate0.98 0.92 0.15 So, the contingency tables corresponding to the n-gram pairs (-milk-, -lech-) and (-milk-, -toma-) are as follows:

O11 = 0:98 + 0:92 =1.90

O12 = 0:98 + 3 0:92 + 3 0:15 =4.19

V 6= -lechO22 = 3 0:92 =2.76

C2 =6.95

R1 =6.09 R2 =3.68

N =9.77 O12 = 2 0:98 + 4 0:92 + 2 0:15 =5.94

V 6= -tomaO22 = 4 0:92 =3.68

C2 =9.62

R1 =6.09 R2 =3.68 N =9.77 U = -milkU 6= -milk

U = -milkU 6= -milk

V = -lechO21 =0.92

C1 =2.82 V = -tomaO11 =0.15

O21 =0

C1 =0.15

It can be seen in the example that the O11 observed frequency corresponding to the n-gram pair (-milk-, -lech-) is not 2 as it could be expected, but 1.90. This is because it appears in 2 alignments, milk with leche and milky with lechoso, but each cooccurrence in a alignment must also be weighted according to its translation probability like this: 0.98 (probability of the alignment of milk with leche) + 0.92 (probability of the alignment of milky with lechoso) = 1.90.

Once the contingency tables have been obtained, the Dice coe cients corresponding to each n-gram pair can be computed. As expected, the association measure of the pair (-milk-, -lech-) |the correct one| is much higher than that of the pair (-milk-, -toma-) |the wrong one: 2 1:90 Dice(-milk-, -lech-)= 6:09+2:82 = 0.43 2 0:15

Dice(-milk-, -toma-)= 6:09+0:15 = 0.05

If we consider that a real existing cooccurrence instance |such as those of the word-based algorithm used for illustrating| corresponds to a 100% probability, we can think about the original

Task

Language Size (in MB) # of docs.

Monolingual

PT word-based algorithm for building the contingency table and calculating word-level associative measures as a particular case of the generalized algorithm we have proposed.

This new approach we have proposed for n-gram direct translation increases the speed of the process, concentrating most of the complexity in the word-level alignment phase. This rst step acts as a initial lter, since only those n-gram pairs corresponding to aligned words will be considered, whereas in the original JHU/APL approach all n-gram pairs corresponding to aligned paragraphs were considered. On the other hand, since the n-gram alignment phase is much faster, di erent n-gram alignment techniques can be easily tested. Another advantage of this approach is that the n-gram alignment process can take as input previously existing lists of aligned words or even bilingual dictionaries, theoretically improving the results. 4

Experiments

In this our rst joint participation as CoLesIR group, we have taken part in two tasks of the ad-hoc track: the Portuguese monolingual task, and the robust task. Unfortunately, the lack of time did not allow us to tune accurately our retrieval system, either to complete our n-gram direct translation tool. So, we can show here only the results intended to be used as baselines for future tests. This way, no tuning has been made with respect to the possibility of removing high or lowfrequency n-grams, the employment of relevance feedback, or the use of pre or post-translation expansion techniques in the case of translingual runs [ 10 ].

So, documents were just split into n-grams and indexed, as were the queries. Before that, the text had been converted into lowercase and punctuation marks were removed [ 10 ]. Diacritics, however, have been kept in this rst rst set of experiments.

The open-source Terrier platform [ 1 ] has been employed as retrieval engine using a InL24 ranking model [ 2 ]. No stopword removal or query expansion have been applied at this point. The same running parameters have been used for all the experiments performed. With respect to the n-gram length, we decided to use 4-grams as a compromise size after studying the results previously obtained by the JHU/APL group [ 9, 8, 10, 11 ] using di erent n-gram lengths. 4.1

Portuguese Monolingual Task

The possibility of working with Portuguese caught our attention because of its proximity to Galician language. Our Spanish part, the CoLe group, has been working for many years on NLP and IR in Galician, a Romance language spoken in Galicia, in the North-West of Spain, where it is co-o cial language. Nevertheless, the lack of freely available resources for this language has limited such work, particularly in the case of IR. This way, the existence of a Portuguese corpus for IR evaluation is very interesting because of the proximity of both languages, Galician and Portuguese, since they were a single language in their origin, and their linguistic phenomena are still very similar today.

The document collection used for this task comprises news published during 1994 and 1995 by the newspapers Publico |Portuguese| and Folha de Sa~o Paulo |Brazilian. See column PT of Table 1 for more details. The test set includes 50 topics (C301{C350). Only title and description elds were used in the submitted queries.

4Inverse Document Frequency model with Laplace after-e ect and normalization 2. PTI

PTD ferred to InL2 model, and the hypergeometric model named DLH5. The results obtained are shown in Table 2, columns PTI and PTD, respectively. The performance of the system is measured using the parameters contained in each row: number of documents retrieved, number of relevant documents expected, number of relevant documents retrieved, average precision (non-interpolated) for all relevant documents (averaged over queries), R-precision, binary preference, geometric average precision, precision at 11 standard levels of recall, and precision at N documents retrieved. 4.2

Robust Task

Since our CLIR system is still in its rst stages, we preferred to test it with the most commonly used languages in CLIR before trying more exotic or less-known languages. This is the main reason for participating in the robust task. The robust task is essentially an ad-hoc task which makes use of the topics and collections used from CLEF 2001 to CLEF 2003. The data collections, whose content is described in Table 1, are formed by newspapers and newswires written in six languages: German (DE ), English (EN ), Spanish (ES ), French (FR), Italian (IT ) and Dutch (NL). The test set is formed by 160 topics (C041{C200). This initial set has been divided into two subsets: a so-called training topics subset |formed by topics C050{C059, C070{C079, C100{C109, C120{ C129, C150{159, C180{189|, to be used for tuning purposes, and a so-called test topics subset |formed by the rest of the topics|, for testing purposes. Again, only title and description elds were used in the submitted queries. 4.2.1

Monolingual experiments

We have participated in all the monolingual subtasks of the robust task: German (DE ), English (EN ), Spanish (ES ), French (FR), Italian (IT ) and Dutch (NL). Results are shown in Table 2. 5http://ir.dcs.gla.ac.uk/wiki/HypergeometricModel In this case, we have just participated in the English-to-German bilingual subtask. Since our direct n-gram translation tool was not ready on time, we opted for a similar approach to that used by JHU/APL group in their rst translingual retrieval experiments [ 9 ]. This way, the source language query is rst translated into the target language before splitting it into n-grams to be submitted to the retrieval engine. In our case we have used Altavista's Babel Fish6 for translating the queries. Columns ENDE of Table 3 show the results obtained. 4.2.3

Multilingual experiments

In this nal multilingual task, English has been used as the source language, whereas all six available collections are used for retrieving. As we explained, our direct n-gram translation tool could not be used because it was not ready on time. Our initial baseline runs were submitted instead.

As before, source language |English| queries were translated into each of the target languages using Altavista's Babel Fish. Once translated, they were split into n-grams for querying their corresponding target language collection. Next, the di erent rankings retrieved for each target language collection are normalized. The similarity value or retrieval status value (RSV) of the i th document retrieved is normalized by the maximum and minimum of the ranking [ 6 ] as follows: RSVi0 =

RSVi RSVmax

RSVmin

RSVmin where RSVi is the original similarity value, RSVi0 is the normalized one, and RSVmin and RSVmax are the minimal and maximal similarity values of that ranking, respectively. Once normalized, all individual rankings are merged into the nal output ranking to be retrieved. (3)

Columns ENxx of Table 3 show the results obtained. 5

Conclusions and future work

This paper describes our initial work in the eld of Cross-Language Information Retrieval. Using our past experience in the application of Natural Language Processing techniques to Information Retrieval, and our recent work in Machine Translation (MT), we have developed an n-gram-based system which uses such subwords not only as indexing units, but also as translating units.

This work has been inspired in the previous work of the Johns Hopkins University Applied Physics Lab [ 9, 8, 10, 11 ]. However, its the training algorithm was too slow for our purposes. Thus we decided to develop our own n-gram based retrieval system for testing our ideas. Freely available resources have been used when possible in its design in order to make it more transparent and to minimize e ort. For speeding up the training process, we have opted for a slightly di erent algorithm to the original one, now consisting of two phases. In the rst phase, the slowest one, word-level alignment of the text is made through a statistical alignment tool. In the second phase, n-gram translation scores are computed employing association measures taking as input the translation probabilities calculated in the previous phase. This new approach increases the speed of the training of the process, concentrating most of the complexity in the word-level alignment phase. Another advantage is that the n-gram alignment process can take as input previously existing aligned word lists or even bilingual dictionaries, which should improve the results.

Unfortunately lack of time did not allow us to get our n-gram direct translation system ready on time. Thus we have only included those baseline results to be used in the future evaluation of our approach.

With respect to future work, we intend to complete and test both our n-gram direct translation system and our retrieval module as soon as possible. Once the base system is working, we intend to test the behavior of new association measures [ 19 ].

Acknowledgments

This research has been partially supported by Ministerio de Educacion y Ciencia and FEDER (TIN2004-07246-C03-02), Xunta de Galicia (PGIDIT05PXIC30501PN, PGIDIT05SIN044E), and Direccion Xeral de Investigacion, Desenvolvemento e Innovacion (Programa de Recursos Humanos grants).

References

[1] http://ir.dcs.gla.ac.uk/terrier/ (visited on August 2006 ).

[2] G. Amati and C.J. van Rijsbergen . Probabilistic models of Information Retrieval based on measuring divergence from randomness . ACM Transactions on Information Systems , 20 ( 4 ): 357 { 389 , 2002 .

[3] V ctor M. Darriba , Gabriel P. Lopes, and Tiago Ildefonso . Measuring the impact of cognates in parallel text alignment . In Proc. of 12th Portuguese Conference on Arti cial Intelligence (EPIA 2005 ), Covilha, Portugal, December 5- 8 , pages 334 { 343 . IEEE Press, 2005 .

[4] Philipp Koehn . Europarl: A Parallel Corpus for Statistical Machine Translation . In Proc. of the 10th Machine Translation Summit (MT Summit X) , September 12-16 , 2005 : Phuket, Thailand, pages 79 { 86 , 2005 . Corpus available in http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/europarl/ (visited on August 2006 ).

[5] Philipp Koehn , Franz Josef Och, and Daniel Marcu. Statistical phrase-based translation . In NAACL '03: Proc. of the 2003 Conference of the North American Chapter of the ACL , pages 48 { 54 , Morristown , NJ, USA, 2003 . Association for Computational Linguistics .

[6] Joon Ho Lee . Analyses of multiple evidence combination . In Proc. of SIGIR '97, July 27-31 , Philadelphia, PA, USA, pages 267 { 276 . ACM Press, 1997 .

[7] Christopher D. Manning and Hinrich Schutze. Foundations of Statistical Natural Language Processing . The MIT Press, Cambridge (Massachusetts) and London (England) , 1999 .

[8] Paul McNamee and James May eld. Scalable multilingual information access . In volume 2785 of Lecture Notes in Computer Science , pages 207 { 218 . Springer-Verlag, Berlin-Heidelberg-New York, 2003 .

[9] Paul McNamee and James May eld. Character N-gram Tokenization for European Language Text Retrieval . Information Retrieval , 7 ( 1 -2): 73 { 97 , 2004 .

[10] Paul McNamee and James May eld . JHU/ APL experiments in tokenization and non-word translation . In volume 3237 of Lecture Notes in Computer Science , pages 85 { 97 . SpringerVerlag, Berlin-Heidelberg-New York, 2004 .

[11] Paul McNamee and James May eld. Cross-Language Retrieval Using HAIRCUT at CLEF 2004 . In volume 3491 of Lecture Notes in Computer Science, pages 50 { 59 . Springer-Verlag, Berlin-Heidelberg-New York, 2005 .

[12] Enrique Mendez , Jesus Vilares, and David Cabrero. COLE experiments at QA @ CLEF 2004 Spanish monolingual track . In volume 3491 of Lecture Notes in Computer Science , pages 544 { 551 . Springer-Verlag, Berlin-Heidelberg-New York, 2005 .

[13] Michael P. Oakes and Souvik Banerjee . Regular sound changes for Cross-Language Information Retrieval . In volume 3237 of Lecture Notes in Computer Science , pages 263 { 270 . Springer-Verlag, Berlin-Heidelberg-New York, 2004 .

[14] Franz Josef Och and Hermann Ney . A systematic comparison of various statistical alignment models , 2003 . Source code available at http://www.fjoch.com/GIZA++. html (visited on August 2006 ).

[15] Jacques Savoy . Cross-Language Information Retrieval: experiments based on CLEF 2000 corpora . Information Processing and Management , 39 : 75 { 115 , 2003 .

[16] Jesus Vilares , Miguel A. Alonso , and Francisco J. Ribadas. COLE experiments at CLEF 2003 Spanish monolingual track . In volume 3237 of Lecture Notes in Computer Science , pages 345 { 357 . Springer-Verlag, Berlin-Heidelberg-New York, 2004 .

[17] Jesus Vilares , Miguel A. Alonso , Francisco J. Ribadas , and Manuel Vilares . COLE experiments at CLEF 2002 Spanish monolingual track . In volume 2785 of Lecture Notes in Computer Science , pages 265 { 278 . Springer-Verlag, Berlin-Heidelberg-New York, 2003 .

[18] Manuel Vilares , Francisco J. Ribadas , and Jorge Gran~a. On pattern-matching as query facility . In Alexander Gelbukh, editor, Topics in Computational Linguistics and Intelligent Text Processing, Lecture Notes in Computer Science . Springer-Verlag, Berlin-Heidelberg-New York, 2006 .

[19] Julie Weeds and David Weir. Co-occurrence retrieval: A exible framework for lexical distributional similarity . Computational Linguistics , 31 ( 4 ): 439 { 475 , 2005 .