In the Morpho Challenge 2009 unsupervised algorithms that provide morpheme analyses for words in different languages were evaluated in various practical applications. Morpheme analysis is particularly useful in speech recognition, information retrieval and machine translation for morphologically rich languages where the amount of different word forms is very large. The evaluations consisted of: 1. a comparison to grammatical morphemes, 2. using morphemes instead of words in information retrieval tasks, and 3. combining morpheme and word based systems in statistical machine translation tasks. The evaluation languages in 2009 were: Finnish, Turkish, German, English and Arabic. This overview paper describes the tasks, evaluation methods, and obtained results. The Morpho Challenge is part of the EU Network of Excellence PASCAL Challenge Program and organized in collaboration with CLEF.
Unsupervised morpheme analysis is still one of the important but unsolved tasks in computational
linguistics and its applications, such as speech recognition (ASR) [
It is possible to construct rule based tools that perform morphological analysis quite well, but of the large number of languages in the world, only few have such tools available. This is because the work of human experts to generate the rules or annotate the morpheme analysis of words and texts is expensive. Thus, learning to perform the analysis based on unannotated text collections is an important goal. Even for those languages that already have existing analysis tools, the statistical machine learning methods still propose interesting and competitive alternatives.
The scientific objectives of the Morpho Challenge are: to learn about the word construction in
natural languages, to advance machine learning methodology, and to discover approaches that are
suitable for many languages. In Morpho Challenge 2009, the participants first developed
unsupervised algorithms and submitted their analyses for the word lists in different languages provided by
the organizers. Then various evaluations were carried out using the proposed morpheme analysis
to find out how they performed in different tasks. In 2009 Challenge the evaluations consisted
of both a comparison to grammatical morphemes (Competition 1) and information retrieval and
statistical machine translation tasks. The IR experiments (Competition 2) contained CLEF tasks,
where the all the words in the queries and text corpus were replaced by their morpheme analyses.
In SMT experiments (Competition 3) identical SMT systems using the same data are first trained
using morpheme analysis and words and then combined for the best performance. The SMT tasks
were first time introduced this year and are based on recent work of the organizers in morpheme
based machine translation [
By the submission deadline in 8th August, 2009, ten research groups had submitted algorithms, which were then evaluated by the organizers. The authors and the names of their algorithms are listed in Table 1. The total number of tasks that the algorithms were able to participate in was 11: six in Competition 1, three in Competition 2, and two in Competition 3. The submissions for the different tasks are presented in Table 2. The final number of algorithms per task varied from 6 to 15.
Statistics of the output of the submitted algorithms are briefly presented in Tables 3 – 8 for each of the languages. The average amount of analyses per word is shown in the column “#a”. It is interesting that in contrary to previous years, now all algorithms ended up mostly suggesting only one analysis per word. From the column “#m” we see the average amount of morphemes per analysis, which reflects the level of details the algorithm provides. The total amount of morpheme types is given in the column “lexicon”.
As baseline results for unsupervised morpheme analysis, the organizers provided morpheme
analysis by a publicly available unsupervised algorithm called “Morfessor Categories-MAP” (or
“Morfessor CatMAP” for short) developed at Helsinki University of Technology [
The task was to return the given list of words in each language with the morpheme analysis added after each word. It was required that the morpheme analyses should be obtained by an unsupervised learning algorithm that would preferably be as language independent as possible. In each language, the participants were pointed to a training corpus in which all the words occur (in a sentence), so that the algorithms may also utilize information about the word context. The tasks were the same as in the Morpho Challenge 2008 last year.
The training corpora were the same as in the Morpho Challenge 2008, except for Arabic: 3 million sentences for English, Finnish and German, and 1 million sentences for Turkish in plain unannotated text files that were all downloadable from the Wortschatz collection1 at the University of Leipzig (Germany). The corpora were specially preprocessed for the Morpho Challenge (tokenized, lower-cased, some conversion of character encodings).
For Arabic, we tried this year a very different data set, the Quran, which is smaller (only
78K words), but has also a vowelized version (as well as the unvowelized one) [
vowelized and the unvowelized gold standard analysis, respectively. For all Arabic data, the Arabic writing script were provided as well as the Roman script (Buckwalter transliteration http://www.qamus.org/transliteration.htm.). However, we only morpheme analysis submitted in Roman script, was evaluated.
The exact syntax of the word lists and the required output lists with the suggested morpheme
analyses have been explained in [
In Competition 1 the proposed unsupervised morpheme analyses were compared to the correct
grammatical morpheme analyses called here the linguistic gold standard. The gold standard
morpheme analyses were prepared in exactly the same format as the result file the participants
were asked to submit, alternative analyses separated by commas. For the other languages except
Arabic, the gold standard reference analyses were the same as in the Morpho Challenge 2007
[
The evaluation of Competition 1 in Morpho Challenge 2009 was similar as in Morpho Challenges 2007 and 2008, but few changes were made to the evaluation measure: small bugs related to the handling of alternative analyses are fixed from the scripts, and points were now measured as one per word, not one per word pair. The data sets were the same as before for English, Finnish, German and Turkish. For Arabic, we had a new data set, the Quran, which was somewhat smaller (only 78K words) than the data set used in 2008, but has also a vowelized version (as well as the unvowelized one). The text corpus was also made available. The participants could try to analyze the vowelized words or the unvowelized, or both, and they were evaluated separately against the vowelized or the unvowelized gold standard analysis, respectively.
Because the morpheme analysis candidates are achieved by unsupervised learning, the morpheme labels can be arbitrary and different from the ones designed by linguists. The basis of the evaluation is, thus, to compare whether any two word forms that contain the same morpheme according to the participants’ algorithm also has a morpheme in common according to the gold standard and vice versa. In practice, the evaluation is performed by randomly sampling a large number of morpheme sharing word pairs from the compared analyses. Then the precision is calculated as the proportion of morpheme sharing word pairs in the participant’s sample that really has a morpheme in common according to the gold standard. Correspondingly, the recall is calculated as the proportion of morpheme sharing word pairs in the gold standard sample that also exist in the participant’s submission. The sample size in different languages varied depending on the size of the word lists and gold standard: 200,000 (Finnish), 50,000 (Turkish), 50,000 (German), 10,000 (English), and 5,000 (Arabic) word pairs.
Precision was calculated as follows: A number of word forms were randomly sampled from the result file provided by the participants; for each morpheme in these words, another word containing the same morpheme was chosen from the result file by random (if such a word existed). We thus obtained a number of word pairs such that in each pair at least one morpheme is shared between the words in the pair. These pairs were compared to the gold standard; a point was given if the word pair had at least the same number of common morphemes according to the gold standard as they had in the proposed analysis. If the gold standard had common morphemes, but less than proposed, fractions of points were given. In the case of alternative analyses in the gold standard, the best matching alternative was used. The maximum number of points for one sampled word was normalized to one. The total number of points was then divided by the total number of sampled words.
For instance, assume that the proposed analysis of the English word “abyss” is “abys +s”. Two word pairs are formed: Say that “abyss” happens to share the morpheme “abys” with the word “abysses”; we thus obtain the word pair “abyss - abysses”. Also assume that “abyss” shares the morpheme “+s” with the word “mountains”; this produces the pair “abyss - mountains”. Now, according to the gold standard the correct analyses of these words are: “abyss N”, “abyss N +PL”, “mountain N +PL”, respectively. The pair “abyss - abysses” is correct (common morpheme: “abyss N”), but the pair “abyss - mountain” is incorrect (no morpheme in common). Precision for the word “abyss” is thus 1/2 = 50%.
For words that had several alternative analyses, as well as for word pairs that have more than one morpheme in common, normalization of the points was carried out. In short, an equal weight is given for each alternative analysis, as well as each word pair in an analysis. E.g., if a word has three alternative analyses, the first analysis has four morphemes, and the first word pair in that analysis has two morphemes in common, each of the two common morphemes will amount to 1/3 ∗ 1/4 ∗ 1/2 = 1/24 of the one point available for that word.
Recall was calculated analogously to precision: A number of word forms were randomly sampled from the gold standard file; for each morpheme in these words, another word containing the same morpheme was chosen from the gold standard by random (if such a word existed). The word pairs were then compared to the analyses provided by the participants; a full point was given for each sampled word pair that had at least as many morphemes in common also in the analyses proposed by the participants’ algorithm. Again, points per word was normalized to one and the total number of points was divided by the total number of words.
The F-measure, which is the harmonic mean of precision and recall, was selected as the final evaluation measure:
F-measure = 1/(1/Precision + 1/Recall) . (1) 3.3
The results of the Competition 1 are presented in Tables 9–14. In three languages, Turkish, Finnish and German, the algorithms with the clearly highest F-measure were “ParaMor-Morfessor Mimic” and “Union”. In English, however, “Allomorfessor” was better and also the algorithm by Lignos et al. was quite close. In Arabic, the results turned out quite surprising, because most algorithms gave rather low recall and F-measure and nobody was able to beat the simple “letters” reference. “Promodes” and “Ungrade” methods scored clearly better than the rest of the participants in Arabic.
The tables contain also results of the best algorithms from Morpho Challenges 2008 [
Morpheme analysis is important in a text retrieval task because the user will want to retrieve all documents irrespective of which word forms are used in the query and in the text. Of the tested languages, Finnish is the most complex morphologically and is expected to gain most from a successful analysis. Compound words are typical of German while English is morphologically the simplest.
The participants’ submissions were compared against a number of reference methods. Like
the participants’ methods, Morfessor baseline [
In a text retrieval task, the user formulates their information need to a query and the system has to return all documents from the collection that satisfy the user’s infomation need. To evaluate the performance of a retrieval system, a collection of documents, a number of test queries and a set of human relevance assessments are needed.
In Competition 2, the participants’ only task was to provide segmentations for the given word lists. The word lists were extracted from the test corpora and queries. In addition, the words in the Competition 1 word lists were added to the Competition 2 lists. Optionally, the participants could also register to the Cross-Language Evaluation Forum (CLEF)2 and use the full text corpora for preparing the morpheme analysis. The IR experiments were performed by the Morpho Challenge organizers by using the submitted word lists to replace the words both in the documents and in the queries by their proposed analyses.
The corpora, queries and relevance assessments were provided by CLEF and contained news paper articles as follows: • In Finnish: 55K documents from short articles in Aamulehti 1994-95, 50 test queries on specific news topics and 23K binary relevance assessments (CLEF 2004) • In English: 170K documents from short articles in Los Angeles Times 1994 and Glasgow Herald 1995, 50 test queries on specific news topics and 20K binary relevance assessments (CLEF 2005). • In German: 300K documents from short articles in Frankfurter Rundschau 1994, Der Spiegel 1994-95 and SDA German 1994-95, 60 test queries with 23K binary relevance assessments (CLEF 2003). 4.2
The performance of the participating algorithms was compared to a number of reference methods.
Some these methods are commonly used in IR and the purpose of providing these methods is to
evaluate the usefulness of the unsupervised algorithms for the task. The reference methods are
the same as used in Morpho Challenge 2008 [
English, German and Finnish IR tasks were used to evaluate the submitted morpheme analyses. Unfortunately, neither Turkish or Arabic IR test corpora were available for the organizers. The experiments were performed by replacing the words in the corpus and the queries by the submitted morpheme analyses. Thus, the retrieval was based on morphemes as index terms. If a segmentation for a word was not provided, it was left unsegmented and used as a separate morpheme. The queries were formed by using the title and description (“TD”) fields from the topic descriptions.
The IR experiments were performed using the freely available LEMUR toolkit4 version 4.4. The popular Okapi BM25 ranking function was used. In the 2007 challenge [14], it was noted that the performance of Okapi BM25 suffers greatly if the corpus contains morphemes that are very common. The unsupervised morpheme segmentation algorithms tend to introduce such morphemes when they e.g. separate suffixes. To overcome this problem, a method for automatically generating a stoplist was introduced. Any term that has a collection frequency higher than 75000 (Finnish) or 150000 (German and English) is added to the stoplist and thus excluded from the corpus. Even though the method is quite simplistic, it generates reasonable sized stoplists (about 50-200 terms) and is robust with respect to the cutoff parameter. With a stoplist, Okapi BM25 clearly outperformed TFIDF ranking and thus the approach has been adopted for later evaluations as well. The evaluation criterion for the IR performance is the Mean Average Precision (MAP) that was calculated using the trec eval program. 4.4
Three research groups submitted total of five different segmentations for the Competition 2 word lists. In addition, for the 6 groups and 10 algorithms that did not provide segmentations for the Competition 2 word lists, the smaller Competition 1 word list was used. None of the participants used the option to use the full text corpora to provide analyses for words in their context.
Tables 15, 16 and 17 show the obtained MAP values for the submissions in English, German and Finnish respectively. For English, the best performance was achieved by the algorithm by Lignos et al. even though only the shorter Competition 1 word list was available for evaluation. “ParaMor-Morfessor Mimic” and “ParaMor-Morfessor Union” by Monson et. al gave the best performance for German and Finnish respectively. Overall, the algorithms by Monson et al., especially “ParaMor-Morfessor Union”, gave good performance across all tested languages. Also, “Allomorfessor” by Virpioja & Kohonen was a solid performer in all languages. However, none of the submitted algorithms could beat the winners of last year’s competition.
In all languages, the best performance was achieved by one of the reference algorithms. The rule based word normalizer, TWOL, gave best performance in German and Finnish. In the
4http://www.lemurproject.org/
English task, TWOL was only narrowly beaten by the traditional Porter stemmer. For German
and Finnish, stemming was not nearly as efficient. Of the other reference methods, “Morfessor
Baseline” gave good performance in all languages while the “grammatical” reference based on
linguistic analyses did not perform well probably because the gold standards are quite small.
For practical reasons, a limited set of queries (50-60) are used in evaluation of the IR-performance.
The obtained results will include variation between queries as well as between methods. Statistical
testing was employed to determine what differences in performance between the submissions are
greater than expected by pure chance. The methodology we use follows closely the one used in
TREC [10] and CLEF [
Analysis was performed with Two-way ANOVA using MATLAB Statistics Toolbox. Since ANOVA assumes the samples to be normally distributed, a transformation for the average precision values was made with the arcsin-root function: f (x) = arcsin(√x). (2) The transformation makes the samples more normally distributed. Statistical significances were examined using MATLAB’s multcompare function with the Tukey t-test and 0.05 confidence level.
Results of the test are summarized for English, German and Finnish in Figures 1, 2 and 3 respectively. Participant or reference submission name is shown on the y-axis and the performance on the x-axis. The average performance of the method is indicated by a circle and the bars show the confidence interval in which the difference in performance is not statistically significant. The “top group” or the submissions that have no significant difference to the best result of each language is highlighted.
The confidence intervals are relatively wide and a large proportion of the submissions are in
the top group for all languages. It is well known and also noted in the CLEF Ad Hoc track [
One interesting comparison is to see if there are significant differences to the “dummy” case where no morphological analysis is performed. For German and Finnish, “ParaMor-Morfessor Union” is the only submission that is significantly better than the dummy method. For English, none of the participants’ results can significantly improve over “dummy”. Only the Porter stemmer is significantly better according to the test. 4.6
The results of the Competition 2 suggest that unsupervised morphological analysis is a viable approach for information retrieval. Some of the unsupervised methods were able to beat the “dummy” baseline and the best were close to the language specific rule-based “TWOL” word normalizer. However, this year’s competition did not offer any improvements to previous results.
The fact that segmentations of the full Competition 2 word list was not provided by all participants makes the comparison of IR performance a bit more difficult. The participants that were evaluated using only the Competition 1 word lists had a disadvantage, because then the additional words in the IR task were indexed as such without analysis. In the experiments in Morpho Challenge 2007 [14], the segmentation of the additional words improved performance in the Finnish task for almost all participants. In German and English tasks the improvements were small. However, if the segmentation algorithm is not performing well, leaving some of the words unsegmented only improves the results for that participant.
Most of the methods that performed well in the Competition 2 IR task were also strong in the corresponding linguistic evaluation of Competition 1 and vice versa. The biggest exeptions were in the Finnish task where the “PROMODES committee” algorithm gave reasonably good results in the linguistic evaluation but not in the IR task. The algorithm seems to oversegment words Tukey T−test for English and the suggested morphemes give good results when compared to gold standard analysis but do not seem to work well as index terms. On the other hand, “Allomorfessor” and the “Morfessor Baseline”methods performed well in the IR task but were not at the top in the linguistic evaluation where they suffered from low recall. In general, it seems that precision in the Competition 1 evaluation is a better predictor of IR performance than recall or F-measure.
The statistical testing revealed very few significant differences in the IR performance between participants. This is typical for the task. However, we feel that testing the algorithms in a realistic application gives information about the performance of the algorithms that the linguistic comparison can not offer alone.
The participants were offered a chance to access the IR corpus to use the full text context in the unsupervised morpheme analysis. However, this version of task was not attempted by anyone. We are thinking of ways to make this version of task more accessible for competitors as using the context of words seems like a natural way to improve the models. Other future work includes expanding the IR task to new languages like Arabic which pose new kinds of morphological problems. 5
In Competition 3, the morpheme analyses proposed by the participants’ algorithm were evaluated in a statistical machine translation (SMT) framework. The translation models were trained to translate from a morphologically complex source language to English. The words of the source language were replaced by their morpheme analyses before training the translation models. The two source languages used in the competition were Finnish and German. Both the input data for the participants’ algorithms and training the SMT system were from the proceedings of the European Parliament. The final SMT systems were evaluated by measuring the similarity of the translation results to a human-made reference translation. 5.1
As a data set, we used Finnish-English and German-English parts of the European Parliament parallel corpus (release v2) [11]. The participants were given a list of word forms extracted from the corpora, and similarly to the Competitions 1 and 2, they were asked to apply their algorithms to the word list, and return the morphological analyses for the words. It was also possible to use the context information of the words by downloading the full corpus. Furthermore, the data sets from Competitions 1 and 2 were allowed to use for training the morpheme analyses. However, they were used by none of the participants.
For training and testing the SMT systems, the Europarl data sets were divided into three subsets: training set for training the models, development set for tuning the model parameters, and test set for evaluating the translations. For the Finnish-English systems, we had 1 180 603 sentences for training, 2 849 for tuning, and 3 000 for testing. For the German-English systems, we had 1 293 626 sentences for training, 2 665 for tuning, and 3 000 for testing. 5.2 In principle, the evaluation is simple: First, we train a translation system that can translate the morphologically analyzed Finnish or German sentence to English. Then, we use it to translate new sentences, and compare the translation results to the reference translations. If the morphological analysis is good, it reduces the sparsity of the data and helps the translation task. If the analysis contains many errors, they should degrade the translation results. However, a SMT system has many components and parameters that can affect the overall results. Here we describe the full evaluation procedure in detail.
As the SMT models and tools are mainly designed for word-based translations, the results
obtained for morpheme-based models are rarely better than the word-based baseline models (see,
e.g., [
Training phrase-based SMT systems The individual models, including the baseline word-to-word model and the morpheme-to-word models based on the participants’ methods, were trained with the open source Moses system [12]. Moses translates sequences of tokens, called phrases, at a time. The decoder finds the most probable hypothesis as a sequence of target language tokens, given a sequence of tokens in source language, a language model, a translation model and possible additional models, such as a reordering model for phrases in the hypothesis.
Training a translation model with Moses includes three main steps: (1) alignment of the tokens in the sentence pairs (2) extracting the phrases from the aligned data, and (3) scoring the extracted phrases. As there are more morphemes than words in a sentence, two limitations affect the results: First, the alignment tool cannot align sentences longer than 100 tokens. Second, the phrases have a maximum length, which we set to be 10 for the morpheme-based models.
The weights of the different components (translation model, language model, etc.) are tuned
by maximizing the BLEU score [
Minimum Bayes-Risk decoding for system combination Minimum Bayes-Risk (MBR) decoding for machine translation [13] selects the translation hypothesis that has the lowest expected risk given the underlying probabilistic model. For loss function L bounded by maximum loss Lmax, we choose the hypothesis that maximises the conditional expected gain according to the decision rule (3) (4) Eˆ = argmax X G(E, E′)P (E|F ),
E′∈E E∈E
where G(E, E′) = Lmax − L(E, E′) is the gain between reference E and hypothesis E′ and P (E|F )
is the posterior probability of translation. The search is performed over all hypotheses E′ in the
evidence space E, typically an n-best list or lattice. An appropriate gain function for machine
translation is the sentence-level BLEU score [
Eˆ = argmax θ0|E′| + X θw#w(E′)p(w|E) ,
E′∈E w∈N
where p(w|E) is the posterior probability of the n-gram w and N = {w1, . . . , w|N |} denotes the
set of all n-grams in the evidence space. The posterior probabilities are computed efficiently using
the OpenFst toolkit [
We used minimum Bayes-risk system combination [
Between the results from the MBR combinations, only some of the differences are statistically significant. The significances were inspected with paired t-test on ten subsets of the test data. In the Finnish to English task, Morfessor Baseline, Allomorfessor, Morfessor CatMAP and MetaMorph are all significantly better than the rest of the algorithms. Between them, the difference between Allomorfessor and the both Morfessor algorithms is not significant, but Allomorfessor and Morfessor Baseline are significantly better than MetaMorph. The differences between the results of the last four algorithms (MorphoNet and ParaMor:s) are not statistically significant. Neither they are significantly better than the word-based system alone.
In the German to English task, only the results of Morfessor Baseline and Allomorfessor have significant differences to the rest of the systems. Morfessor Baseline is significantly better than any of the others expect Allomorfessor and ParaMor Mimic. Allomorfessor is significantly better than the others expect Morfessor Baseline, ParaMor Mimic, ParaMor-Morfessor Mimic and Morfessor CatMAP. None of the rest of the MBR results is significantly higher than the word-based result.
Overall, the Morfessor family of algorithms performed very well in both translation tasks. Categories-MAP was not as good as Morfessor Baseline or Allomorfessor, which is probably explained by the fact that it segmented words to shorter tokens. Also MetaMorph improved significantly the Finnish translations, but was not as useful in German. 5.4
This was the first time that machine translation system was used to evaluate the quality of the morphological analysis. As the SMT tools applied are designed mostly for word-based translations, it was not a surprise that some problems arose.
The word alignment tool used by the Moses system, Giza++, has strict limits on sentence lengths. A sentence cannot be longer than 100 tokens, and neither over 9 times longer or shorter than its sentence pair. Too long sentences are pruned away from the training data. Thus, the algorithms that segmented more, generally got less training data for the translation model. However, the dependency between average tokens per word and the amount of filtered training data was not linear, as seen from the left side of Figure 4. For example, the Morfessor CatMAP system could use much more training data than some of the algorithms that, on average, segmented less. Even without considering the decrease to the amount of training data available, oversegmentation is likely to be detrimental in the task, because it makes, e.g., the word alignment problem more complex.
After MBR combination, the rank of the algorithms was not the same as with the individual systems. The respective scores are plotted in the right side of Figure 4. Especially ParaMorMorfessor Union system helped the word-based model more than its own BLEU score indicated. However, as the improvements were not statistically significant, the improved rank in the MBR combination may be affected more by just chance.
In addition to making the evaluation more fair to the algorithms that use shorter tokens that the others, future work includes testing TWOL-based morphological analyses or gold standard segmentations in the task. 0.24 0.26 0.28 0.3 BLEU score (individual systems) 0.32 The Morpho Challenge 2009 was a successful follow-up to our previous Morpho Challenges 20052008. Since some of the tasks were unchanged from 2008, the participants of the previous challenges were able to track improvements of their algorithms. It also gave a possibility for the new participants and those who missed the previous deadlines to try more established benchmark tasks. New tasks were introduced for statistical machine translation which offer yet another viewpoint on what is required from morpheme analysis in practical applications.
We thank all the participants for their submissions and enthusiasm. We owe great thanks as well to the organizers of the PASCAL Challenge Program and CLEF who helped us organize this challenge and the challenge workshop. We are most grateful to the University of Leipzig for making the training data resources available to the Challenge, and in particular we thank Stefan Bordag for his kind assistance. We are indebted to Ebru Arisoy for making the Turkish gold standard available to us. We are most grateful to Majdi Sawalha and Eric Atwell from the University of Leeds for making the Arabic data available to the Challenge and for their kind assistance in preparing it to meet the Challenge file formats. We acknowledge also the Computational Linguistics Group at University of Haifa who supplied their tagged Arabic database. This work was supported by the Academy of Finland in the project Adaptive Informatics, the graduate schools in Language Technology and Computational Methods of Information Technology, in part by the GALE program of the Defense Advanced Research Projects Agency, Contract No. HR0011-06-C-0022, and in part by the IST Programme of the European Community, under the FP7 project EMIME (213845) and PASCAL Network of Excellence. This publication only reflects the authors’ views. We acknowledge that access rights to data and other materials are restricted due to other commitments. [7] Mathias Creutz and Krista Lagus. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor. Technical Report A81, Publications in Computer and Information Science, Helsinki University of Technology, 2005. URL: http://www.cis.hut.fi/projects/morpho/. [8] Mathias Creutz and Krister Linden. Morpheme segmentation gold standards for finnish and english. Technical Report A77, Publications in Computer and Information Science, Helsinki University of Technology, 2004. URL: http://www.cis.hut.fi/projects/morpho/. [9] Adri`a de Gispert, Sami Virpioja, Mikko Kurimo, and William Byrne. Minimum bayes risk combination of translation hypotheses from alternative morphological decompositions. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pages 73–76, Boulder, USA, June 2009. Association for Computational Linguistics. [10] David A. Hull. Using statistical testing in the evaluation of retrieval experiments. In SIGIR ’93: Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, pages 329–338, New York, NY, USA, 1993. ACM Press. [11] Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit, pages 79–86, Phuket, Thailand, 2005. [12] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Annual Meeting of ACL, demonstration session, Czech Republic, June 2007. [13] Shankar Kumar and William Byrne. Minimum Bayes-Risk decoding for statistical machine translation. In Proceedings of Human Language Technologies: The 2004 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 169– 176, 2004. [14] Mikko Kurimo, Mathias Creutz, and Ville Turunen. Unsupervised morpheme analysis evaluation by IR experiments – Morpho Challenge 2007. In Working Notes for the CLEF 2007 Workshop, Budapest, Hungary, 2007.