We describe our participation in TweetMT for three language pairs in both directions: Spanish from/to Catalan, Basque and Portuguese. We used a range of techniques: statistical and rule-based MT, morph segmentation, data selection with ParFDA and system combination. As for resources, our focus was on crawling vast amounts of tweets to perform monolingual domain adaptation. Our system was the best of all systems submitted for ve out of the six language directions.
While statistical machine translation (SMT) can be considered a mature technology nowadays, one of its requirements is the availability of considerable amounts of parallel text for the language pair of interest. Ideally, the parallel text to train an SMT system should come from the same domain and genre as the text the system is going to be applied to. Thus, using MT to translate types of text for which no parallel data is available constitutes a challenge. This is the case for tweets and social media in general, the target text of the TweetMT shared task.
The main objective of our participation in the TweetMT 2015 shared task was to build the best MT systems for tweets we could with a clear constraint, i.e. it had to be done in a very short period and, to a large extent, be limited to available resources. We have taken part for three language pairs in both directions: Spanish (ES) from/to Catalan (CA), Basque (EU) and Portuguese (PT).
We decided to focus on making the best
possible use of available techniques, tools and
resources. Regarding techniques and tools,
we rely on state-of-the-art SMT, morph
segmentation for morphologically rich languages
(EU), data selection with ParFDA for fast
development of accurate SMT systems
The rest of the paper is organised as follows. Sections 2 and 3 detail the systems built and the resources used, respectively. Section 4 presents the evaluation and, nally, Section 5 outlines conclusions and lines of future work. 2
Here we describe the components used in our translation pipeline. First, we pre-process the datasets (Section 2.1), then we use a set of MT systems (Section 2.2) that can incorporate additional functionality (Sections 2.3 and 2.4). Finally, we combine MT systems (Section 2.5). 2.1
Prior to be used, all the datasets used in our systems are preprocessed, as follows: 1. Punctuation normalisation,
Moses'
Freeling
We build SMT systems using two paradigms:
phrase-based with Moses
The SMT systems use 5-gram LMs with
Knesser-Ney smoothing
Morphological segmentation is a popular method to deal with SMT for morphologically di ering languages by simply splitting words into sub-word units. The main bene ts of morphological segmentation are to reduce the out-of-vocabulary (OOV) rate and to increase the percentage of 1 to 1 word alignments between morphosyntactically di erent languages; e.g. in our case, by matching inectional su xes in EU to syntactic prepositions in ES, we expect to improve the MT quality for the EU{ES language pair. The segmentation and de-segmentation is able to create word-forms not present in the training data by matching a translated stem with a correct su x.
In our participation, morphological segmentation was only used for EU{ES on the EU side, since EU's morphology is signi cantly more complex than that of ES. For the remaining languages of the shared task, there is no such big di erence in morphology complexity (all of them are closely-related as they belong to the same family) so the expected gains do not outweigh the added complexity of segmentation.
We use unsupervised statistical
segmentation as provided by Morfessor 2.0
Baseline
To gauge the e ects of our method as well as the morphological complexity of EU as compared to ES we show in Table 1 the OOV rates and vocabulary sizes of the ES and EU sides of the ES{EU training corpus, and EU corpora after morphological segmentation. Segmentation reduces the type-totoken ratio by a factor of 6 and the OOV rate by almost a factor of 10. 2.4
ParFDA parallelizes instance selection with an optimized parallel implementation of 3http://www.cis.hut.fi/projects/morpho/ morfessor2.shtml
FDA5 and signi cantly reduces the time
to deploy accurate SMT systems especially
in the presence of large training data and
still achieve state-of-the-art SMT
performance
For each language direction we have built up
to ve systems, as detailed in Sections 2.2
to 2.4: (i) phrase-based and (ii) hierarchical
SMT, (iii) phrase-based with morph
segmentation, (iv) phrase-based with ParFDA and
(v) RBMT. We hypothesise these systems to
have complementary strengths, and thus we
decide to perform system combination. To
that end we use MEMT
Ideally, we would use data in the same domain and genre as the test set, i.e. tweets. We have access to parallel tweets provided by the task for ES{CA and ES{EU (4,000 parallel tweets for each language pair, we use 1,000 for dev and the remaining 3,000 for training). For ES{PT we have access to 999 parallel tweets (we use them for dev) from Brazilator,4 a recent project by DCU and Microsoft to translate tweets from the 2014 soccer World Cup across 24 language directions.
As the availability of parallel tweets for
the language pairs of TweetMT 2015 is rather
limited (at most we have 4,000 per language
pair), we use additional sources of
parallel data. For ES{CA we use elPeriodico
(eP)5 and a selection of contemporary
novels. For ES{EU, translation memories (TMs)
provided by the shared task6 and two corpora
from Opus
Our main source of monolingual data is
indomain and comes from crawled tweets. We
use TweetCat
For each language we create two lists of words as required by the crawler: (i) most common discriminating words (up to 100), these are words that are unique to the language and they are used to seed the crawler so that it can nd candidate tweets; and (ii) most common words of the language (200), these are used to determine the language of 4http://www.cngl.ie/brazilator 5http://catalog.elra.info/product_info. php?products_id=1122
6http://komunitatea.elhuyar.org/tweetmt/ resources/ 7http://opus.lingfil.uu.se/ 8http://www.statmt.org/europarl/ crawled tweets. These two lists are derived from a list of the most common words found in a corpus of subtitles.9
The tweets crawled are post-processed with langid10 to identify their language. We keep the tweets whose langid's con dence score is above a certain threshold, which is set empirically at 0.7 by inspecting tweets.
In addition to crawled tweets, we use the
target sides of the parallel corpora (cf.
Section 3.1 and a set of monolingual corpora as
follows. For CA we use caWaC
Table 4 shows details on these corpora including their interpolation weights (cf. Section 2.2). 4
We report our results on the development set (all systems built) and then on the test set (systems submitted).
Table 5 presents the results obtained on the devset by the individual systems and a set of 9https://onedrive.live.com/?cid= 3732e80b128d016f&id=3732E80B128D016F!3584 10https://github.com/saffsd/langid.py 11http://www.statmt.org/wmt13/ 12http://www.linguateca.pt/cetempublico/ 13http://www.linguateca.pt/cetenfolha/
CA ES EU PT
Corpus
tweets
caWaC
eP
tweets
news
europarl
tweets
Wikipedia
TMs
tweets
Wikipedia
Others
combinations for the three language pairs we
covered: ES{CA, ES{EU and ES{PT. The
scores were obtained on raw MT output (i.e.
tokenised and truecased) as calculated by us
with BLEU
At least one of the combinations obtains better scores (both in terms of BLEU and TER) than the best individual system (except for ES$PT with BLEU and for CA!ES with TER), supporting our hypothesis that the individual systems built are complementary. Although SMT systems outperform RBMT systems for all directions,14 the addition of RBMT in system combinations has a positive impact (except for ES$PT). Phrasebased SMT outperforms hierarchical SMT for related language pairs (ES{CA and ES{PT), but the opposite is true for the unrelated language pair ES{EU. We hypothesise this is due to the fact that ES and EU follow different word orders (SVO and SOV, respectively), and this leads to pervasive long reorderings in translation, that are better modelled with a hierarchical approach.
14When interpreting the results, it should be taken
into account that automatic metrics are known to be
biased towards statistical MT approaches
Moses (1) cdec (2) A ParFDA (3) C Apertium (4) S!1+2 E 1+4 1+2+4 Moses (1) cdec (2) S ParFDA (3) E Apertium (4) !1+2 A C 1+4 1+2+4 1+3+4 Moses (1) cdec (2) ParFDA (3)
Matxin (4) EU Morph (5) !1+2 S E 1+4 1+2+4 1+2+4+5 1+2+3+4+5 Moses (1) cdec (2)
ParFDA (3) S Apertium (4) E!Morph (5) U 1+2 E 1+4 1+2+4 1+2+4+5 Moses (1) cdec (2) T ParFDA (3) P Apertium (4) ! S 1+2 E 1+4 1+2+4 Moses (1) cdec (2) S ParFDA (3) E !Apertium (4) T 1+2 P 1+4 1+2+4 Table 6 presents the results on the test set of the systems we submitted. The scores shown are the ones reported by the organisers (case-insensitive BLEU and TER) on post-processed MT outputs (detokenised and detruecased). For each language direction we submitted the three systems that obtained the best performance on the dev set. The scores of the best submitted system are shown in bold.
Out of six directions, our best submission is the top performing system for ve of them (indicated with y). For most directions, the addition of a RBMT system leads to better performance. Similarly, for the directions where we have used segmentation (ES$EU) and ParFDA (CA!ES and ES!EU), the addition of systems based on these techniques had a positive impact on the results.
We now delve deeper into the results obtained by SMT systems based on ParFDA (cf. Section 2.4). Although ParFDA systems were submitted to the shared task only as part of system combinations, we have evaluated a posteriori the performance of this technique by means of standalone systems on the test set. ParFDA Moses SMT system obtains top results in CA!ES and ES!CA and close to top results in other language pairs with 1:21 BLEU points average di erence to the top (Table 7). An interesting feature of
ParFDA regards its ability to build and deploy SMT systems in a quick manner. In the speci c case of TweetMT, ParFDA took about 8 hours to build for ES!CA and 28 hours for PT!ES taking about 11 GB and 27 GB disk space in total, respectively. 5
This paper has described our participation in the TweetMT 2015 shared task. Our focus has been on rapid development of MT systems adapted to tweets by making the best possible use of available techniques, tools and resources. Our best submissions have been the ones that combine di erent MT systems (except for ES!CA), supporting our hypothesis that the techniques we have used are complementary.
As for future work, we consider several possible avenues. First, we would like to analyse in detail the translations produced by our systems in order to derive ndings beyond the ones we can extract from the automatic evaluation metrics used in the task. Second, most of the tweets in the test set use formal language,15 and thus we would like to test our systems in a more representative set of tweets where informal language would be expected to be more pervasive.
This research is supported by the EU 7th Framework Programme FP7/2007-2013 un15This is due to the fact that they are extracted from twitter accounts that publish tweets in multiple languages, and such accounts belong, to a large extent, to institutions that use formal language. der grant agreement PIAP-GA-2012-324414 (Abu-MaTran), by SFI as part of the ADAPT research center (07/CE/I1142) at Dublin City University and the project \Monolingual and Bilingual Text Quality Judgments with Translation Performance Prediction" (13/TIDA/I2740). We also thank the SFI/HEA Irish Centre for HighEnd Computing (ICHEC) for the provision of computational facilities and support. Finally, we would like to thank Mikel L. Forcada and Iacer Calixto for their advice on normalising tweets for Basque and Portuguese, respectively, and Gorka Labaka for his help with Matxin's API. Association for Computational Linguistics (ACL).