This paper describes the participation of the IXA group from the UPV/EHU (University of the Basque Country) in the TweetMT shared task at the SEPLN-2015 conference. We have adapted existing MT engines for the es-eu and eu-es pairs, obtaining good results (better than other experiments reported in previous work). Three main aspects are described: resource compilation, engine adaptation and results.
As the organizers of the workshop say in
the home page1 \the machine translation
of tweets is a complex task that greatly
depends on the type of data we work with.
The translation process of tweets is very
di erent from that of correct texts posted for
instance through a content manager. The
texts also vary in terms of structure, where
the latter include tweet-speci c features such
as hashtags, user mentions, and retweets,
among others." The translation of tweets
can be tackled as a direct translation
(tweet-to-tweet) or as an indirect translation
(tweet normalization to standard text, text
translation and, if needed, tweet generation)
When analyzing the released development
corpus we observed that most of the messages
were formal tweets, and we therefore decided
to face the problem following the direct
approach, adapting previous engines to the
structure of these texts. We have adapted
three systems:
an RBMT system named Matxin for the
es-eu pair
Based on the development set provided, preliminary work was carried out to obtain useful resources to adapt the systems: an out-of-vocabulary (OOV) dictionary was obtained for Basque using our Basque morphological analyzer. We observed that the percentage of OOVs was low and only few of them were common in the development set. Even if there are a very small number of entries, we built a bilingual dictionary with the most frequent OOVs (5 entries). a dictionary of bilingual hashtags was obtained by aligning hashtags (using a simple program and manual revision) from parallel tweets. After a manual review of the pairs with more than two occurrences, a dictionary of 60 pairs was generated.
Tweets of monolingual corpora from previous shared tasks were compiled in order to enrich the language models.
Corpora from the TweetNorm shared
task
In order to increase the low volume of data for the Basque LM we use a corpus of tweets supplied by the CodeSyntax company2 which has a tweet-oriented service for Basque called UMAP3. They identify Twitter accounts that use Basque and compile the tweets from these users. Most of these accounts are multilingual, so language identi cation was a key next step. We used our language identi er (LangId, a free language identi er based on word and trigram frequencies developed by the IXA group of the University of the Basque Country, and which is specialized in recognizing Basque and its surrounding languages (Spanish, French and English)) and lter candidates with high percentage of OOVs (thus, priorizing formal tweets and adding precision to the results obtained from Langid) and compiled a corpus of 454,790 tweets. 3
The RBMT system was adapted to the task manually and two new models for the SMT systems were trained and tuned. 3.1
As mentioned, the Matxin system was used for es-eu translation. Matxin is an open source Spanish-to-Basque RBMT engine which follows the traditional transfer model. It consists of three main components: 1) analysis of the source sentence into a dependency tree structure; 2) transfer from the source language dependency tree to a target language dependency structure; and 3) 2http://www.codesyntax.com 3http://umap.eu generation of the output translation from the target dependency structure.
Matxin was adapted to the idiosyncratic
features of tweets (URLs, hashtags...).
For this purpose, the de-formatter module
in Matxin
Hashtags inside the text are given for translation. Some will be translated (#Escocia / #Eskozia) while others will remain untranslated (#Hackathon, #Olasdeenergia) IDs will receive the same treatment as other named entities 3.2
The adaptation and tuning of the SMT systems was laborious. First of all, the provided development corpus was divided into 3 subsets: training (2,000 pairs), tuning (1,500) and test (500). Because the alignment of the corpus was automatically done, we manually reviewed the training part and observed that the error rate was high. After discarding non-parallel tweets 1,444 pairs remained in the training corpus.
We used a previously compiled parallel
corpus for the translation model. This 7.4
million segment corpus was compiled by the
Elhuyar Foundation and the University of
the Basque Country. It includes public
corpora, private corpora and a corpus built
by web-as-corpus paradigm
For the language model, previous models for Spanish and Basque were retrained adding the corpora described in the previous section.
Table 1 shows the gures for the corpora used.
Bilingual
General 7,463,951 Tweets 1,444 General 28,823,939 Tweets 213,141 General 1,290,501 Tweets 454,800 The development of the system was carried out using publicly available state-of-the-art tools: the GIZA++ toolkit, the SRILM toolkit and the Moses decoder. More concretely, we followed the phrase-based approach with standard parameters: a maximum length of 80 tokens per sentence, translation probabilities in both directions with Good Turing discounting, word-based translation probabilities (lexical model, in both directions), a phrase length penalty and the target language model. The weights were adjusted using MERT tuning with n-best list of size 100.
For the idiosyncratic features of the tweets we analyzed the errors when the system was applied in the test extracted from the development corpus and we decided to implement the following pre- and post-processing steps:
Tokenization: special treatment of hyphens (`-') before declension cases of IDs, hashtags, gures, time...
Post-processing: simple rules for xing recurrent surface-errors: double hyphen or colon, special symbols (e.g. `>' is used in Spanish but not in Basque) and similar issues. 4
The systems prepared and tuned using the development corpus were directly used to process the test. So we presented two systems for the eu-es pair (RBMT and SMT) and one system (SMT) for the es-eu pair. For these language pairs only another group presented results (3 systems).
Table 2 shows the results on the test
corpus provided. We use the most common
measures: BLEU and NIST
We want to underline that the results for
the es-eu pair are better than previous results
reported in some papers
These results are surprising if we consider tweet texts in general, but note that all tweets used in the shared-task are formal and that the most of them were designed to be multilingual (and so, perhaps, to be easily translated). Therefore, we could say that the task was easier than usual tasks in MT, at least for this language pair.
The good performance of the RBMT system on the formal tweets was expected, as syntax use to be simple in the short texts from Twitter.
In Table 3 there some examples of the results. In sentences #4, #5 and #6 RBMT gets very food translations but in the previous sentences the translations from the SMT system are more precise. In the near future we want to check if combining both techniques improvements can be lead.
We can draw the following general conclusions: These results cannot be extrapolated to the general task of translating tweets. Translating informal tweets will be much harder.
MT can help community managers who manage multilingual Twitter accounts. A Twitter oriented MT post-editing system could be developed and evaluated.
This work has been supported by the Spanish MICINN project Tacardi (Grant No. TIN2012-38523-C02-01). CodeSyntax company and Elhuyar Foundation have collaborated with us providing several corpora for the translation and language models. Thanks to Josu Azpillaga (CodeSyntax) and to In~aki San Vicente, Igor Leturia, Itziar Cortes and Justyna Pietrzak (Elhuyar) for their assistance. We would also like to thank the anonymous referees for their comments and suggestions.