This article presents an overview of the shared task that took place as part of the TweetMT workshop held at SEPLN 2015. The task consisted in translating collections of tweets from and to several languages. The article outlines the data collection and annotation process, the development and evaluation of the shared task, as well as the results achieved by the participants.
While research in machine translation has been studied for a while now, the application of machine translation techniques to tweets is still in its infancy. The machine translation of tweets is a challenging task which, to a great extent, depends on the spelling and grammatical quality of the tweets that one has to provide the translations for. In fact, the di culty of a tweet translation process varies dramatically for di erent types of tweets ranging from informal posts to formal announcements and news headlines posted by social media editors or community managers. The former are often written from mobile devices, which exacerbates the poor quality of the spelling, and include linguistic inaccuracies, symbols and diacritics. Tweets also vary in terms of structure, including features which are exclusively used in the platform, such as hashtags, user mentions, and retweets, among others. These characteristics make the application of machine translation tools to tweets a new problem that requires speci c processing techniques to perform e ectively.
The machine translation of tweets is
usually tackled in two di erent ways: (1) as
a direct translation task (tweet-to-tweet),
or (2) as an indirect translation task (tweet
normalization to standard text
Despite the paucity of research in the
speci c task of translating tweets, an
increasing interest can be observed in the
scienti c community
Provided the dearth of benchmark
resources and comparison studies bringing
to light the potential and shortcomings
of today's machine translation techniques
applied to tweets, we organized TweetMT,
a workshop and shared task1 on machine
translation applied to tweets. This workshop
is a follow-up to two other related workshops
organized in previous years also at SEPLN:
TweetNorm 2013
As a starting point, and especially given the little work performed so far in the eld, the corpora we compiled for the shared task includes tweets that are mostly formal and correctly written, while keeping the brevity inherent to tweets. While the corpora might not be fully representative of the texts that one can nd on Twitter, it is instead intended to boost the work performed within the eld, encouraging researchers to submit preliminary contributions that will then help better understand the state of the art so that future work can be set forth. As this research matures, subsequent corpora will include a wide variety of informal and misspelled tweets to keep making progress. 2
To the best of our knowledge, there is no
parallel tweet dataset available apart from
that produced by
In order to facilitate corpus generation, we developed a semi-automatic method to retrieve and align parallel tweets. The semi-automatic method consists in 1http://komunitatea.elhuyar.eus/tweetmt/ identifying multiple Twitter authors that tweet identical content, albeit in di erent languages, either from a single account or from two di erent accounts. Hence, whenever possible, the parallel corpora have been generated from multilingual Twitter accounts; this methodology was applied for the Catalan{Spanish (ca-es) and Basque{Spanish (eu-es) language pairs, as we found authors that concurrently tweet in these languages. However, we did not nd authors that meet these characteristics for the other two language pairs, i.e., Portuguese{Spanish and Galician{Spanish (pt-es and gl-es); in these cases, the parallel tweets were manually produced through crowdsourcing. Di erent to the language pairs that could be automatically aligned, in the latter cases only test sets were generated due to time and budget constraints.
The following sections give details about the creation of the datasets. Table 1 shows some statistics of those datasets. 2.1
The corpus creation process out of multilingual Twitter accounts can be divided into two steps: (i) identifying the accounts and collecting the messages, and (ii) semi-automatic alignment of translated tweets.
Di erent to
We identi ed two kinds of \authors" following this strategy: (i) authors that use a single account to post messages in di erent languages, and (ii) authors that have parallel accounts to post in di erent languages using separate accounts. The initial collection of tweets amounted to 23 Twitter accounts (from 16 authors) for the eu-es pair and 19 accounts (from 14 authors) for the ca-es pair. In all, 75,000 tweets were collected for eu-es and 51,000 tweets for the ca-es language pair. The collection includes tweets posted between November 2013 and March 2015.
The initial corpus was then split into two datasets: one development-set composed of 4,000 parallel tweets for each language pair and one test-set composed of 2,000 parallel tweets for each language pair.
Author distribution in the development set was limited to account with most tweets (2 for ca-es and 4 for eu-es). Test-sets also contain tweets from the authors in the development set, but tweets from new "unseen" authors are also introduced. This way we have the possibility to evaluate systems both on "in-domain" and "out-of domain" scenarios.
As we said before, one of the limitations of our strategy is that it is only applicable to certain language pairs. The linguistic realities of Basque and Catalan (both are considered to be co-o cial together with Spanish in certain regions that support bilingualism) make the application of such methods viable for our purposes. Unfortunately, it was not the case for pt-es and gl-es pairs. It is understandable that few or no users have the need to tweet both in Spanish and Portuguese, which have little or no geographical overlap; it was however a surprise not to nd any such example for Galician and Spanish, which has the same status as Catalan and Basque of being co-o cial. In consequence, we only could provide development corpora for the eu-es and ca-es language pairs. For the Galician{Spanish and Portuguese{Spanish language pairs, test sets were manually generated through crowdsourcing. Speci cally, we used the CrowdFlower platform to translate tweets into the other language. Section 2.2 further discusses this process.
The large volume of tweets collected in the previous step needs to be properly aligned in order to create the parallel corpus. Aligning tweets of an author within and across accounts requires both to nd matching translations as well as to occasionally get rid of tweets that have no translations. We perform this process semi-automatically, rst by automatically aligning tweets that are likely to be each other's translation, and then by manually checking the accuracy of those alignments.
Before we can even align tweets with their
likely translations, we needed to identify the
language each tweet is written in through
language identi cation
Once we have an author's tweets separated by language, and hence with source language tweets and target language tweets separated, we need to align them with likely translations for each tweet. For the automated process, we de ned a set of heuristics and statistics that would help us nd matches quite accurately. Speci cally, we looked at the following three characteristics to nd likely matches:
Publication date. Translations must be published within a certain period range to be apped as possible translations of each other. The di erence between source and target timestamps must not exceed a certain threshold. The value of the treshold was set overall to 10 hours, although for a few accounts the publication date di erence was restricted to 1 hour after empirically detecting too much noise with the more relaxed standard threshold.
Overlap of hashtag and user mentions in source and target tweets. It is very rare to change the user (@) mentions across language, only in a few cases was observed that phenomenon (e.g., using @FCBarcelona ca in a tweet in catalan and using @FCBarcelona es in a Spanish written tweet) are usually maintained across languages. Hashtags are translated, often, depending on the popularity of a given hashtag in the target audience. A minimum number of user name and hashtags were required to overlap between source and target parallel tweet candidates. The overlap is computed as the division between the number of entities in the intersection of both tweets and the entities in the 2http://www.let.rug.nl/vannoord/TextCat/ union. The threshold is empirically set to 0.76.
Longest Common Subsequence
ratio (LCSR)
As for the performance of the heuristics, publication date closeness is e ective for ltering out wrong candidates, but it is not enough to nd the correct parallel tweet, so it is applied rst of the three. User and hashtag overlapping ration proved to be very successful, up to the point that the contribution of LCSR was minimal.
The output of this alignment is then corrected through manual checks by native speakers of their respective languages. The manual inspection showed a low error rate in the automatic alignment, especially for ca-es. For this language pair we found a 2% error rate, evaluated over a sample of 400 tweets on the development set. For eu-es the percentage increased to 15%, also evaluated over a sample of 400 tweets. Error rate over the collections manually reviewed to create the test-sets was 7% for the ca-es language pair (12500 tweets) and was 32% for the eu-es language pair (15045).
As we did not nd bilingual
Portuguese{Spanish or Galician{Spanish
Twitter accounts, we used the CrowdFlower
platform3 to build the test data for this
language pair. CrowdFlower provides
a cheap and fast method for collecting
annotations from a broad base of paid
non-expert contributors over the Web.
It works in a similar way to Amazon's
Mechanical Turk
In the task we de ned, the contributors had to translate manually, from Spanish 3http://www.crowdflower.com/ to Portuguese and Galician, a dataset with 2; 552 Spanish tweets, taken from both our ca-es and eu-es parallel corpora, and divided in working tasks of 10 tweets each.
Instructions were provided to workers in order to make sure that the translations were consistent. For instance, contributors were asked not to translate user mentions (keywords with a leading @) and URLs, while hashtags should only be translated if the contributor considered that it would be natural to use the Portuguese/Galician hashtag.
The crowdsourcing platform allows to con gure the jobs using a number of options. We used some of them with the aim of obtaining translations of a reasonable quality:
Geography. One can select a set of countries from which workers are allowed to work on the job. We limited the countries to Spain for Galician and to Portugal and Brazil for Portuguese.4 Performance level. Contributors of the platform fall into three levels, according to their performance. Our jobs were limited to contributors in level 3 (the top level), de ned by Crowd ower as \the highest performance contributors who account for 7% of monthly judgments and maintain the highest level of accuracy across an even larger spectrum of CrowdFlower jobs [compared to contributors in levels 1 and 2]". In the case of Galician, we had to change this setting to level 1 as the tasks were getting completed too slowly.
restrict the contributors that can work in the job by their language skills. For translations into Portuguese, we restricted the contributors to those who are veri ed speakers of Portuguese. Galician is not in the list of languages provided in CrowdFlower, so this job was not con gured in this case.
4Initially, the task to translate into Portuguese was only opened for users from Portugal as the focus is on Iberian Portuguese, but after we realized we were having no contributions, we broadened the geographical scope to Brazil as well, which helped to obtain contributions more swiftly.
Speed trap. If set, contributors are automatically removed from the job if they take less than a speci ed amount of time to complete a task. Our jobs contained tasks of 10 translations each and the time trap was set to 150 seconds. Hence if a worker toke less than 15 seconds to translate per tweet he/she would be automatically removed from the job.
The task of translating into Portugese was completed by 40 di erent contributors, all of them from Brazil. The contributors were inquired about the quality of the task; they were asked to rank out of 5 the clarity of the instructions (average 4.12), ease of the job (3.78), pay (4.17) and overall satisfaction (4.05). The task to translate into Galician was carried out by 10 contributors. They ranked the task as follows: clarity of the instructions (4.90), ease of the job (3.59), pay (3.82) and overall satisfaction (4.00).
As a nal result, we obtained a parallel corpus with 2; 500 pt-es and 777 gl-es tweets which were split into two test datasets with 1; 225 entries for each translation direction for pt-es and 388 for gl-es. To verify the quality of the translations, samples of 30 tweets were evaluated both for Portuguese and for Galician. In both cases they were considered acceptable by the Portuguese and Galician authors of the current paper, even if some errors were detected. In the case of Galician, we found some mistakes derived the new spelling rules imposed since 2003. In the case of Portuguese, six errors (most of them lexical problems) were found from the 30 tweets evaluated.
Before delivering the data sets to the participants the test was pre-processed. The development corpus includes the original tweets, neither @user nor URLs were normalized, but they are in the test corpus where @user and URLs are standardized to IDIDID and URLURLURL, respectively.
Datasets are distributed in tab separated format (tsv) les. For each language pair two les are provided, one for every translation direction. For the language pairs where the parallel corpus was gathered exclusively from Twitter |this includes ca, es, and eu| the les contains the tweetID, userID, date and the text of each tweet. For the language Dataset eu-esdev ca-esdev eu-estest es-eutest ca-estest es-catest gl-estest es-gltest pt-estest es-pttest
Tweets Authors Tokens URL @user pairs where the corpus was obtained via crowdsourcing |gl and pt |, the le contains a segmentID and the text of the tweet. 3
The test sets just described were delivered to the participants which had to return the translations with the following tab separated format: tweet Id <tab> source language text <tab> translation nn
The translated test would then be extracted, cut to a maximum length of 140 characters, and evaluated by automatic means.
The performance of the systems is assessed with lexical and syntactic automatic evaluation measures compared against a single reference. Lexical metrics which are mostly based on n-gram matching are available for all the language pairs under study. However, syntactic metrics are only available for Spanish and some of them for Catalan.
In order to study the quality of the translations at di erent levels we use a wide set of metrics as de ned as follows:
PER
Ol
ULC
SP-Op, SP-Oc, SP-pNIST
All measures have been calculated
with the Asiya toolkit7 for MT
evaluation
Participants were required to register8 in order to obtain the development and test data-sets. Each participant had only 72 5Family of metrics only available for Catalan and Spanish.
6Family of metrics only available for Spanish. 7http://nlp.cs.upc.edu/asiya/ 8http://komunitatea.elhuyar.eus/tweetmt/ participation/ hours to work on the test set and to send the results. 4.1
Out of the 5 initially registered participants,
only three teams ended up submitting their
results: DCU (Dublin City University)
for 3 tracks (ca-es, eu-es, pt-es)
The related shared tasks that we
organized in recent years (i.e., TweetNorm
The main characteristics of the systems submitted are compiled in Table 2 and can be summarized as follows: DCU This team submitted systems for three language pairs in both directions: Spanish from/to Catalan, Basque and Portuguese. They used a range of techniques including state-of-the-art SMT, morph segmentation (only for Basque as a morphologically rich language), data selection as a means of domain adaptation, available open-source rule-based systems and,
nally, system combination to combine the strengths of the di erent systems that were built. DCU gathered vast amounts of tweets (from 11M for Basque to 130M for Spanish) to perform monolingual domain adaptation and complemented this with publicly available general-domain monolingual and parallel corpora. The rst (DCU1), second (DCU2) and third (DCU3) systems submitted for each language
Distinctive features DCU1 DCU2 DCU3 EHU1 EHU2 UPC1 UPC2
System combination
or SMT SMT RBMT SMT SMT
Moses and Apertium (ES$CA), Moses, cdec and Apertium (ES!EU), cdec (EU!ES), Moses (ES$PT).
Moses (ES!CA), Moses, cdec and Apertium (CA!ES, EU!ES), Moses, cdec, ParFDA, Matxin and Morph (ES!EU), Moses and cdec (ES$PT).
Moses, cdec and Apertium (ES!CA, ES$PT), Moses, ParFDA and Apertium (CA!ES), Moses, cdec, Matxin and Morph (ES!EU), Moses, cdec, Apertium and Morph (EU!ES).
Speci c language model and pre- and post-processing for tweets Adaptation to Tweets (mainly hashtags) Moses system
Document-level system (Docent), semantic models direction were the individual systems or combinations that obtained the best, second best and third best result, respectively, on the development set.
EHU This team submitted systems for the Basque{Spanish pair. They have adapted previous MT engines for the es{eu and eu-es directions. For the translation into Basque RBMT and SMT were adapted whereas for the translation from Basque only a SMT based system was used. The main work was pre- and post-processing for adaptation to tweets and collecting new resources for training and tuning the systems. For RBMT, a small dictionary of hashtags was obtained from the development-set. For SMT, language models were improved using monolingual corpora from previous shared tasks and a new corpora of tweets in Basque.
UPC The team submitted two systems for the Catalan{Spanish language pair. The
rst one (UPC1) is a standard SMT
system built with Moses
The authors report some problems with this con guration and include both the o cial and new results in their paper.
Here only the o cial results are shown. 4.2
Participants had a 72-hour window to work with the test set and submit up to three results per track. This section is a recap of the results of all the tracks and systems.
Table 3 and Table 4 show the results for the participants in the ca-es track. In Table 3, the lexical measures introduced in the previous section are shown and in Table 4 the syntactic ones. Five systems from two teams have been evaluated. DCU3 system was the best for the ca-es direction, a system combining two kinds of SMT engines plus a RBMT one. For the es-ca direction, the two simplest pure phrase-based SMT systems, UPC1 and DCU2, obtained the highest scores. The two teams used very similar corpora in their experiments, so the techniques they used make the di erence in this case.
Tables 5, 6, 7 and 8 show the results for the participants in the eu-es and pt-es tracks. For the eu-es track four or ve (depending on the direction) systems were presented by two teams. In general, the best translator for this language pair is the statistical system EHU1 in both directions. When translating from Spanish into Basque, however, DCU2 with the combination of 5 di erent systems gets very similar scores. Di erences in this case are in general not statistically signi cant.
Finally, in the pt-es track DCU submitted the results of three systems. DCU3 was the best in the pt-es direction. As in the ca-es track, their best system is again a combination of two kinds of SMT engines and a RBMT one. On the opposite direction the best system, DCU2, does not include translation options from the RBMT, probably re ecting a lower quality for this engine on tweets. Notice that their best system in development does not correspond to the best system in test.
Based on the previous gures as well as
on the conclusions drawn by the authors
of the papers submitted to the shared
task
The results are in general very good when compared to previous results for
Expanding the context by using a user's
tweets within the same day can be of use
to boost the performance of the machine
translation system
The shared task organized at TweetMT has enabled us to come up with a benchmark parallel corpus of tweets for translation applied to four language pairs: ca-es, eu-es, gl-es and pt-es. This has allowed participants to tune and compare their MT systems. The corpus developed for the shared task can in turn be downloaded from the workshop's website10, which we expect that will enable further research in the eld.
The participants of the shared task have applied and studied the suitability of state-of-the-art MT techniques. These techniques have been adapted to the speci c features of tweets, including conventions such as hashtags, and user mentions, as well as considering the brevity of the texts. The study of the results achieved by the submitted systems enables us to draw conclusions to better inform future research in the eld.
The results achieved by the participants of the shared task are surprisingly high, especially considering that we are dealing with tweets, whose brevity and speci c characteristics make them more challenging to translate. Still, it is worthwhile noting that the tweets considered in this shared task can largely be deemed formal. Therefore, we could say that the task (translating formal tweets generated from multilingual Tweet IDs) was easier than usual tasks in MT.
A more thorough analysis of the task and performance of the participating systems will follow in an extended version of this paper, including the conclusions after the discussion in the workshop.
We should also emphasize that these results cannot be generalized to broader tasks of translating tweets. However, the fact that 10http://komunitatea.elhuyar.eus/tweetmt/ resources/ formal tweets can be accurately translated encourages its use by community managers who tweet in di erent languages, by making their work easier. One of our main objectives for future work is to further generalize the machine translation task by including all kinds of tweets, to assess the ability of MT systems to translate informal tweets too. A second version of the TweetMT dataset would include:
Tweets in English, so that we can attract a larger number of participants, comparing a larger number of MT systems.
A more generalistic Twitter dataset including informal tweets as well, in order to test the result of MT to a large and diverse corpus like Twitter.
One of the main remaining challenges is
the need to come up with a methodology
to put together a gold standard corpus that
encompasses the di erent types of tweets
that one can nd on Twitter, including
more informal tweets than those we have
considered here. To tackle such a process,
we would rst need to solve some open
questions such as whether or not and how
to translate words that are not written in
its normalized form, as well as how to
deal with multilingualism in a single tweet.
We are con dent that the discussion among
attendees of the workshop, the presentations
of accepted papers, as well as the invited talk
This work has been supported by the following projects: Abu-Matran (FP7-PEOPLE-2012-IAPP) PHEME (FP7, grant No. 611233), Tacardi (Spanish MICINN TIN2012-38523-C02-01), QTLeap (FP7, grant No. 610516), HPCPLN (Galician Gov, EM13/041), Celtic (Innterconecta program, 2012-CE138). Normalizacion lexica de tuits en espan~ol.
In Tweet-Norm@SEPLN, pages 1{9.