This article presents a summary of the TweetLID shared task and workshop held at SEPLN 2014. It brie y summarizes the data collection and annotation process, the development and evaluation of the shared task, as well as the results achieved by the participants.
Recent research shows that while Twitter's
predominant language was English in the
early days, the global growth and adoption
of the social media platform in recent years
has increased the diversity in the use of
languages
Twitter itself does provide a language id
along with each tweet's metadata, but as
we show in this article it leaves much to
be desired in terms of accuracy. Besides,
it is intended to detect major languages,
and does not identify other languages with
lesser presence on the platform such as
Catalan, Basque or Galician, which account
for millions of native speakers within the
Iberian Peninsula. Following up on a
recent shared task on normalization of
tweets
This shared task has enabled the development of an annotated tweet corpus that set out participants to deal with three novel aspects with respect to previous research in tweet language identi cation: (i) some of the languages considered in the task belong to the same language family, which makes the distinction of these languages an extra challenge, (ii) tweets can be occasionally multilingual, with parts of it written in di erent languages {manually annotated as lang1+lang2{, and (iii) tweets can be ambiguous occasionally given their brevity, i.e., it is not possible to determine which of two (or more) languages a tweet is written in, and therefore any of them should be deemed correct. This article serves as an introductory overview of the shared task, describing the process of generation of the corpus, providing a brief description of the systems developed by the participants, as well as the performance of these systems evaluated in comparison with human assessments. 2
Language identi cation consists in
determining the language in which a
text is written. It has usually been tackled
as a classi cation problem in previous
research, and the best known approaches
make use of n-grams to learn the model for
each of the languages, as well as to represent
each of the documents to be categorized into
one of the languages
Language identi cation has progressed
signi cantly in recent years. The task has
been considered solved for certain situations
This task includes the ve top languages of the Iberian Peninsula, which are spoken in di erent regions, and four of them {Spanish, Portuguese, Catalan, and Galician{ are romance languages with certain similarities among them, which makes the task more challenging. The fth language {Basque{, and English, belong to di erent language families, and therefore are rather di erent from the rest. Still, their cultural proximity, and the fact that many users in the area are bilingual, entails that they often mix words and spellings across languages. For instance, a Basque native might naturally write something like "nos vemos, agur!" (see you later, bye!), when "nos vemos" is in Spanish, while "agur" is Basque to say good bye; similarly, a Catalan speaker might often misspell the Spanish word "prueba" (test) as "prueva", given that the Catalan translation of the word ("prova") is written with v. These characteristics are common in bilingual areas, and have been considered in the de nition of this task in order to carefully develop the annotation guidelines and to pursue the nal annotation of the corpora. 3
The TweetLID shared task consists in identifying the language or languages in which tweets are written. It is focused on the most widely used languages of the Iberian Peninsula, which provides an ideal context where news and events are likely to be shared and discussed in multiple languages.
To the end of setting up a common evaluation framework to enable comparison of di erent language identi cation systems, we have put together an annotated corpus of nearly 35,000 tweets and de ned a methodology to evaluate the multi-label output of the language identi cation systems. Splitting the corpus into a training set with 15k tweets, and a test set with 20k tweets, the participants had a month to develop their language identi cation systems making use of the training set. They then had 72 hours to work on the test set and submit their results.
Besides the challenge of dealing with the short and often informal texts found in tweets, the task has considered that a tweet is not necessarily written in a single language. This is especially true in bilingual regions, where speakers that feel equally comfortable with either of their two native languages tend to switch between them and mix them in a sentence quite frequently. Hence, the task has also considered a number of cases where the response is not basically one of the languages in the list: (i) a tweet can combine two {or occasionally three{ languages in a tweet, e.g., when a tweet has parts in Catalan and Spanish, (ii) given the similarity and cultural proximity between some of the languages, it is not possible to determine which of two {or more{ languages a tweet is written in, e.g., some tweets might be written equally in Catalan or Spanish, (iii) despite the geographical restriction of the tweets in the task, it is also likely that tweets in other languages occur, such as French, and (iv) it is not possible to determine which of the 6 languages considered in the task a tweet is written in, e.g., when a tweet only mentions entities, smileys, or onomatopoeias. 4
To collect an unrestricted set of tweets, but rather focused on the set of languages within the scope of TweetLID, we relied on geolocation to retrieve tweets posted from areas of interest. We used Twitter's streaming API's statuses/filter endpoint to collect geolocated tweets posted within the Iberian Peninsula from March 1 to 31, 2014. While this stream is limited to tweets explicitly providing geolocation metadata, it allows to track a diverse set of tweets that is not restricted to a speci c set of users or domain. Having collected these tweets, we used Nominatim1 to obtain speci c location information for each tweet. Given the coordinates of a tweet as input, Nominatim queries OpenStreetMap for the speci c address associated with those coordinates, i.e., region, city, and street (if available) from which the tweet has been sent. This led to the collection of 9.7 million tweets with location details associated. From this set of tweets, we sampled tweets from
Basque Country, where Basque and Spanish are spoken. Tweets from the 1http://wiki.openstreetmap.org/wiki/Nominatim province of Gipuzkoa were chosen here to represent the Basque Country.
Catalonia, where Catalan and Spanish are spoken. Tweets from the province of Girona were chosen to represent Catalonia.
Galicia, where Galician and Spanish are spoken. Tweets from the province of Lugo were chosen.
One province was picked from each of the regions to avoid cases such as that of the province of Barcelona in Catalonia, which is much more diverse in terms of languages due to tourism. These three bilingual regions enabled us to sample tweets in Basque, Catalan, Galician, and Spanish, and we could sample Portuguese tweets from Portugal. English is the sixth language in the corpus, which can be found all across the aforementioned regions. For the nal corpus to be manually annotated, we picked 10k tweets from each of the bilingual regions, and 5k from Portugal. The tweets picked here had to contain at least one word (i.e., string fully made of a-z characters), so that there is some text, and tweets with e.g. only a link are not considered. The next section describes the manual annotation performed on this corpus with 35k tweets. 5
The collection of 35k tweets resulting from the aforementioned process was then manually annotated. Each of the tweets was associated with its corresponding language code in the manual annotation process. The manual annotation was conducted by annotators who were native or pro cient speakers in at least three languages considered in the task. This enabled us to distribute the tweets from each of the four regions to di erent annotators, so that each annotator was a native or pro cient speaker of the languages spoken in the region in question.
The annotators were instructed to assign codes to tweets according to the language in which they were written. We asked them to ignore #hashtags and @user mentions, as well as references to named entities in another language. For instance, in the tweet Acabo de ver el ultimo cap tulo de la temporada de 'the walking dead', muy bueno! (Spanish: I just saw the season nale of 'the walking dead', it's amazing!), only Spanish should be annotated.
They had to assign codes to the tweets as follows: eu for Basque, ca for Catalan, gl for Galician, es for Spanish, pt for Portuguese, and en for English. When a di erent language was found in a tweet {e.g., French or German{, they had to annotate it as other. Additionally, when the text of a tweet included words that are widely used in any of the languages in the task {e.g., onomatopoeias such as 'jajaja' or 'hahaha', or internationalized words such as 'ok'{, which makes it impossible to determine the language being used in that speci c case, they were asked to annotate it as und (eterminable).
In the above situations, the annotators had to mark a tweet as either being written in one of the 6 languages, other or und. However, two more cases were identi ed and included in the annotation guidelines: multilingual tweets, and ambiguous tweets.
Multilingual tweets contain parts of a tweet in di erent languages, where the annotators were instructed to annotate all of the languages being used. For instance, Qeeeee matadaaa (Spanish: that was exhausting) da Biyar laneaaaa... (Basque: and gotta go to work tomorrow) should be annotated as es+eu, and Acho que vi a Ramona hoje (Portuguese: man, I've seen Ramona today) but im not sure (English) should be annotated as pt+en. Occasionally, three languages were also found, e.g., Egun on! Buenos d as! Good morning! (Good morning in Basque, Spanish and English), annotated as eu+es+en. The annotation had to consider all the languages being used, in no speci c order, except when a single word or term was used as a constituent of a sentence in another language, e.g., es un outsider (Spanish: he is an outsider), where only one language is annotated.
Ambiguous tweets were de ned as the tweets that can be categorized into the list of languages being considered, but may have been written in at least two of them. Given the similarity and cultural proximity of some of the languages, it is likely that some short texts are written equally in some languages. For instance, Acabo de publicar una foto (I just published a photo) can be either Spanish or Catalan, and cannot be disambiguated in the absence of more context. This case had to be annotated as es/ca. 6
The annotated corpus is composed of 34,984 tweets, with manually annotated language labels following the procedure described above. Table 1 shows the distribution of the manual annotation, where it can be seen that Spanish is the predominant language, which amounts to 61.22% of the tweets. This is why we use a macroaverage approach to evaluate the systems, as we describe later, which rewards the systems that perform well for all the languages rather than just for the predominant language.
For the purposes of the shared task, the corpus was split into two random sets of tweets: a training set with 14,991 tweets, and a test set with 19,993 tweets. However, due to restrictions on the use of the Twitter API2, we distributed the corpora to the participants by including only the tweet IDs. We also provided them with a script to download the content of the tweets having the IDs, which scrapes the web page of each tweet to retrieve the content.
Once the participation period ended we checked the set of tweets in the test set that were still available at the moment. This was done speci cally on the 7th of July, with the submission deadlines closed for all the participants. This nal check found that 18,423 out of the initial 19,993 tweets, i.e., 92.1%, were available at the moment. For further details into the composition of the corpora, Table 2 shows the distribution of categories for the train and test datasets. 6.1
The fact that the corpora (as well as the reality of Twitter itself) is unbalanced, and some languages are far more popular than others is an important issue to be considered when de ning the evaluation measures. Besides, given that the language identi cation task has been de ned as a classi cation problem where tweets can be either multilingual, with more than a language per tweet, or ambiguous, where it is not possible to disambiguate among a set of target languages, the evaluation measures 2https://dev.twitter.com/terms/api-terms need to be carefully de ned to take these into account.
To deal with the imbalance, we compute the precision, recall, and F1 values for each language, and the macroaveraged measures for all languages afterwards. This is intended to provide higher scores to systems that perform well for many languages, rather than those performing very well in the most popular languages such as Spanish and Portuguese.
Given the characteristics of the task,
we rely on a concept-based evaluation
methodology for multi-label classi cation
P = R = 1 X jCj i2C T Pi + F Pi jCj i2C T Pi + F Ni
T Pi 2 T Pi F1 =
jCj i2C 2 T Pi + F Pi + F Ni Where C = fca; en; es; eu; gl; pt; amb; undg is the set of labels de ned in our classi cation task, and T P , F P and F N refer to the counts of true positive, false positive and false negative answers respectively.
To determine whether a system's output for a tweet is correct, we compare it with the manually annotated ground truth. Given that tweets are not simply multilingual, the T P , F P and F N values are computed as follows: (1) (2) (3) For monolingual tweets, the T P count is incremented by 1 if the answer is correct, and F P is incremented by 1 for the language output by the system otherwise. If a system's prediction contains more than one language, incorrect languages will be penalized, e.g., for a tweet annotated as "pt", a system that outputs "pt+en" will increment T P for "pt" but also F P for "en". F N will be incremented for the language in the ground truth if the answer does not contain the correct language. Hence, the system that outputs "eu" for a tweet that is actually "pt", will count as an additional F P for "eu", and as a F N for "pt". For multilingual tweets, we apply the same evaluation methodology as for the multilingual tweets above repeatedly for each of the languages in the ground truth, e.g., for a tweet annotated manually as "ca+es", a system that outputs just "ca" will count as T P for "ca" and as F N for "es".
For ambiguous tweets that could have been written in any of a set of languages, any of the responses in the ground truth is deemed correct, e.g., for a tweet annotated as "ca/es", either "ca" or "es" is deemed correct as a response, counting as T P of the "amb" category in either case. If, instead, the system outputs "pt", which is not among the languages listed in the ground truth of the ambiguous tweet, the evaluation counts as a F P for "pt", and as a F N for "amb".
Finally, note that we decided to merge tweets annotated as "other" or "und" for evaluation purposes. We did not di erentiate between them as those are the tweets that need to be ruled out for being out of the scope of the task. If a system determines that a tweet is "other", and the ground truth is "und", or vice versa, it is deemed correct. 7
Out of the initially registered 16 participants, 7 groups submitted their results for either one or both of the tracks. Participants had a 72 hour window to work with the test set and submit up to two results per track. Next, we rst summarize the types of approaches that the participants utilized, and further detail the technique used by each of the participants afterwards. 7.1
The participants have relied on very diverse and di erent techniques in their systems. They have employed di erent classi cation algorithms, di erent methods to learn the models for each language, as well as di erent criteria to determine the languages of a tweet. This diversity of approaches enables us to broaden the conclusions drawn from the analysis of the performance of di erent systems. One aspect that the participants agreed upon is the need to preprocess tweets by removing some tokens that do not help for the language identi cation task such as URLs and user mentions, as well as by lowercasing and reducing the repetition of characters, among others.
The participants have used di erent classi cation algorithms to develop their systems. The classi cation algorithms used by most participants include Support Vector Machines (SVM), and Naive Bayes, which have proven e ective in previous research in language identi cation for longer texts.
Not all the participants have developed multilabel techniques that can deal with multilingual tweets. Only two of them actually did, mostly by de ning a threshold that determines the languages to be picked for the output when the classi er provides a higher con dence score for them.
Table 3 summarizes the characteristics of the approaches developed by each of the participants. 7.2
Citius-imaxin
RAE
UB/UPC/URV
IIT-BHU
n-gram distances
Classi er 1) ranked n-grams 2) naive bayes support vector machines 1) linear interpolation 2) out-of-place measure n-gram distances 1) support vector machines 2) Freeling TextCat & langid.py & langdetect
Representation words & n-grams & su xes n-grams n-grams n-grams 3-grams news corpora Europarl corpus
Wikipedia
Yali Ext. Resources
Multiling.
no
yes
no
no
no
yes
no
words & 4-grams
Wikipedia
method
IIT-BHU
CERPAMID
ELiRF @ UPV
LYS @ UDC
Table 4 shows the results for the constrained track, and Table 5 shows the results for the unconstrained track. The ELiRF @ UPV group performed best for the constrained track with an F1 of 0.752, and Citius-imaxin presented the most accurate system for the unconstrained track, with a very similar F1 value, 0.753.
One of the aspects that stands out from the results of the participants is the fact that most of the systems performed better in the constrained track, and the lower performance of their unconstrained counterparts suggests that either the external resources used are not suitable for the task, or they were not properly exploited. Surprisingly, only the unconstrained version of Citius-imaxin's bayesian technique outperformed its constrained counterpart. This posits an important caveat of the presented systems, which needs to be further studied in the future.
Figure 1 summarizes in a boxplot the distribution of precision values achieved by the 21 submitted systems for the di erent categories. It can be seen that the systems performed poorly especially for Galician (gl); this can be due to its similarity to Spanish (es) and Portuguese (pt), and its small presence in the corpus. Because of this similarity, and of course the cultural proximity where users tend to mix up spellings, the system might have had a tendency to picking the most popular languages in these cases as output. The systems performed better for the rest of the languages, but still surprisingly there is a high variation of performances for Basque (eu), where we can see that some of the systems performed poorly. This is rather surprising given that Basque is very di erent from the rest of the languages, being an isolate language. It also stands out that all the systems performed very well for Spanish, being this the majority language with over 60% of the tweets in the corpora.
es eu ca und gl pt amb en
For the sake of comparison with the
performance reported in other research
works, we also show here the microaveraged
evaluation of the three best systems in
each track. Note that the micro-averaged
evaluation favors the overall performance
of the systems, regardless of their likely
poor performance for some of the languages.
Tables 6 and 7 show the microaveraged
results, with an overall boost in the results
for all the contestants. Still, the best results
obtained in this shared task are from the
99.4% accuracy score reported for formal
text, or the 92.4% accuracy score reported
for microblogs by Carter et al.
P 0.891 0.897 0.891
P 0.898 0.839 0.820
Table 8 includes two additional results as baselines that we computed using the following two solutions: (i) Twitter's metadata, which the system itself provides with each tweet, but it does not recognize Basque, Catalan, and Galician, and (ii) TextCat, a state-of-the-art n-gram-based language identi cation system developed for formal texts, which can deal with the six languages considered in the task. Note that TextCat was run after cleaning up the tweets by removing hashtags, and user mentions, as well lower-casing the text. The low performance of both solutions, with F1 values below 0.5, emphasizes the di culty of the task, as well as the need for proper alternatives for social media texts. 8
The shared task organized at TweetLID has enabled us to come up with a benchmark corpus of nearly 35,000 tweets with manual annotations of the language in which they are written, as well as to de ne an evaluation methodology that allowed participants to compare their systems. For this task, we have considered the ve top languages of the Iberian Peninsula {Spanish, Portuguese, Catalan, Basque, and Galician{ as well as English. This has allowed participants to compare their systems with four romance languages that share similarities with one another, and two more languages that are substantially di erent from the rest, i.e., English and Basque.
The participants have applied state-ofthe-art language identi cation techniques designed for other kinds of texts such as news articles, as well as adapted approaches that take into account the nature of the brevity and chatspeak found in tweets. Still, the performance of the systems posits the need of further research to come up with more accurate language identi cation systems for social media. Some of the key shortcomings that the shared task has brought to light include the need for a better choice of external resources to train the systems, the low accuracy of the systems when dealing with underrepresented languages which are very similar to others {as occurred with Galician here{, and the inability to identify multilingual tweets. Future work on tweet language identi cation should look into these issues to develop more accurate systems. A thorougher analysis of the task and performance of the participating systems will follow in an extended version of this paper.
This work has been supported by the following projects: PHEME FP7 project (grant No. 611233), QTLeap FP7 project (grant No. 610516), Spanish MICINN projects Tacardi (Grant No. TIN2012-38523-C02-01) and Skater (Grant No. TIN2012-38584-C06-01), Galician HPCPLN project (Grant No. EM13/041), Celtic (Innterconecta program, Grant No. 2012-CE138).