This paper describes the techniques used by the system presented at the TweetLID shared task for Twitter language identi cation. The system is based on Support Vector Machines and Rational Kernels. An algorithm for multilanguage labeling is described. Its evaluation and application to Sociolinguistics is also included.
The TweetLID shared task1 consists in
identifying the language or languages in which
tweets are written, focusing on events and
news generated within the Iberian Peninsula
Kernel functions are commonly used to
extend statistical learning methods such as
SVMs to de ne non-linear decision
boundaries. The most widely used kernels are the
linear, polynomial or Gaussian ones which
are applied over vector spaces
Lang. #Ex.
ca 1,435 en 1,058 es 7,670 eu 478 gl 696 pt 1,920
For TweetLID, the problem of labeling multiple languages has been tackled with binary classi ers trained with the so-called one-versus-all technique: learning k binary classi ers, discriminating one language from the rest. Di erent variable length n-gram kernels have been used for each language. The best parameters for each kernel have been estimated from the results on the unambiguous examples in the training dataset by cross-validation. A preprocessing step is carried out with a transducer that removes URLs, hashtags (`#Buzz'), and username mentions (`@justinbieber'); converts the text to lower case; splits words and punctuation; normalizes blanks; and removes reduplicates (`hoooola' ! `hola'). Other manipulations like diacritics removal has not found to improve results. This transducer is incorporated into the rational kernel in order to be applied before the n-gram kernel. Classi ers have been implemented with the OpenKernel library2 with default parameters. Accuracy results for each language are shown in Table 1, where the improvement due to the adding of the preprocessing step can be noticed.
To assign more than one language to a tweet, the classifying function ck(x) = sgn(sk(x)), mapping the score of each example to f 1, +1g, has been replaced with the 2http://www.openkernel.org underlying scoring function sk, whose values can be interpreted as con dence scores and can be used to compare classi ers without calibration. The algorithm for assigning one or more languages to a tweet combines the output of the classi ers with heuristic criteria in the form of a decision tree as follows: For each text sample x and each language k, the scores s1(x), . . . , sk(x) are computed. There are three situations: (a) there exists only one sk(x) > 0; (b) there is more than one sk(x) > 0; and (c), there is no sk(x) > 0. In (a), language k is assigned to x. Case (b) has several subcases, depending on the number of languages with positive scores, the length in words of x and the di erence in scores, x is nally labeled either with many languages or it is assigned the `und(e ned)' label. In case (c), x is classi ed with the higher scored language, if its value is over a given empirically determined threshold, or as `other', otherwise. 3
The distribution of errors of the classi er on the test dataset is shown in Table 3. At the level of single language tweets, Galician (gl) obtains the worst results. As can be seen in Table 2, most Galician tweets are incorrectly classi ed as Portuguese (pt) or Spanish (es), but Portuguese, which is genetically most related to Galician3, does not su er from this problem. A shift in classi cation from underrepresented to overrepresented languages could be caused by the unbalanced representation of languages in the training set. Precision and recall of `other' is rather low when compared to speci c language gures.
Due to restrictions on the distribution of Twitter content, the tweets of the TweetLID corpus were provided through their IDs. Unfortunately, a number of tweets of the reference were not always available to the participants for di erent reasons, but are taken into consideration in the o cial nal evaluation, a ecting negatively recall and F-score (see Table 3). An alternative evaluation considering only the tweets in the reference available to the system is shown in Table 4. Results on both tables are similar, indicating that performance estimation is sound.
The evaluation of the multiple language labelings has led to the following section.
3Galician is genetically related to Portuguese but orthographically related to Spanish.
Lang.
ca en es eu gl pt amb und Total Lang.
ca en es eu gl pt amb und Total
Actual ca en es eu gl pt other amb und at the inter-sentential, intra-sentential and even morphological levels. The system presented in this paper could be applied to CS to unveil part of the underlying sociolinguistic structure of communities and, at the same time, when this structure is known in advance, it can also be used to evaluate the predictive power of the method used by the system. In the case of the Iberian Peninsula, it is the westernmost southern European peninsula separated from the rest of Europe at the north-east edge by the Pyrenees. In the Iberian Peninsula, the six top languages found in tweets are Basque, Catalan, Galician, Spanish, Portuguese and English. Except for English, which is a global language, and Basque, which is a language isolate, the rest of Iberian languages descend from Vulgar Latin spoken in the Peninsula.
Spain has an o cial language, Castilian (also
known as Spanish) but the central
government has transferred some of its powers to
regional governments, known as autonomous
communities, some of them having co-o cial
languages. Table 5 contains two matrices
with the number of pairs of languages
cooccurring in tweets. Table 5.a is computed
using the manually labeled examples of the
training and test datasets while Table 5.b is
computed from the predictions on the test
dataset. There are four identi able blocks in
those matrices: (1) English (en), which is a
global language, and cooccurs with the rest
of languages; (2) Portuguese (pt), which is
a national language with little contact with
Spanish; (3) Spanish (es) a national
language cooccurring with the Spain's co-o cial
languages: Catalan (ca), Galician (gl) and
Basque (eu); and (4) the block of the
coo cials, which are not seen together in tweets
because they are not languages in contact.
Language contact has been a hot issue in
linguistics since the publication of Languages in
contact
Conclusions and Future Work
Results from the Evaluation Section suggest
there is still room for potential
improvements. A more balanced representation of
languages or the introduction of a cost
matrix could improve the performance of
underrepresented languages as Galician. The
labeling of tweets in the language category
`other' could improve both precision and
recall of other languages. Finally, it is also left
as future work to combine the output of the
individual classi ers with multilabel learning
methods