Resumen: Este artículo presenta las técnicas empleadas por el sistema presentado a la tarea compartida TweetLID para la identificación de lengua en Tuiter. Se describen tanto el uso de máquinas de soporte vectorial con kernels racionales como el algoritmo para el etiquetado de varias lenguas. También se incluye una evaluación y una aplicación a la sociolingüística. Palabras clave: Tuiter, Identificación de lengua, Máquinas de soporte vectorial, Kernels racionales, Cambio de código.
The TweetLID shared task 1 consists in identifying the language or languages in which tweets are written, focusing on events and news generated within the Iberian Peninsula (San Vicente et al., 2014). However, despite language identification (LI) has reached a great success in discriminating between distant languages, fine-grained identification is still a challenge for language technologies and there remain two major bottlenecks according to Zampieri (2013): the discrimination between similar languages, varieties and dialects; and multilingualism, code-switching and moreover, noisy or non-standard features in texts. The first problem was addressed by the author with maximum entropy models at word level (Porta and Sancho, 2014) and others in the DSL shared task (Zampieri et al., 2014). Multilingualism was addressed by Lui, Jey Han Lau, and Baldwin (2014) using probabilistic mixture models and the identification of language in short texts by Vatanen, Väyrynen, and Virpioja (2010) with n-gram language models. However, in this paper, to 1 http://komunitatea.elhuyar.org/tweetlid/ address the task defined in TweetLID, related with the second of the aforementioned bottlenecks in LI, we will use n-grams of characters and support vector machines (SVMs) with rational kernels.
Kernel functions are commonly used to extend statistical learning methods such as SVMs to define non-linear decision boundaries. The most widely used kernels are the linear, polynomial or Gaussian ones which are applied over vector spaces (Cortes and Vapnik, 1995). In the case of natural language processing (NLP), it is common practice to represent text sequences into vector spaces as bags of words or n-grams of words or characters. However, it is possible to use string kernels to define other similarity measures between sequences as the number of common substrings of two sequences, allowing mismatches, gaps or wildcards, or the weights assignment to particular substrings (Lodhi et al., 2002;Leslie, Kuang, and Bennett, 2004). Rational kernels are a family of sequence kernels constructed from weighted finite state transducers covering all string kernels commonly used in machine learning applications in bioinformatics and NLP (Cortes et al., 2004) For TweetLID, the problem of labeling multiple languages has been tackled with binary classifiers trained with the so-called one-versus-all technique: learning k binary classifiers, discriminating one language from the rest. Different variable length n-gram kernels have been used for each language. The best parameters for each kernel have been estimated from the results on the unambiguous examples in the training dataset by cross-validation. A preprocessing step is carried out with a transducer that removes URLs, hashtags ('#Buzz'), and username mentions ('@justinbieber'); converts the text to lower case; splits words and punctuation; normalizes blanks; and removes reduplicates ('hoooola' → 'hola'). Other manipulations like diacritics removal has not found to improve results. This transducer is incorporated into the rational kernel in order to be applied before the n-gram kernel. Classifiers have been implemented with the OpenKernel library2 with default parameters. Accuracy results for each language are shown in Table 1, where the improvement due to the adding of the preprocessing step can be noticed.
To assign more than one language to a tweet, the classifying function c k (x) = sgn(s k (x)), mapping the score of each example to {−1, +1}, has been replaced with the underlying scoring function s k , whose values can be interpreted as confidence scores and can be used to compare classifiers without calibration. The algorithm for assigning one or more languages to a tweet combines the output of the classifiers with heuristic criteria in the form of a decision tree as follows:
For each text sample x and each language k, the scores s 1 (x), . . . , s k (x) are computed. There are three situations: (a) there exists only one s k (x) > 0; (b) there is more than one s k (x) > 0; and (c), there is no s k (x) > 0. In (a), language k is assigned to x. Case (b) has several subcases, depending on the number of languages with positive scores, the length in words of x and the difference in scores, x is finally labeled either with many languages or it is assigned the 'und(efined)' label. In case (c), x is classified with the higher scored language, if its value is over a given empirically determined threshold, or as 'other', otherwise.
The distribution of errors of the classifier on the test dataset is shown in Table 3. At the level of single language tweets, Galician (gl) obtains the worst results. As can be seen in Table 2, most Galician tweets are incorrectly classified as Portuguese (pt) or Spanish (es), but Portuguese, which is genetically most related to Galician3 , does not suffer from this problem. A shift in classification from underrepresented to overrepresented languages could be caused by the unbalanced representation of languages in the training set. Precision and recall of 'other' is rather low when compared to specific language figures.
Due to restrictions on the distribution of Twitter content, the tweets of the TweetLID corpus were provided through their IDs. Unfortunately, a number of tweets of the reference were not always available to the participants for different reasons, but are taken into consideration in the official final evaluation, affecting negatively recall and F-score (see Table 3). An alternative evaluation considering only the tweets in the reference available to the system is shown in Table 4. Results on both tables are similar, indicating that performance estimation is sound.
The evaluation of the multiple language labelings has led to the following section.
Language contact has been a hot issue in linguistics since the publication of Languages in contact (Weinreich, 1953) and represents one of the most common scenarios for the study of language variation and change, including code-switching (CS). In informal communication, CS is a pervasive phenomenon by which multilingual speakers switch back and forth between their languages. CS is present at the inter-sentential, intra-sentential and even morphological levels. The system presented in this paper could be applied to CS to unveil part of the underlying sociolinguistic structure of communities and, at the same time, when this structure is known in advance, it can also be used to evaluate the predictive power of the method used by the system. In the case of the Iberian Peninsula, it is the westernmost southern European peninsula separated from the rest of Europe at the north-east edge by the Pyrenees. In the Iberian Peninsula, the six top languages found in tweets are Basque, Catalan, Galician, Spanish, Portuguese and English. Except for English, which is a global language, and Basque, which is a language isolate, the rest of Iberian languages descend from Vulgar Latin spoken in the Peninsula.
Spain has an official language, Castilian (also known as Spanish) but the central government has transferred some of its powers to regional governments, known as autonomous communities, some of them having co-official languages. 5: Matrices with the language cooccurrences on tweets. For tweets containing more than two languages, (e.g., 'en+es+eu'), all their pairs have been computed (e.g., 'en+es', 'en+eu' and 'es+eu'). The matrix in (a) has been computed from the labeled examples in the datasets of TweetLID and the system's predictions for the test dataset in (b).
Results from the Evaluation Section suggest there is still room for potential improvements. A more balanced representation of languages or the introduction of a cost matrix could improve the performance of underrepresented languages as Galician. The labeling of tweets in the language category 'other' could improve both precision and recall of other languages. Finally, it is also left as future work to combine the output of the individual classifiers with multilabel learning methods (Zhang and Zhou, 2014), in order to improve and replace the heuristic presented in this paper for multilanguage labeling.