Twitter Language Identification using Rational Kernels and its potential application to Sociolinguistics Identificación de lengua en Tuiter con kernels racionales y su potencial aplicación a la sociolingüı́stica Jordi Porta Departamento de Tecnologı́a y Sistemas Centro de Estudios de la Real Academia Española c/ Serrano 187-189. Madrid 28002 porta@rae.es Resumen: Este artı́culo presenta las técnicas empleadas por el sistema presentado a la tarea compartida TweetLID para la identificación de lengua en Tuiter. Se describen tanto el uso de máquinas de soporte vectorial con kernels racionales como el algoritmo para el etiquetado de varias lenguas. También se incluye una evaluación y una aplicación a la sociolingüı́stica. Palabras clave: Tuiter, Identificación de lengua, Máquinas de soporte vectorial, Kernels racionales, Cambio de código. Abstract: This paper describes the techniques used by the system presented at the TweetLID shared task for Twitter language identification. The system is based on Support Vector Machines and Rational Kernels. An algorithm for multilanguage labeling is described. Its evaluation and application to Sociolinguistics is also in- cluded. Keywords: Twitter, Language Identification, Support Vector Machines, Rational Kernels, Code-switching. 1 Introduction address the task defined in TweetLID, related with the second of the aforementioned bot- The TweetLID shared task1 consists in iden- tlenecks in LI, we will use n-grams of char- tifying the language or languages in which acters and support vector machines (SVMs) tweets are written, focusing on events and with rational kernels. news generated within the Iberian Peninsula (San Vicente et al., 2014). However, de- 2 System Description spite language identification (LI) has reached a great success in discriminating between dis- Kernel functions are commonly used to ex- tant languages, fine-grained identification is tend statistical learning methods such as still a challenge for language technologies and SVMs to define non-linear decision bound- there remain two major bottlenecks accord- aries. The most widely used kernels are the ing to Zampieri (2013): the discrimination linear, polynomial or Gaussian ones which between similar languages, varieties and di- are applied over vector spaces (Cortes and alects; and multilingualism, code-switching Vapnik, 1995). In the case of natural lan- and moreover, noisy or non-standard features guage processing (NLP), it is common prac- in texts. The first problem was addressed tice to represent text sequences into vector by the author with maximum entropy models spaces as bags of words or n-grams of words at word level (Porta and Sancho, 2014) and or characters. However, it is possible to use others in the DSL shared task (Zampieri et string kernels to define other similarity mea- al., 2014). Multilingualism was addressed by sures between sequences as the number of Lui, Jey Han Lau, and Baldwin (2014) using common substrings of two sequences, allow- probabilistic mixture models and the identifi- ing mismatches, gaps or wildcards, or the cation of language in short texts by Vatanen, weights assignment to particular substrings Väyrynen, and Virpioja (2010) with n-gram (Lodhi et al., 2002; Leslie, Kuang, and Ben- language models. However, in this paper, to nett, 2004). Rational kernels are a family of sequence kernels constructed from weighted 1 http://komunitatea.elhuyar.org/tweetlid/ finite state transducers covering all string kernels commonly used in machine learn- underlying scoring function sk , whose values ing applications in bioinformatics and NLP can be interpreted as confidence scores and (Cortes et al., 2004). can be used to compare classifiers without calibration. The algorithm for assigning one Lang. #Ex. Len. %Acc1 %Acc2 or more languages to a tweet combines the ca 1,435 2–5 96.44 98.22 output of the classifiers with heuristic crite- en 1,058 2–5 97.73 98.26 ria in the form of a decision tree as follows: es 7,670 1–5 88.47 93.38 For each text sample x and each language eu 478 1–5 98.13 99.37 k, the scores s1 (x), . . . , sk (x) are computed. gl 696 3–4 95.24 97.02 There are three situations: (a) there exists pt 1,920 1–5 95.62 97.66 only one sk (x) > 0; (b) there is more than one sk (x) > 0; and (c), there is no sk (x) > 0. In Table 1: Minimum and maximum lengths (a), language k is assigned to x. Case (b) has of the n-grams used for each language clas- several subcases, depending on the number sifier and estimated accuracies using cross- of languages with positive scores, the length validation on the unambiguous examples in in words of x and the difference in scores, x the training dataset applying different pre- is finally labeled either with many languages processing. In %Acc1 only hashtags, user or it is assigned the ‘und(efined)’ label. In mentions and URLs are removed, and tokens case (c), x is classified with the higher scored and punctuation are split. Additionally, in language, if its value is over a given empiri- %Acc2, text is lowercased and reduplicates cally determined threshold, or as ‘other’, oth- are removed. erwise. For TweetLID, the problem of labeling 3 Evaluation multiple languages has been tackled with The distribution of errors of the classifier on binary classifiers trained with the so-called the test dataset is shown in Table 3. At the one-versus-all technique: learning k binary level of single language tweets, Galician (gl) classifiers, discriminating one language from obtains the worst results. As can be seen in the rest. Different variable length n-gram Table 2, most Galician tweets are incorrectly kernels have been used for each language. classified as Portuguese (pt) or Spanish (es), The best parameters for each kernel have but Portuguese, which is genetically most re- been estimated from the results on the un- lated to Galician3 , does not suffer from this ambiguous examples in the training dataset problem. A shift in classification from un- by cross-validation. A preprocessing step is derrepresented to overrepresented languages carried out with a transducer that removes could be caused by the unbalanced represen- URLs, hashtags (‘#Buzz’), and username tation of languages in the training set. Pre- mentions (‘@justinbieber’); converts the text cision and recall of ‘other’ is rather low when to lower case; splits words and punctuation; compared to specific language figures. normalizes blanks; and removes reduplicates Due to restrictions on the distribution of (‘hoooola’ → ‘hola’). Other manipulations Twitter content, the tweets of the TweetLID like diacritics removal has not found to im- corpus were provided through their IDs. Un- prove results. This transducer is incorpo- fortunately, a number of tweets of the ref- rated into the rational kernel in order to be erence were not always available to the par- applied before the n-gram kernel. Classifiers ticipants for different reasons, but are taken have been implemented with the OpenKer- into consideration in the official final evalu- nel library2 with default parameters. Accu- ation, affecting negatively recall and F-score racy results for each language are shown in (see Table 3). An alternative evaluation con- Table 1, where the improvement due to the sidering only the tweets in the reference avail- adding of the preprocessing step can be no- able to the system is shown in Table 4. Re- ticed. sults on both tables are similar, indicating To assign more than one language to that performance estimation is sound. a tweet, the classifying function ck (x) = The evaluation of the multiple language sgn(sk (x)), mapping the score of each exam- labelings has led to the following section. ple to {−1, +1}, has been replaced with the 3 Galician is genetically related to Portuguese but 2 http://www.openkernel.org orthographically related to Spanish. Predicted Actual ca en es eu gl pt other amb und ca 1,248 2 102 1 0 6 48 18 1 en 18 773 54 2 0 10 41 10 2 es 58 81 11,264 13 44 54 200 30 8 eu 3 1 34 298 0 1 10 9 2 gl 1 0 201 0 101 40 59 21 0 pt 3 3 112 1 1 1,736 52 19 1 other 70 30 105 6 4 15 155 0 0 amb 26 24 226 32 2 14 20 9 0 und 85 29 407 11 2 110 156 4 10 Table 2: Confusion matrix of the test dataset Lang. Prec. Rec. F-score at the inter-sentential, intra-sentential and ca 0.838 0.850 0.844 even morphological levels. The system pre- en 0.840 0.737 0.786 sented in this paper could be applied to CS to es 0.921 0.952 0.936 unveil part of the underlying sociolinguistic eu 0.905 0.746 0.818 structure of communities and, at the same gl 0.665 0.284 0.398 time, when this structure is known in ad- pt 0.912 0.898 0.905 vance, it can also be used to evaluate the amb 1.000 0.746 0.855 predictive power of the method used by the und 0.366 0.298 0.328 system. In the case of the Iberian Penin- Total 0.806 0.689 0.734 sula, it is the westernmost southern Euro- pean peninsula separated from the rest of Table 3: Results taking into account all the Europe at the north-east edge by the Pyre- 18,423 tweets in the reference. Unavailable nees. In the Iberian Peninsula, the six top tweets of the reference (72) affect both recall languages found in tweets are Basque, Cata- and F-score negatively. lan, Galician, Spanish, Portuguese and En- glish. Except for English, which is a global Lang. Prec. Rec. F-score language, and Basque, which is a language ca 0.838 0.855 0.846 isolate, the rest of Iberian languages descend en 0.840 0.741 0.787 from Vulgar Latin spoken in the Peninsula. es 0.921 0.955 0.938 Spain has an official language, Castilian (also eu 0.905 0.747 0.819 known as Spanish) but the central govern- gl 0.665 0.284 0.398 ment has transferred some of its powers to pt 0.912 0.905 0.908 regional governments, known as autonomous amb 1.000 0.749 0.856 communities, some of them having co-official und 0.366 0.298 0.328 languages. Table 5 contains two matrices Total 0.806 0.692 0.735 with the number of pairs of languages cooc- curring in tweets. Table 5.a is computed Table 4: Results taking only into account using the manually labeled examples of the submitted results in the reference (18,351 training and test datasets while Table 5.b is tweets in common). computed from the predictions on the test dataset. There are four identifiable blocks in 4 Application to Sociolinguistics those matrices: (1) English (en), which is a global language, and cooccurs with the rest Language contact has been a hot issue in lin- of languages; (2) Portuguese (pt), which is guistics since the publication of Languages in a national language with little contact with contact (Weinreich, 1953) and represents one Spanish; (3) Spanish (es) a national lan- of the most common scenarios for the study guage cooccurring with the Spain’s co-official of language variation and change, including languages: Catalan (ca), Galician (gl) and code-switching (CS). In informal communi- Basque (eu); and (4) the block of the co- cation, CS is a pervasive phenomenon by officials, which are not seen together in tweets which multilingual speakers switch back and because they are not languages in contact. forth between their languages. CS is present es pt en ca gl eu es pt en ca gl eu es 20,356 1 275 111 25 231 es 12,546 21 12 34 25 13 pt 1 4,094 34 - 5 - pt 21 2,003 3 2 3 - en 275 34 1,913 44 4 16 en 12 3 949 3 1 - ca 111 - 44 2,901 - - ca 34 2 3 1,517 - - gl 25 5 4 - 930 - gl 25 3 1 - 155 - eu 231 - 16 - - 738 eu 13 - - - - 365 (a) Labeled examples in TweetLID datasets (b) Predictions on the TweetLID test dataset Table 5: Matrices with the language cooccurrences on tweets. For tweets containing more than two languages, (e.g., ‘en+es+eu’), all their pairs have been computed (e.g., ‘en+es’, ‘en+eu’ and ‘es+eu’). The matrix in (a) has been computed from the labeled examples in the datasets of TweetLID and the system’s predictions for the test dataset in (b). 5 Conclusions and Future Work eties. In Proceedings of the 1st Workshop Results from the Evaluation Section suggest on Applying NLP Tools to Similar Lan- there is still room for potential improve- guages, Varieties and Dialects (VarDial- ments. A more balanced representation of 14), Dublin, Ireland. languages or the introduction of a cost ma- San Vicente, Iñaki, Arkaitz Zubiaga, Pablo trix could improve the performance of un- Gamallo, José Ramon Pichel, Iñaki Ale- derrepresented languages as Galician. The gria, Nora Aranberri, Aitzol Ezeiza, and labeling of tweets in the language category Vı́ctor Fresno. 2014. Overview of Tweet- ‘other’ could improve both precision and re- LID: Tweet language identification at SE- call of other languages. Finally, it is also left PLN 2014. In TweetLID @ SEPLN 2014. as future work to combine the output of the Vatanen, Tommi, Jaakko J. Väyrynen, and individual classifiers with multilabel learning Sami Virpioja. 2010. Language identifi- methods (Zhang and Zhou, 2014), in order to cation of short text segments with n-gram improve and replace the heuristic presented models. In Proceedings of the Seventh In- in this paper for multilanguage labeling. ternational Conference on Language Re- References sources and Evaluation (LREC-10). Eu- ropean Language Resources Association Cortes, Corinna and Vladimir Vapnik. 1995. (ELRA). Support-vector networks. Machine learn- ing, 20(3):273–297. Weinreich, Uriel. 1953. Languages in con- tact. Findings and Problems. Mouton, Leslie, Christina, Rui Kuang, and Kristin Hague and Paris. Bennett. 2004. Fast string kernels using inexact matching for protein sequences. Zampieri, Marcos. 2013. Using bag-of-words Journal of Machine Learning Research, to distinguish similar languages: How ef- 5:1435–1455. ficient are they? In Proceedings of the 14th IEEE International Symposium on Lodhi, Huma, Craig Saunders, John Shawe- Computational Intelligence and Informat- Taylor, Nello Cristianini, and Chris ics (CINTI-13), pages 37–41. Watkins. 2002. Text classification using Zampieri, Marcos, Liling Tan, Nikola string kernels. Journal of Machine Learn- Ljubešić, and Jörg Tiedemann. 2014. ing Research, 2:419–444, March. A report on the DSL shared task 2014. Lui, Marco, Jey Han Lau, and Timothy Bald- In Proceedings of the First Workshop win. 2014. Automatic detection and lan- on Applying NLP Tools to Similar Lan- guage identification of multilingual doc- guages, Varieties and Dialects, pages 58– uments. Transactions of the Association 67, Dublin, Ireland, August. for Computational Linguistics, 2:27–40. Zhang, M.-L. and Z.-H. Zhou. 2014. A re- Porta, Jordi and José-Luis Sancho. 2014. Us- view on multi-label learning algorithms. ing maximum entropy models to discrim- IEEE Transactions on Knowledge and inate between similar languages and vari- Data Engineering, 26(8):1819–1837.