=Paper=
{{Paper
|id=Vol-1228/paper2
|storemode=property
|title=Twitter Language Identification using Rational Kernels and its potential application to Sociolinguistics
|pdfUrl=https://ceur-ws.org/Vol-1228/tweetlid-2-porta.pdf
|volume=Vol-1228
|dblpUrl=https://dblp.org/rec/conf/sepln/Porta14
}}
==Twitter Language Identification using Rational Kernels and its potential application to Sociolinguistics==
https://ceur-ws.org/Vol-1228/tweetlid-2-porta.pdf
Twitter Language Identification using Rational Kernels
and its potential application to Sociolinguistics
Identificación de lengua en Tuiter con kernels racionales
y su potencial aplicación a la sociolingüı́stica
Jordi Porta
Departamento de Tecnologı́a y Sistemas
Centro de Estudios de la Real Academia Española
c/ Serrano 187-189. Madrid 28002
porta@rae.es
Resumen: Este artı́culo presenta las técnicas empleadas por el sistema presentado
a la tarea compartida TweetLID para la identificación de lengua en Tuiter. Se
describen tanto el uso de máquinas de soporte vectorial con kernels racionales como
el algoritmo para el etiquetado de varias lenguas. También se incluye una evaluación
y una aplicación a la sociolingüı́stica.
Palabras clave: Tuiter, Identificación de lengua, Máquinas de soporte vectorial,
Kernels racionales, Cambio de código.
Abstract: This paper describes the techniques used by the system presented at
the TweetLID shared task for Twitter language identification. The system is based
on Support Vector Machines and Rational Kernels. An algorithm for multilanguage
labeling is described. Its evaluation and application to Sociolinguistics is also in-
cluded.
Keywords: Twitter, Language Identification, Support Vector Machines, Rational
Kernels, Code-switching.
1 Introduction address the task defined in TweetLID, related
with the second of the aforementioned bot-
The TweetLID shared task1 consists in iden-
tlenecks in LI, we will use n-grams of char-
tifying the language or languages in which
acters and support vector machines (SVMs)
tweets are written, focusing on events and
with rational kernels.
news generated within the Iberian Peninsula
(San Vicente et al., 2014). However, de-
2 System Description
spite language identification (LI) has reached
a great success in discriminating between dis- Kernel functions are commonly used to ex-
tant languages, fine-grained identification is tend statistical learning methods such as
still a challenge for language technologies and SVMs to define non-linear decision bound-
there remain two major bottlenecks accord- aries. The most widely used kernels are the
ing to Zampieri (2013): the discrimination linear, polynomial or Gaussian ones which
between similar languages, varieties and di- are applied over vector spaces (Cortes and
alects; and multilingualism, code-switching Vapnik, 1995). In the case of natural lan-
and moreover, noisy or non-standard features guage processing (NLP), it is common prac-
in texts. The first problem was addressed tice to represent text sequences into vector
by the author with maximum entropy models spaces as bags of words or n-grams of words
at word level (Porta and Sancho, 2014) and or characters. However, it is possible to use
others in the DSL shared task (Zampieri et string kernels to define other similarity mea-
al., 2014). Multilingualism was addressed by sures between sequences as the number of
Lui, Jey Han Lau, and Baldwin (2014) using common substrings of two sequences, allow-
probabilistic mixture models and the identifi- ing mismatches, gaps or wildcards, or the
cation of language in short texts by Vatanen, weights assignment to particular substrings
Väyrynen, and Virpioja (2010) with n-gram (Lodhi et al., 2002; Leslie, Kuang, and Ben-
language models. However, in this paper, to nett, 2004). Rational kernels are a family of
sequence kernels constructed from weighted
1
http://komunitatea.elhuyar.org/tweetlid/ finite state transducers covering all string
kernels commonly used in machine learn- underlying scoring function sk , whose values
ing applications in bioinformatics and NLP can be interpreted as confidence scores and
(Cortes et al., 2004). can be used to compare classifiers without
calibration. The algorithm for assigning one
Lang. #Ex. Len. %Acc1 %Acc2 or more languages to a tweet combines the
ca 1,435 2–5 96.44 98.22 output of the classifiers with heuristic crite-
en 1,058 2–5 97.73 98.26 ria in the form of a decision tree as follows:
es 7,670 1–5 88.47 93.38 For each text sample x and each language
eu 478 1–5 98.13 99.37 k, the scores s1 (x), . . . , sk (x) are computed.
gl 696 3–4 95.24 97.02 There are three situations: (a) there exists
pt 1,920 1–5 95.62 97.66 only one sk (x) > 0; (b) there is more than one
sk (x) > 0; and (c), there is no sk (x) > 0. In
Table 1: Minimum and maximum lengths (a), language k is assigned to x. Case (b) has
of the n-grams used for each language clas- several subcases, depending on the number
sifier and estimated accuracies using cross- of languages with positive scores, the length
validation on the unambiguous examples in in words of x and the difference in scores, x
the training dataset applying different pre- is finally labeled either with many languages
processing. In %Acc1 only hashtags, user or it is assigned the ‘und(efined)’ label. In
mentions and URLs are removed, and tokens case (c), x is classified with the higher scored
and punctuation are split. Additionally, in language, if its value is over a given empiri-
%Acc2, text is lowercased and reduplicates cally determined threshold, or as ‘other’, oth-
are removed. erwise.
For TweetLID, the problem of labeling 3 Evaluation
multiple languages has been tackled with
The distribution of errors of the classifier on
binary classifiers trained with the so-called
the test dataset is shown in Table 3. At the
one-versus-all technique: learning k binary
level of single language tweets, Galician (gl)
classifiers, discriminating one language from
obtains the worst results. As can be seen in
the rest. Different variable length n-gram
Table 2, most Galician tweets are incorrectly
kernels have been used for each language.
classified as Portuguese (pt) or Spanish (es),
The best parameters for each kernel have
but Portuguese, which is genetically most re-
been estimated from the results on the un-
lated to Galician3 , does not suffer from this
ambiguous examples in the training dataset
problem. A shift in classification from un-
by cross-validation. A preprocessing step is
derrepresented to overrepresented languages
carried out with a transducer that removes
could be caused by the unbalanced represen-
URLs, hashtags (‘#Buzz’), and username
tation of languages in the training set. Pre-
mentions (‘@justinbieber’); converts the text
cision and recall of ‘other’ is rather low when
to lower case; splits words and punctuation;
compared to specific language figures.
normalizes blanks; and removes reduplicates
Due to restrictions on the distribution of
(‘hoooola’ → ‘hola’). Other manipulations
Twitter content, the tweets of the TweetLID
like diacritics removal has not found to im-
corpus were provided through their IDs. Un-
prove results. This transducer is incorpo-
fortunately, a number of tweets of the ref-
rated into the rational kernel in order to be
erence were not always available to the par-
applied before the n-gram kernel. Classifiers
ticipants for different reasons, but are taken
have been implemented with the OpenKer-
into consideration in the official final evalu-
nel library2 with default parameters. Accu-
ation, affecting negatively recall and F-score
racy results for each language are shown in
(see Table 3). An alternative evaluation con-
Table 1, where the improvement due to the
sidering only the tweets in the reference avail-
adding of the preprocessing step can be no-
able to the system is shown in Table 4. Re-
ticed.
sults on both tables are similar, indicating
To assign more than one language to
that performance estimation is sound.
a tweet, the classifying function ck (x) =
The evaluation of the multiple language
sgn(sk (x)), mapping the score of each exam-
labelings has led to the following section.
ple to {−1, +1}, has been replaced with the
3
Galician is genetically related to Portuguese but
2
http://www.openkernel.org orthographically related to Spanish.
Predicted
Actual ca en es eu gl pt other amb und
ca 1,248 2 102 1 0 6 48 18 1
en 18 773 54 2 0 10 41 10 2
es 58 81 11,264 13 44 54 200 30 8
eu 3 1 34 298 0 1 10 9 2
gl 1 0 201 0 101 40 59 21 0
pt 3 3 112 1 1 1,736 52 19 1
other 70 30 105 6 4 15 155 0 0
amb 26 24 226 32 2 14 20 9 0
und 85 29 407 11 2 110 156 4 10
Table 2: Confusion matrix of the test dataset
Lang. Prec. Rec. F-score at the inter-sentential, intra-sentential and
ca 0.838 0.850 0.844 even morphological levels. The system pre-
en 0.840 0.737 0.786 sented in this paper could be applied to CS to
es 0.921 0.952 0.936 unveil part of the underlying sociolinguistic
eu 0.905 0.746 0.818 structure of communities and, at the same
gl 0.665 0.284 0.398 time, when this structure is known in ad-
pt 0.912 0.898 0.905 vance, it can also be used to evaluate the
amb 1.000 0.746 0.855 predictive power of the method used by the
und 0.366 0.298 0.328 system. In the case of the Iberian Penin-
Total 0.806 0.689 0.734 sula, it is the westernmost southern Euro-
pean peninsula separated from the rest of
Table 3: Results taking into account all the Europe at the north-east edge by the Pyre-
18,423 tweets in the reference. Unavailable nees. In the Iberian Peninsula, the six top
tweets of the reference (72) affect both recall languages found in tweets are Basque, Cata-
and F-score negatively. lan, Galician, Spanish, Portuguese and En-
glish. Except for English, which is a global
Lang. Prec. Rec. F-score language, and Basque, which is a language
ca 0.838 0.855 0.846 isolate, the rest of Iberian languages descend
en 0.840 0.741 0.787 from Vulgar Latin spoken in the Peninsula.
es 0.921 0.955 0.938 Spain has an official language, Castilian (also
eu 0.905 0.747 0.819 known as Spanish) but the central govern-
gl 0.665 0.284 0.398 ment has transferred some of its powers to
pt 0.912 0.905 0.908 regional governments, known as autonomous
amb 1.000 0.749 0.856 communities, some of them having co-official
und 0.366 0.298 0.328 languages. Table 5 contains two matrices
Total 0.806 0.692 0.735 with the number of pairs of languages cooc-
curring in tweets. Table 5.a is computed
Table 4: Results taking only into account using the manually labeled examples of the
submitted results in the reference (18,351 training and test datasets while Table 5.b is
tweets in common). computed from the predictions on the test
dataset. There are four identifiable blocks in
4 Application to Sociolinguistics those matrices: (1) English (en), which is a
global language, and cooccurs with the rest
Language contact has been a hot issue in lin- of languages; (2) Portuguese (pt), which is
guistics since the publication of Languages in a national language with little contact with
contact (Weinreich, 1953) and represents one Spanish; (3) Spanish (es) a national lan-
of the most common scenarios for the study guage cooccurring with the Spain’s co-official
of language variation and change, including languages: Catalan (ca), Galician (gl) and
code-switching (CS). In informal communi- Basque (eu); and (4) the block of the co-
cation, CS is a pervasive phenomenon by officials, which are not seen together in tweets
which multilingual speakers switch back and because they are not languages in contact.
forth between their languages. CS is present
es pt en ca gl eu es pt en ca gl eu
es 20,356 1 275 111 25 231 es 12,546 21 12 34 25 13
pt 1 4,094 34 - 5 - pt 21 2,003 3 2 3 -
en 275 34 1,913 44 4 16 en 12 3 949 3 1 -
ca 111 - 44 2,901 - - ca 34 2 3 1,517 - -
gl 25 5 4 - 930 - gl 25 3 1 - 155 -
eu 231 - 16 - - 738 eu 13 - - - - 365
(a) Labeled examples in TweetLID datasets (b) Predictions on the TweetLID test dataset
Table 5: Matrices with the language cooccurrences on tweets. For tweets containing more than
two languages, (e.g., ‘en+es+eu’), all their pairs have been computed (e.g., ‘en+es’, ‘en+eu’ and
‘es+eu’). The matrix in (a) has been computed from the labeled examples in the datasets of
TweetLID and the system’s predictions for the test dataset in (b).
5 Conclusions and Future Work eties. In Proceedings of the 1st Workshop
Results from the Evaluation Section suggest on Applying NLP Tools to Similar Lan-
there is still room for potential improve- guages, Varieties and Dialects (VarDial-
ments. A more balanced representation of 14), Dublin, Ireland.
languages or the introduction of a cost ma- San Vicente, Iñaki, Arkaitz Zubiaga, Pablo
trix could improve the performance of un- Gamallo, José Ramon Pichel, Iñaki Ale-
derrepresented languages as Galician. The gria, Nora Aranberri, Aitzol Ezeiza, and
labeling of tweets in the language category Vı́ctor Fresno. 2014. Overview of Tweet-
‘other’ could improve both precision and re- LID: Tweet language identification at SE-
call of other languages. Finally, it is also left PLN 2014. In TweetLID @ SEPLN 2014.
as future work to combine the output of the
Vatanen, Tommi, Jaakko J. Väyrynen, and
individual classifiers with multilabel learning
Sami Virpioja. 2010. Language identifi-
methods (Zhang and Zhou, 2014), in order to
cation of short text segments with n-gram
improve and replace the heuristic presented
models. In Proceedings of the Seventh In-
in this paper for multilanguage labeling.
ternational Conference on Language Re-
References sources and Evaluation (LREC-10). Eu-
ropean Language Resources Association
Cortes, Corinna and Vladimir Vapnik. 1995. (ELRA).
Support-vector networks. Machine learn-
ing, 20(3):273–297. Weinreich, Uriel. 1953. Languages in con-
tact. Findings and Problems. Mouton,
Leslie, Christina, Rui Kuang, and Kristin Hague and Paris.
Bennett. 2004. Fast string kernels using
inexact matching for protein sequences. Zampieri, Marcos. 2013. Using bag-of-words
Journal of Machine Learning Research, to distinguish similar languages: How ef-
5:1435–1455. ficient are they? In Proceedings of the
14th IEEE International Symposium on
Lodhi, Huma, Craig Saunders, John Shawe- Computational Intelligence and Informat-
Taylor, Nello Cristianini, and Chris ics (CINTI-13), pages 37–41.
Watkins. 2002. Text classification using
Zampieri, Marcos, Liling Tan, Nikola
string kernels. Journal of Machine Learn-
Ljubešić, and Jörg Tiedemann. 2014.
ing Research, 2:419–444, March.
A report on the DSL shared task 2014.
Lui, Marco, Jey Han Lau, and Timothy Bald- In Proceedings of the First Workshop
win. 2014. Automatic detection and lan- on Applying NLP Tools to Similar Lan-
guage identification of multilingual doc- guages, Varieties and Dialects, pages 58–
uments. Transactions of the Association 67, Dublin, Ireland, August.
for Computational Linguistics, 2:27–40.
Zhang, M.-L. and Z.-H. Zhou. 2014. A re-
Porta, Jordi and José-Luis Sancho. 2014. Us- view on multi-label learning algorithms.
ing maximum entropy models to discrim- IEEE Transactions on Knowledge and
inate between similar languages and vari- Data Engineering, 26(8):1819–1837.