This paper describes the language identification method presented in Twitter Language Identification Workshop (TweetLID-2014). The proposed method represents tweets by weighted character-level trigrams. We employed three different weighting schemes used in Text Categorization to obtain a numerical value that represents the relation between trigrams and languages. For each language, we add up the importance of each trigram. Afterward, tweet language is determined by simple majority voting. Finally, we analyze the results.
With the growing interest in social networks like Twitter and Facebook, the research community has focused on applying data mining techniques to such sources of information. One of these sources are the messages produced in the social network Twitter, known as tweets. Tweets represent a challenge to traditional Text Mining techniques mainly due to two characteristics, the length of the texts (only 140 characters allowed) and the Internet Slang present in these texts. Because of the limitations of 140 characters, people create their own informal linguistic style by shortening words and using acronyms.
Language identification (LID) is the task of identifying the language in which a text is written. This is an important pre-processing step necessary for traditional Text Mining techniques; also, Natural Language Processing tasks like machine translation, part of speech tagging and parsing are language dependent.
Some work has been done for LID in long
and well-formed texts; traditional approaches
are focused on words and n-grams (of
characters). In word-based features, we can find
short word-based and frequency-based
approaches.
The n-gram based approach uses n-grams of
different
The n-gram technique described by
We consider that for LID to be effective, the
inflected forms of a root word should being
related to the same word, and knowing that the
character-level n-grams of different
morphological variations of a word tend to
produce many of the same n-grams, we
chose to use character-level n-grams as features
in our approach. Since trigrams have proven
good results in LID
Some studies have shown that system
designed for other types of texts perform well
on tweet language identification (TLID) (Lui
and Baldwin, 2012), but some systems which
were specifically designed for the
characteristics of tweets performed
better.
There is also a body of work in TLID
employing different techniques, for example
graph representation of languages based in
trigrams
We propose a language identification system based on feature weighting schemes (FWS), commonly used in Text Categorization (TC). We obtain a numerical value that represents the relation between features, trigrams of characters in our case, and languages. This proposal can be extended to words and longer or shorter ngrams.
The remainder of the paper is structured as follows. In Section 2, we describe our tweet language identification system (CerpamidTLID2014) and the feature weighting schemes tested. In Section 3, we present experiments conducted for estimating the parameters of our system and we analyze the effect of feature weighting schemes in tweets language identification. Finally, conclusions and attractive directions for future work are exposed. 2
In this section, we describe our system and the feature weighting schemes that we used in our experiments. 2.1
Dimensionality reduction (DR) is an important
step in Text Categorization. It can be defined as
the task of reducing the dimensionality of the
traditional vector space representation for
documents; these are two main approaches to
this task
There are two distinct ways of viewing DR, depending on whether the task is performed locally (i.e., for each individual category) or globally.
We focus on local feature selection schemes, since our interest is to obtain the importance of every trigram (features) for every language (categories).
Many locally feature selection techniques
have been tried. We show in Table 1 those used
in this paper, GSS Coefficient (GSS)
FWS MI NGL GSS
Mathematical form log
( , ) ( ) ∗ ( ) √ ∗ [ ( , ) ∗ ( ̅ , ̅ ) − ( , ̅ ) ∗ ( ̅ , )] √ ( ) ∗ ( ) ∗ ( ̅ ) ∗ (̅̅ ) ( , ) ∗ ( ̅ , ̅ ) − ( , ̅ ) ∗ ( ̅ , )
In our case, in order to make the feature
weighting schemes depending of the available
amount of text of each language, and not of the
number of documents. The probabilities in
Table 1 are interpreted on an event space of
features
For each language, we keep the most important trigrams and discard the rest. Our system is a three-step procedure; first, trigrams are extracted from the tweet, then a filtering phase takes place, in this phase those tweets that do not belong to the set of languages that our system identify are labeled as other. Finally, a language is assigned for the tweet. We present these steps in Algorithm 1.
Algorithm 1. Cerpamid-TLID2014 Considert the tweet to identify the language, c the content of t and Lj the list of weighted trigrams for language j.
Step 1: Split c in trigrams a) Split c in words. b) Remove numbers, punctuation marks and make all the text lowercase. c) Add underscore in the beginning and the ending of every word. d) Obtain the list lt of trigrams that represent t.
Step 2: Filtering a) Let trigrams_c be the number of trigrams in lt and trigrams_in the number of trigrams in lt that appear in any Lj.
trigrams _ in and θ a threshold of b) Let n
trigrams _ c known trigrams in c. c) If n > θ go to step 3, else set language as other.
a) For each language Lj: i) vote(Lj ) weightti , Lj ti lt b) Label t with the most voted language.
In this section we explain the estimation of threshold θ (see section 2.1, Algorithm 1, Step 2), the experiments over the feature weighting schemes and results of our proposal in TweetLID-2014. In order to evaluate the feature weighting schemes and to estimate the threshold of filtering θ, the corpus provided by the organizers of TweetLID-2014 was divided into training, test and development sets. The training set is the 70% of the tweets not labeled as language undefined or other. The remaining 30% was divided again in 70% and 30%.This 30% is our development set, this last 70% and the tweets labeled undefined or other form the test set.
In TweetLID-2014 the organizers proposed
two modalities; constrained, where the training
only can be performed over the training set
provided at TweetLID-2014 and free training
(unconstrained) where is possible to use any
other resource. For the free training mode, we
decided increase the amount of text per
language provided at TweetLID-2014 in order
to provide our proposal with greater ability to
differentiate one language from another. For
English (161 mb), Portuguese (174mb) and
Spanish (174 mb) we used texts from the
Europarl corpus
For estimating θ, first we obtain the list of trigrams weighted from the training set, even when the weights are not used in this stage. Later we obtain a list L of the values n (see section 2.1, Algorithm 1, Step 2) for all the tweets in the development set. Then, we repeat 10000 times a sampling with replacement over L, in every one of these iterations we select the lowest value different from zero. These values are averaged and that is our threshold θ. The idea is to estimate statistically the value of n for a tweet written in one of the languages that we identify. The value obtained in our experiment was 0.9, and this was used for all runs.
In Table 2 we show the results obtained with each feature weighting scheme in our test sets, in the two modalities, constrained and unconstrained. The best FWS in both modes was MI, while for every FWS the constrained version obtained better results for LID.
In order to make a deeper analysis of our system, we show in Table 3 the precision and the numbers of assignations to every language for our best combinations of feature weighting scheme and task mode (MI in Constrained Mode).
FWS MI MI NGL NGL GSS GSS For our participation at TLID-2014 we used the full corpus provided by the organizers and, in addition, the documents extracted from Europarl and Wikipedia for the free training mode. In Table 4 and Table 5 we show our results at TweetLID-2014. As can be seen we placed 8th between 12 about runs and 5th between 7 about groups in the constrained mode (Table 4). Our results in precision at TweetLID-2014 are similar to the results in precision that we obtained with our own test set, while the F1 measure was dropped for the lows values in recall. About the unconstrained version, we placed last with our two runs; almost all team did worst in this mode. We presented a tweet language identification system based on trigrams of characters and feature weighting schemes used for Text Categorization. One of our run placed 8th between 12 in the constrained version at TLID2014 whilst in the free training version we placed last. Most of the system performed better in the constrained version. We found as the main weakness of our proposal the identification of tweets labeled other. As future work; we consider exploring other features, test others feature weighting schemes and tackle the problem of the identification of tweets labeled as other with the inclusion of lists of common terms used in tweets in the step of filtering.