This article describes two systems participating to the TweetLID-2014 competition focused on language detection in tweets. The systems are based on two different strategies: ranked dictionaries and Naive Bayes classifiers. The results show that ranking dictionaries performs better with small training corpora whose language distribution is similar to that of the test dataset, while a Naive Bayes algorithm improves the scores with large models even if the data are unbalanced with regard to the test dataset. The experiments also showed that the models based on word unigrams outperform the use of n-grams of characters. In the final evaluation the Naive Bayes classifier got the first position among the unconstrained systems (trained with external sources) participating in the competition.
TweetLID Competition
In related work, two types of models have
been used for language detection: those made
of n-grams of characters
The objective of the article is to compare two methods for language detection in tweets. On the one hand, we describe a ranking approach based on small dictionaries built according to the Zipf’s law, i.e. the frequency of any word is inversely proportional to its rank in the frequency table and, on the other hand, we also describe a Naive Bayes system which uses either n-grams of characters or word unigrams.
2.1
Our system, called Quelingua1, was implemented using a dictionary-based method and a ranking algorithm. It is based on the observation that for each language, there is a set of words that make up a large portion of any text and their presence is to be expected as word distribution follows Zipf’s law.
For each word w found in a corpus of a particular language, and for the N most frequent words in that corpus, we define its inverse ranking (IR) as follows:
IR(w) = N − (rank(w) − 1) (1) where rank(w) is the rank of w in the dictionary of N most frequent words. For instance, if the dictionary contains 1000 words, the IR for the most frequent word (ranking 1) is 1000. Specifying the size N of the dictionary is a critical issue of the method. The final weight of a specific language lang given a text is computed in equation 2, where K is the size of the input text:
K weight(lang, text) = X IR(wordi) i=1 (2)
This is computed for all available languages, and that with the highest weight is selected as the detected language for the input text.
In order to give more coverage to the system, we added a suffix module containing the most frequent suffixes of the target languages. For instance, “-c¸˜ao” is associated to Portuguese, “-ak” to Basque, “-cio´n” to Spanish and Galician, etc. This information can be automatically extracted or manually added to the module. The IR of any word that is not in the dictionary but has a suffix found in the suffix module is computed as the average IR, i.e.: N/2. 2.2
To compare our dictionary-based system
with a state-of-the-art approach, we
implemented a Naive Bayes (NB) classifier
1Freely available at:
http://gramatica.usc.es/ gamallo/tools/quelingua.htm
based on the system we previously
created for a sentiment analysis task, and
described in Gamallo, Garcia, and
Ferna´ndezLanza (2013). According to recent research
Our NB classifier was trained with two
different models: a model based on
character n-grams and another one based on word
unigrams. The best character n-gram model
turned out to be constituted by trigrams with
also bigrams just for prefix and suffix
positions. This is in accordance with previous
research on NB classifiers for language
detection where the best models were constituted
by small n-grams, with n < 4
To evaluate the performance of the two
systems described above, the development
corpus of tweets provided by the
organization of TweetLID2014 was divided into two
parts: 65% used for training and 35% as
test dataset. In addition, the systems were
also trained with further texts constituted by
recent news extracted from online journals
for English (11Mb), Spanish (7.3Mb),
Portuguese (6.6Mb), and Galician (4.2Mb). The
Catalan texts were taken from the Ancora
corpus
Before building the features used by the systems, the main preprocessing tasks we considered are the following: removing urls, references to usernames, hashtags, and emoticons; reduction of replicated characters for voyels (e.g. loooveeee → love) ; normalizing the text by using a small list of abbreviations (e.g. x → por ). 3.3
To evaluate the two systems, we used in
our experiments the evaluation script
provided by the TweetLID-2014 organization.
As far as the NB classifier is concerned, we
performed some experiments with both the
constrained and unconstrained training data,
as well as with both character n-grams and
word unigrams (bag of words). The best
results were achieved with unconstrained
training data and word unigrams. The
highest F1-Score reached with character n-grams
was 63, 56% using unconstrained training and
n < 4. By contrast, the best results achieved
with word unigrams was 77, 94% also using
the unconstrained training. This is in
accordance with
Concerning the dictionary-based system (Quelingua), the results obtained with the constrained training data clearly outperformed those obtained with the unconstrained version. Then, for the constrained system, we performed some experiments focused on determining the best size of the dictionary (i.e. of the language model). Figure 1 depicts the growth curve of F1-Score as a function of the size of the dictionary. It shows that the peak is achieved with a size of 1000 words. In the following experiments, Quelingua was trained with a dictionary of this size. 3.4
Table 1 shows the results obtained by our two classifiers, NB and Que(lingua), using different resources to train the model: only training tweets (constrained), only external resources (external), and both tweets and external resources (unconstrained).
The best constrained system is Quelingua
0,7
0,68
0,66
0,64
0,62
0
1000
2000
3000
4000
5000
while the best unconstrained is NB (which
also reaches the highest score overall).
Unlike Quelingua, the NB system achieves the
best results with the unconstrained model.
The behavior of NB is not only different from
Quelingua, but also from the other systems
participating in TweetLID-2014 competition,
since all systems except NB perform better
with the constrained version. We cannot
afford a full explanation for the other
systems in the competition, but the main reason
of the NB behavior with regard to
Quelingua is that NB-based systems tend to
perform better with large model sizes than
classifiers based on ranking methods
However the two systems behave in a similar way when they are observed across the different target languages. Both systems reach acceptable results (between 85 and 95% FScore) in Portuguese, English, Spanish, and Catalan, and poor results in Basque, Galician, and Und(efined).
Four runs were sent to the final TweetLID2014 evaluation: the constrained and unconstrained versions of both NB and Quelingua trained with the whole training dataset. The unconstrained version of NB achieved the highest score among all participants (75.3% F1-score). The constrained version of Quelingua achieved the fourth position out of 12 runs (72.6% F1-score) . It is worth noting that the final results obtained with the test dataset follow a similar tendency as that observed in our previous experiments (Table 1). 3.5
In terms of memory use, Quelingua loads a light dictionary of 35Kb (1000 words per language), while the NB systems requires loading a language model of 9Mb. Concerning speed, classification based on NB models is much slower than classification with the ranking method of Quelingua. More precisely, Quelingua is about 10 times faster than NB. 4
Conclusions and Future Work We compared several strategies for language detection in noisy short messages (tweets). First, we observed that models with word unigrams perform better than those based on n-grams of characters. We also observed that our Naive Bayes classifier outperforms the ranking-based method (Quelingua) if they are trained with external corpus (unconstrained models). By contrast, the ranking method performs better than NB when they use a small training set of tweets containing similar data (and same language distribution) to the test dataset (constrained model). Besides the fact of performing reasonably well with a small and balanced training corpus, another benefit of the ranking model is its small and easy to handle ranked dictionary, which can be easily corrected and updated by human experts.
In fact, in future work, we will measure
the performance effects of using a manually
corrected ranked vocabulary, since the
dictionaries used in the described experiments
were not corrected by humans. We will also
analyze the growth curve of the F1-score
obtained by the NB system over the corpus
size. Finally, it will be interesting to
compare these approaches with contextual-based
strategies such as Markov Models, which
were the best systems according to other
evaluations
Lang es pt en ca eu gl und average