This paper presents the system developed by Elhuyar for the TweetNorm evaluation campaign which consists of normalizing Spanish tweets to standard language. The normalization covers only the correction of certain Out Of Vocabulary (OOV) words, previously identi ed by the organizers. The developed system follows a two step strategy. First, candidates for each OOV word are generated by means of various methods dealing with the di erent error-sources: extension of usual abbreviations, correction of colloquial forms, correction of replication of characters, normalization of interjections, and correction of spelling errors by means of editdistance metrics. Next, the correct candidates are selected using a language model trained on correct Spanish text corpora. The system obtained a 68.3% accuracy on the development set, and 63.36% on the test set, being the 4th ranked system on the evaluation campaign.
by Elhuyar for the TweetNorm task, a task
which consists of normalizing Spanish tweets.
The normalization just covers the correction
of certain OOV words. After tagging the
reference using FreeLing
The developed system follows a two step strategy. First, candidates for each problematic word are generated by means of various methods dealing with the di erent errorsources: extension of usual abbreviations, correction of colloquial forms, correction of replication of characters, normalization of interjections, and correction of orthographical errors by means of edit-distance metrics. The second step selects the correct candidate, by comparing the adequacy of each candidate against a language model trained from standard Spanish text corpora. The EFE news corpus and the Spanish Wikipedia were used for such purposes. The system obtained a 68.3% accuracy on the development set, and 63.6% on the test set, being the 4th ranked system on the evaluation campaign. 2
In the last few years many researchers have
turned their e orts to microblogging sites
such as Twitter. However, the special
characteristics of the language of Twitter require
a special treatment when analyzing the
messages. A special syntax (RT, @user, #tag,...),
emoticons, ungrammatical sentences,
vocabulary variations and other phenomena lead to
a drop in the performance of traditional NLP
tools
To solve this problem, many authors have
proposed a normalization of the text, as a
pre-process of any analysis, reporting an
improvement in the results. Han and
Baldwin
The system performs the normalization process of tweets in two steps (see Figure 1). In a rst step several methods are applied for generating candidates for the OOV words. In the next step a single candidate is selected for each OOV word by using language models.
Two data-sets were provided by the organizers of the Tweet-Norm event. One development-set Cdev composed of 500 tweets, and one test-set Ctest composed of 600 tweets which was used only for evaluation purposes. 3.1
Generation of candidates Some of these methods use reference lexicons for generating candidates. A reference lexicon of correct forms Dr was built by joining the FreeLing's dictionary forms and forms extracted from the EFE news corpus (146M words) and Spanish Wikipedia corpus (41M words), which theoretically include correctly written texts. A minimum frequency threshold was established in order to avoid possible typos, because several of them were found in both EFE and Wikipedia (e.g., 'tambien'). A disadvantage of using these corpora is that they are focused on formal registers while the register of twitter is more informal. However, it is a di cult task to compile a corpus for informal register without including many wrongly written texts. So we sacri ced register adaption in bene t of correctness.
We created a list of colloquial vocabulary (e.g., 'colegui', 'caseto', 'bastorro') by collecting words from two sources: \Diccionario de jerga y expresiones coloquiales"1 dictionary and www.diccionariojerga.com, a crowdsourcing web including colloquial vocabulary edited by users. A di erent word corresponding to the correct form was inserted if necessary, otherwise the word itself is inserted as correct form. This list Lc = f(ci; c0i)g contains 1088 entries.
The method based on this list is simple, if an OOV word ci is included in the list the corresponding correct form c0i is generated as a candidate.
A list containing the most used abbreviations (e.g.,'mn~n'!'man~ana') and contractions (forms that join more than one word, e.g.,'porfa'!'por favor') in Twitter was created. First, the most frequent OOV words of a Twitter corpus (309,276 tweets, 4M tokens) were extracted, and the top 1,500 candidates (f req(abvi) > 25) were analyzed, looking for abbreviations and contractions. Their corresponding correct forms were established by 1http://www.ual.es/EQUALARENA/Documentos/coloquio.pdf hand. As a result, 188 abbreviations were included in the list Labv = f(abvi; abvi0)g. As with the previous method, for each OOV abvi included in the list its standard form abvi0 is proposed as a correct candidate.
Regular expressions were created for matching and normalizing the most common interjections and their variations (e.g., 'jeje', 'puf '), identi ed in the development corpus Cdev.
Repeated letters are removed from an OOV word if it does not appear in the reference lexicon Dr. Then if the modi ed form appears in Dr (e.g., 'calooor'!'calor') it is included as candidate.
A list of usual proper nouns was built from the Wikipedia corpus. Words in uppercase wuc with a minimum frequency (f req(wuc) > 100) and whose frequency is higher than that of their form in lowercase ((f req(wuc) > f req(wlc))) are taken as secure proper nouns. 6,492 words were collected in this manner.
If an OOV word w appears in a list of usual proper nouns and its rst character is in lowercase then it is put in uppercase (e.g.,'betis'!'Betis').
If all characters of an OOV wuc word are in uppercase the following rules are applied: If wuc appears as it is in Dr, wuc is proposed as candidate (e.g., 'IVA'!'IVA'). If wuc is included in Dr in lowercase wlc = lc(wuc), then wlc is proposed as candidate (e.g., IMPORTANTE!'importante').
If wuc is included in Dr with the rst character in uppercase wu0c = ucf irst(wuc), then wu0c is proposed as candidate (e.g., 'MADRID'!'Madrid').
String similarity measures are useful for detecting correct forms of misspelled words. If the string similarity between an OOV word w and a correctly written word w0 exceeds a certain threshold we can take w0 as a correct candidate. We apply edit distance as follows: rst, a set of transliteration rules are applied to both words (trans(w) and trans(w0)) in order to normalize some characters (e.g., b = v, ki = qui, ke = que ...). Then, Longest Common Subsequence Ratio (LCSR) is calculated between trans(w) and trans(w0). In order to reduce the computational cost of the process, LCSR is only computed for those words in our lexicon Dr that share the rst character (except for h) with w and have a similar length ( 20%). LCSR gives a score between 0 (minimum similarity) and 1 (maximum similarity). Those forms that reach a score greater than 0.84 are taken as candidates. 3.2 Selection of correct candidates A tweet t = ff0; :::; fi; :::fng can contain more than one OOV word, and each OOV word fi can have several candidates ffi0; :::; fij ; :::; fimg after applying the abovementioned methods (see Figure 1). Thus, a disambiguation process must be applied in order to obtain a single correct candidate for each OOV word. For that aim we use language models. The system selects for each tweet, the combination of candidates that best ts the language model, that is, the combination which maximizes the log probability of the sequence of words.
SRILM toolkit
Table 1 shows the results for the experiments done on the 500 tweets of the development collection Cdev, depending on the di erent treatments and disambiguating by using an unigram language model trained on Wikipedia and EFE corpora. The baseline consists of selecting the OOV itself as correct candidate.
All the methods proposed provide an improvement over the baseline except for the UC method (see Table 1). The degree of improvement provided by each method varies depending on the frequency of the errortype treated by the method and the performance of the method itself. Thus, the string similarity based method COG provides the highest improvement (76.93% over the baseline), which means that the presence of typos is high and that the performance of the method is good. Both REP and ABV methods o er an improvement around 40% over the baseline. The treatment of interjections (INTJ) is also important, providing an improvement of 25% over the baseline. The error-types treated by the COL and UC are very scarce (14 and 5 respectively on Cdev,). In the rst case, although the methods perform well, the improvement is small. In the case of UC, most of the cases (4 out of 5) concatenate various error-types, and our system can not deal with error concatenations, leading to a performance decrease. Nevertheless, the method does provide an improvement when it is used combined with a bigram or a trigram LM, and thus, we include it in the all con guration. Error-types treated by PN are a bit more frequent (' 40 in Cdev). Although the method is quite precise (P ' 80%) it lacks coverage (R ' 60%). Among all method combinations the best accuracy was achieved when all of them were combined (ALL). So, we conclude that the LM manages properly the candidates provided by all the methods.
We performed further experiments with di erent orders of n-grams and di erent congurations of corpora, using in all cases the ALL con guration. According to the results (table 2), when larger orders of n-grams are used higher accuracies are obtained. This improvement is signi cant between 1-gram and 2-gram models. There is no improvement when using larger orders of n-grams. As for the corpora used, combining Wikipedia and EFE corpora provides the best performance. So it seems that they complement each other. Thus, evaluation over the test-set Ctext was carried out using the bigram LM trained over Acc. on the Improvement
Devel. set over Baseline Baseline 23.28 Baseline+COL 24.2 Baseline+ABV 32.16 Baseline+INTJ 29.1 Baseline+REP 34 Baseline+PN 24.81 Baseline+UC 23.12 Baseline+COG 41.19 ALL 66.16
We performed error analysis over the OOV words not treated correctly by our best system for the 500 tweets of the development collection Cdev. Following, we explain the main problems detected in our system: Concatenation of errors: Generation methods are not combined between each other because LM is not capable of properly managing the noise created (e.g., 'SOI'!'SOY'!'soy', 'cumpleee'!'cumple'!'cumplean~os'). Abbreviations and contractions: The abbreviation and contractions not included in our list are not properly normalized (e.g.,'cmun'!'comun', 'deacuerdo'!'de acuerdo'). LCSR based method is not capable of nding the correct form for the case of abbreviations either, because the distance is very high. If the threshold is decreased too much noise is created.
Lack of domain adaptation: LM is trained from corpora corresponding to news and Wikipedia domains where informal register is not included. Because of that there are some colloquial expressions (e.g., 'maricon', 'bonico', 'comidita') and proper nouns (e.g., 'Pedrete', 'Fanegol') that are not included in our reference lexicon Dr and which are not properly disambiguated.
Keyboard typos: Some errors correspond to key confusion at writing time. In some cases LCSR is not reached. (e.g., 'pa'!'la', 'tenho'!'tengo'). 5
This paper presents a system for normalizing tweets written in Spanish. The system rst generates a number of possible correction candidates for OOV words and then selects the candidate that better matches a language model trained over corpora of standard Spanish. Our system achieved the 4th rank among thirteen contestants in the tweetNorm evaluation campaign. We consider this a satisfactory performance taking into account that, aside from the best system, the next four contestants are quite close to each other. Furthermore, our error analysis has shown that we still have room for improvement.
Edit distance must be adapted to
better deal with abbreviations, contractions and
keyboard errors. An alternative to improve
that aspect could be to use a more complex
strategy based on nite state toolkits such as
Foma
On the other hand, we apply the di erent candidate generation methods in parallel, they are not combined in any way. This leads to a poor performance when an OOV has several errors concatenated. Therefore, we should explore possible method combinations, avoiding at the same time to generate too much noise, because the LMs would lose disambiguation capacity. In addition, we could experiment with larger LMs, and also LMs that are more focused on informal register.
This work has been partially founded by the Industry Department of the Basque Government under grant IE11-305 (KnowTOUR project).