This paper describes a strategy to perform lexical normalisation of outof-vocabulary (OOV) words in Spanish tweets. To correct any ill-formed OOV, the normalisation system generates in-vocabulary (IV) candidates found in several lexical resources, and selects the best one. Our method generates two types of candidates, primary and secondary IV candidates, which will be ranked in di erent ways to select the best candidate.
In this paper, we describe a strategy to perform lexical normalisation of out-ofvocabulary (OOV) words in Spanish tweets. The task can be described as follows. Given an OOV, the algorithm must decide whether the OOV is either correct or ill-formed and, for the latter case, it must propose an invocabulary (IV) word found in a lexical resource to restore the incorrect OOV.
There has been few work on lexical
normalisation in short messages. So far, the
most successful strategy to normalise
English tweets is described in
The rst strategy simply consists in
looking up a normalisation dictionary, which
contains speci c abreviations and other types
of lexical variants found in the Twitter
language. Each lexical variant is associated
to its standard form, for instance gl !
girlf riend. The dictionary lookup method
achieves very high precision, but with low
recall. As recall relies on the size of the
dictionary,
The second strategy is applied when the OOV is a lexical variant that has not been found in the normalisation dictionary. It consists of the following two tasks:
Generation of IV candidates (standard forms) for each particular OOV (lexical variant).
Candidate selection of the best IV candidate.
The objective of the rst task is to build, for each OOV, a list of standard forms which were derived from the OOV using di erent processes. For instance: reduction of character repetitions (e.g., carrrr ! car), or generation of those IV words whose Edit distance with regard to the target OOV is within a given threshold.
The second task consists in selecting the best candidate out of the list generated in the previous step. Two di erent selection methods can be used: string similarity and context inference. To compute string similarity between the OOV and the di erent IV candidates, several measures and strategies can be used: lexical Edit distance, phonemic Edit distance, the longest common subsequence, a x substring, and so on. For context inference, the IV candidates of a given OOV can be ranked and then ltered on the basis of their local contexts. Local contexts are compared against a language model. The main problem of this method is that the local context of an OOV is often constituted by other incorrect lexical variants that are not found in the language model.
These two selection methods (string similarity and context inference) are complementary and then can be used together to select the best candidate.
There are, at least, two signi cant di
erences between the task evaluated in
Finally, there are other approaches to
SMS and tweet normalisation based on very
di erent strategies. For instance (Beaufort
et al., 2010) and
The normalisation method we propose
combines the main strategies and tasks described
in
The design of our algorithm was motivated by the conclusions we draw from the analysis of the development corpus. We observed that the most frequent types of incorrect Spanish OOV are the following: (1) Uppercase/lowercase confusion: patri ! P atri ; (2) character repetition for emphasis: Buuenoo ! Bueno ; (3) language-dependent spelling problems, namely for Spanish: missing accents and letter confusion (v/b, g/j, ll/y, h/; . . . ).
These three types of errors can be solved
using simple speci c rules. For the
remaining phenomena, which correspond to more
heterogeneous problems, we will make use of
generic strategies such as those described in
the previous section: dictionary lookup and
selection of the best IV candidate. For
detection of correct/incorrect OOV, we use the
following method: if no IV associated to an
OOV is found using speci c rules or generic
strategies, then the OOV is considered as
correct. Otherwise it is taken as an ill-formed
OOV. Text is lemmatised and PoS tagged
using FreeLing
Our method contains two modules: a set of lexical resources and an algorithm to detect and correct ill-formed OOV. 2.1
Our system makes use of three di erent lexical resources: ND Normalisation dictionary, containing incorrect lexical variants and their standard forms.
SD Standard dictionary, a list of correct forms generated from the lemmas found in the Real Academia Espan~ola dictionary (DRAE).
PND Proper names dictionary, containing proper names extracted from the Spanish Wikipedia.
In the following, we describe how these three dictionaries have been built.
It was mainly built using the developement data distributed by the organizers for the Tweet Normalization Workshop at SEPLN 2013. We also used as source of data the list of emoticons accesible from http://en.wikipedia.org/wiki/List_of_ emoticons, as well as the list of Spanish abreviations released in http://www.rae. es/dpd/apendices/apendice2.html. Our nal normalisation dictionary contains 824 entries.
The standard dictionary is constituted by
all the forms automatically generated from
the lemmas found in DRAE. These lemmas
have been extracted and freely distributed by
the project http://olea.org/proyectos/
lemarios. Verb forms were generated with
the Cilenis verb conjugator
To make easier the detection of correct OOV
(for instance, proper names and
domainspeci c terms that are not in a standard
vocabulary), it is useful to make use of a large
list of OOV extracted from an enclyclopaedic
resource, for instance the Wikipedia.
Several PND were automatically extracted.
Finally, the PND allowing the best performance
in the normalisation task was extraced as
follows: First, using CorpusPedia
The system takes a list of OOV as input. An OOV is considered as correct if the Dictionary Lookup process is true. Dictionary lookup is a process that consists in searching a token in one of the three lexical dictionaries: ND, SD, or PD. If the OOV is found in one of them, then it is considered as correct. However, even if Dictionary Lookup is false, the OOV will be considered as correct if A x Check is true. A x Check is a process that extracts regular su xes and pre xes from the OOV and veri es whether the stem of the OOV takes part of an entry found in one of the three dictionaries. Otherwise, the OOV can be incorrect.
Given an incorrect OOV, we generate a list of variants. A variant of an OOV is an IV candidate if either Dictionary Lookup or A x Check is true. We distinguish between primary and secondary variants.
Primary variants of an OOV are its most likely IV candidates, according to the type of errors we found in the development corpus. Primary variants will be favoured in the process of candidate selection: if at least a primary variant is found, then the system does not consider secondary variants.
Primary variants of an OOV are those IV candidates derived from the OOV that only di er from the source OOV with regard to one of these linguistic phenomena: Uppercase/lowercase confusion, character repetition, or frequent Spanish spelling errors. The frequent spelling errors include, not only typical problems with accents and frequent letter confusions (v/b, j/g, etc), but also some phonemic conventions, namely the use of \x" for \ch" (e.g. xicle ! chicle). Primary variants generated by simplifying repetition include the cases of interjection reduction: jejeeje ! je. For uppercase and lowercase variation, we take into account that words can be written with only lowercase letters, with capitalisation (proper names or rst position in the sentence), or with only uppercase letters (e.g. acronyms). For instance, given the OOV \pedro", two other variants are generated: \Pedro" and \PEDRO". If one of them is found in the lexical resources, then it is considered as a primary IV candidate. Let us note that a primary variant is considered an IV candidate if either Dictionary Lookup or A x Check is true. 2.2.2
If no primary variant is found as IV candidate, then a large list of secondary variants is generated using Edit distance. In our experiments, we only generate those variants that have Edit distance 1 with regard to the original OOV. Dictionary Lookup and A x Check allow us to identify the list of secondary IV candidates. In the next step, we select the best candidate. 2.2.3
To select the best IV candidate of a given OOV, we compare the local context of each candidate against a language model containing bigrams of tokens found within a window of size 4 (2 tokens to the left and 2 to the right of a given token). More precisely, for each candidate, chi-square measure is computed by considering observed frequencies in the local context against expected frequencies in the language model. The language model was built by selecting lemmas of the following list of PoS categories: nouns, verbs, adjectives, prepositions, and adverbs. Text was processed with FreeLing. We also introduced an important restriction that takes into account whether the IV candidate is either a primary or a secondary variant. A primary variant is always selected even if its chi-square score is 0. It means that a primary variant is always selected even if it is not found in the language model. By contrast, for secondary variants, the chi-square must be higher than 0 to be selected. Candidates are ranked considering chi-square values and the above restriction. The best IV candidate on the top of the rank is selected and given as correction of the OOV. At the end, we apply the capitalisation rule which considers the position of the original ill-formed OOV in the sentence: if it is the rst word in the sentence, then the selected IV candidate must be written with its rst letter in uppercase.
Finally, if no IV candidate (primary or secondary variant) is selected, then the OOV is considered as correct. So, correct OOV are detected in two di erent ways: rst, if Dictionary Lookup or A x Check is true for the original OOV, or if no IV candidate is selected. 3
Some experiments were performed using as test set the development corpus provided by the organisation of the Tweet Normalization Workshop. This corpus contains 500 tweets and 651 OOV manually corrected. The language model used by our system was built from two text sources: the collection of 227,255 tweets provided by the Workshop, which were captured between April 1st and 2nd of 2013, and a collection of news from El Pais and El Mundo captured via RSS Crawling. In sum, the language model was created from 50MB of text. The normalisation dictionary contains annotated information from the sample corpus with 100 tweets provided by the Workshop. For the nal tests, this dictionary also includes the annotated pairs of the development corpus.
Two versions of our system were tested, \Standard" and \Restricted", and compared against two baselines: \Baseline1" and \Baseline2". The standard version has been described in the previous section. The restricted version includes a constraint on short proper names and short acronyms (with less than 5 letters). The constraint prevents short proper names and acronyms from being expanded with secondary variants. For instance, if the OOV is \BBC", the system does not create IV candidates such as \BBV", \ABC", and so on. In Baseline1, we do not separate primary from secondary variants, and all IV candidates are treated as primary variants. Baseline2 does not separate primary from secondary variants, and all IV candidates are treated as secondary variants.
Table 1 shows the results obtained from the experiments performed on the development set. The best performance is achieved with \Restricted", which is based on the algorithm that makes use of restrictions on short proper names. The low scores reached by the baseline systems clearly show that candidates must be separated at di erent levels to be treated in di erent ways. In the test set, \Restricted" achieved 66:3% accuracy, the second best score among the 13 participants in the Tweet-Norm Competition.
Beaufort, Richard, Sophie Roekhaut, LouiseAmelie Cougnon, y Cedrick Fairon. 2010. A hybrid rule/model-based nite-state framework for normalizing SMS messages. En 48th Annual Meeting of the Association for Computational Linguistics, paginas 770{779, Uppsala, Sweden.