This paper presents a linguistic approach based on weighted- nite state transducers for the lexical normalisation of Spanish Twitter messages. The system developed consists of transducers that are applied to out-of-vocabulary tokens. Transducers implement linguistic models of variation that generate sets of candidates according to a lexicon. A statistical language model is used to obtain the most probable sequence of words. The article includes a description of the components and an evaluation of the system and some of its parameters.
Text messaging (or texting) exhibits a considerable degree of departure from the writing norm, including spelling. There are many reasons for this deviation: the informality of the communication style, the characteristics of the input devices, etc. Although many people consider that these communication channels are \deteriorating" or even \destroying" languages, many scholars claim that even in this kind of channels communication obeys maxims and that spelling is also principled. Even more, it seems that, in general, the processes underlying variation are not new to languages. It is under these considerations that the modelling of the spelling variation, and also its normalisation, can be addressed. Normalisation of text messaging is seen as a necessary preprocessing task before applying other natural language processing tools designed for standard language varieties.
Few works dealing with Spanish text
messaging can be found in the literature. To
the best of our knowledge, the most
relevant and recent published works are
The system has three main components that are applied sequentially: An analyser performing tokenisation and lexical analysis on standard word forms and on other expressions like numbers, dates, etc.; a component generating word candidates for outof-vocabulary (OOV) tokens; a statistical language model used to obtain the most likely sequence of words; and nally, a truecaser giving proper capitalisation to common words assigned to OOV tokens.
Freeling
E L
where W is the automata representing the
OOV token, E is an edit transducer
generating possible variations on tokens, and L is
the set of target words. The composition of
these three modules is performed using an
on-line implementation of the e cient
threeway composition algorithm of
In this section, the resources employed by the components of the system are described: the edit transducers, the lexical resources and the language model. 3.1
We follow the classi cation of
Some letters are found used as logograms, with a phonetic value. They are dealt with by optional rewrites altering the orthographic form of tokens: ReplaceLogograms = (x (!) por )
(2 (!) dos) (@ (!) ajo) . . .
Also laughs, which are very frequent, are considered logograms, since they represent sounds associated with actions. The multiple ways they are realised, including plurals, are easily described with regular expressions.
Pictograms like emoticons entered by means of ready-to-use icons in input devices are not treated by our system since they are not textual representations. However textual representations of emoticons like :DDD or xDDDDDD are recognised by regular expressions and mapped to their canonical form by means of simple transducers.
The string operations for initialisms (or acronymisation) and shortenings are di cult to model without incurring in an overgeneration of candidates. For this reason, only common initialisms, e.g., sq (es que), tk (te quiero) or sa (se ha), and common shortenings, e.g., exam (examen) or nas (buenas), are listed.
For the omission of letters several transducers are implemented. The simplest and more conservative one is a transducer introducing just one letter in any position of the token string. Consonantal writing is a special case of letter omission. This kind of writing relies on the assumption that consonants carry much more information than vowels do, which in fact is the norm in same languages like Semitic languages. Some rewrite rules are applied to OOV tokens in order to restore vowels: InsertVowels = invert(RemoveVowels) RemoveVowels = Vowels (!)
We consider non-standard spellings standard when they are widely used. These include spellings for representing regional or informal speech, or choices sometimes conditioned by input devices, as non-accented writing. For the case of accents and tildes, they are restored using a cascade of optional rewrite rules like the following: RestoreAccents = (njnijnyjnh (!) n~) (a (!) a) (e (!) e) . . .
Also words containing k instead of c or qu, which appears frequently in protest writings, are standardised with simple transducers. Some other changes are done to some endings to recover the standard ending. There are complete paradigms like the following, which relates non-standard to standard endings: -a -ada -as -adas -ao -ado -aos -ados
We also consider phonetic writing as a kind of non-standard writing in which a phonetic form of a word is alphabetically and syllabically approximated. The transducers used for generating standard words from their phonetic and graphical variants are: DephonetiseWriting =
invert(PhonographemicVariation) PhonographemicVariation =
GraphemeToPhoneme PhoneCon ation PhonemeToGrapheme
GraphemeVariation In the previous de nitions, the PhoneConation makes phonemes equivalent, as for example the IPA phonemes /L/ and /J/. Linguistic phenomena as seseo and ceceo, in which several phonemes were con ated by 16th century, still remain in spoken variants and are also re ected in texting. The GraphemeVariation transducer models, among others, the writing of ch as x, which could be due to the in uence of other languages.
Spacing in texting is also non-standard. In the normalisation task, some OOV tokens are in fact juxtaposed words. The possible decompositions of a word into a sequence of possible words is: shortest-paths(W SplitConjoinedWords L( L)+), where W is the word to be analysed, L( L)+ represents the valid sequences of words and SplitConjoinedWords is a transducer introducing blanks ( ) between letters and undoing optionally possible fused vowels: SplitConjoinedWords = invert(JoinWords) JoinWords = (a a (!) a<1>) . . . (u u (!) u<1>) ( (!) ) Note that in the previous de nition, some rules are weighted with a unit cost <1>. These costs are used by the shortest-paths algorithm as a preference mechanism to select non-fused over fused sequences when both cases are possible.
Expressive lengthening, which consist in repeating a letter in order to convey emphasis, are dealt with by means of rules removing a varying number of consecutive occurrences of the same letter. An example of a rule dealing with letter a repetitions is a (!) / a . A transducer is generated for the alphabet.
Because messages are keyboarded, some errors found in words are due to letter transpositions and confusions between adjacent letters in the same row of the keyboard. These changes are also implemented with a transducer.
Finally, a Levenshtein transducer with a maximum distance of one has been also implemented. 3.2
The lexicon for OOV token normalisation
contains mainly Spanish standard words,
proper names and some frequent English
words. These constitute the set of target
words. We used the DRAE
We use a language model to decode the word
graph and thus obtain the most probable
word sequence. The model is estimated from
a corpus of webpages compiled with Wacky
The restoring of case information in
badlycased text has been addressed in
In order to generate the confusion sets we used two edit transducers applied in a cascade. If neither of the two is able to relate a token with a word, the token is assigned to itself.
The rst transducer generates candidates according to the expansion of abbreviations, the identi cation of acronyms, pictograms and words which result from the following composition of edit transducers combining some of the features of texting: RemoveSymbols LowerCase Deaccent RemoveReduplicates ReplaceLogograms StandardiseEndings DephonetiseWriting Reaccent MixCase
The second edit transducer analyses tokens that did not receive analyses with the rst editor. This second editor implements consonantal writing, typing error recovery, an approximate matching using a Levenshtein distance of one and the splitting of juxtaposed words. In all cases, case, accents and reduplications are also considered. This second transducer makes use of an extended lexicon containing sequences of simple words.
Several experiments were conducted in order to evaluate some parameters of the system. In particular, the e ect of the order of the n-grams in the language model and the e ect of generating confusion sets for OOV tokens only versus the generation of confusion sets for all tokens. For all the experiments we used the test set provided with the tokenization delivered by Freeling.
For the rst series of experiments, tokens identi ed as standard words by Freeling receive the same token as analysis and OOV tokens are analysed with the system. Recall on OOV tokens is of 89.40 %. Confusion sets size follows a power-law distribution with an average size of 5.48 for OOV tokens that goes down to 1.38 if we average over the rest of the tokens. Precision for 2- and 4-gram language models is 78.10 %, but the best result is obtained with 3-grams, with an precision of 78.25 %.
There is a number of non-standard forms that were wrongly recognised as invocabulary words because they clash with other standard words. In the second series of experiments a confusion set is generated for each word in order to correct potentially wrong assignments. Average size of confusion sets increases to 5.56.1 Precision results for the 2-gram language model is of 78.25 % but 3- and 4-gram reach both an precision of 78.55 %.
From a quantitative point of view, it seems that slighty better results are obtained using a 3-gram language model and generating confusion sets not only for OOV tokens but for all the tokens in the message. In a qualitative evaluation of errors several categories show up. The most populated categories are those having to do with case restoration and wrong decoding by the language model. Some errors are related to particularities of DRAE, from which the lexicon was derived (dispertar or malaleche). Non standard morphology is observed in tweets, as in derivatives (tranquileo or loquendera). Lack of abbreviation expansion is also observed (Hum). Faulty application of segmentation accounts for a few errors (mencantaba). Finally, some errors are not on our output but on the reference (Hojo). 5
No attention has been payed to multilingualism since the task explicitly excluded tweets from bilingual areas of Spain. However, given that not few Spanish speakers (both in Europe and America) are bilingual or live in bilingual areas, mechanisms should be provided to deal with other languages than English to make the system more robust.
We plan to build a corpus of lexically standard tweets via the Twitter streaming API to determine whether n-grams observed in a 1We removed from the calculation the token mes de abril, which receives 308,017 di erent analyses due to the combination of multiple editions and segmentations.
Twitter-only corpus improve decoding or not as a side e ect of syntax being also non standard.
Qualitative analysis of results showed that there is room for improvement experimenting with selective deactivation of items in the lexicon and further development of the segmenting module.
However, initialisms and shortenings are features of texting di cult to model without causing overgeneration. Acronyms like FYQ, which correspond to the school subject of F sica y Qu mica, are domain speci c and difcult to foresee and therefore to have them listed in the resources.