This paper describes a lexicon-based text normalisation approach for Spanish tweets. We rst compare English and Spanish text normalisation, and hypothesise that an approach previously proposed for English can be adapted to Spanish. A corpus-derived normalisation lexicon is built using distributional similarity, and is combined with existing lexicons (e.g., containing Spanish Internet slang). These lexicons enable a very fast, look-up based approach to text normalisation. Experimental results indicate that the corpus-derived lexicon complements existing lexicons, but that the approach could be improved through better handling of certain word types, such as named entities.
A tremendous amount of user-generated text
is produced on social media sites such as
Twitter and Facebook, and can be
leveraged for natural language processing (NLP)
tasks such as sentiment analysis
This paper proposes a lexicon-based approach to Spanish text normalisation. In particular, we adapt the method of Han, Cook, and Baldwin (2012) to build a normalisation lexicon that maps non-standard words to their standard forms relative to a vocabulary, i.e., out-of-vocabulary (OOV) words are mapped deterministically to in-vocabulary (IV) words. This enables a very fast, look-up based approach to text normalisation. In our approach an OOV word is rst looked up in an automatically-derived normalisation lexicon that is complemented with entries from Spanish Internet slang dictionaries and the development data. If the OOV word is found in this lexicon it is normalised according to its entry, otherwise it is left unchanged. During this normalisation step, OOV words and the resulting normalisations are down-cased, so a nal case restoration step is performed to appropriately capitalise the lowercased normalisations.
The lexicon-based normalisation approach of Han, Cook, and Baldwin (2012) was evaluated on English tweets. In this section we consider the plausibility of adapting their method from English to Spanish, and identify the following key factors: Orthography: if we consider diacriticised letters as single characters, Spanish has more characters than English, and diacritics can lead to di erences in meaning, e.g., mas means \more", and mas means \but". The method of Han, Cook, and Baldwin (2012) uses Levenshtein distance to measure string similarity. We simply convert all characters to fused Unicode code points (treating a and a as di erent characters) and compute Levenshtein distance over these forms.
Word segmentation: Spanish and English words both largely use whitespace segmentation, so similar tokenisation strategies can be used.
Morphophonemics: Phonetic modeling of words | a component of the method of Han, Cook, and Baldwin (2012) | is available for Spanish using an o -the-shelf Double Metaphone implementation.1 Lexical resources: A lexicon and slang dictionary | key resources for the method of Han, Cook, and Baldwin (2012) | are available for Spanish.
Overall, English and Spanish text share important features, and we hypothesise that adapting a lexicon-based English normalisation system to Spanish is feasible.
One important component of this Spanish normalisation task is case restoration: e.g., maria as a name should be normalised to \Maria". Most previous English Twitter normalisation tasks have focused on lowercase words and ignored capitalisation. 3
The system consists of two steps: (1) downcase all OOVs and normalise them based on a normalisation lexicon which combines entries from existing lexicons (Section 3.1) and entries automatically learnt from a Twitter corpus (Section 3.2); (2) restore case for normalised words (Section 3.3).
1https://github.com/amsqr/ Spanish-Metaphone 3.1
Our normalisation transforms OOV forms to
IV words, and thus a Spanish lexicon is
required to determine what is OOV. To this
end, we use the Freeling 3.0 Spanish
dictionary
We collected 146 Spanish Internet slang expressions and cell phone abbreviations from the web (Slang lexicon).2 We further extracted normalisation pairs from the development data (Dev Lexicon).
Analysing the development data we noticed that many person names are not correctly capitalised. We formed Name Lexicon from a list of 277 common Spanish names.3 This lexicon maps lowercase person names to their correctly capitalised forms. 3.2
The small, manually-crafted normalisation lexicons from Section 3.1 have low coverage over non-standard words. To improve coverage, we automatically derive a much larger normalisation lexicon based on distributional similarity (Dist Lexicon) by adapting the method of Han, Cook, and Baldwin (2012).
We collected 283 million Spanish tweets
via the Twitter Streaming API4 from
21/09/2011{28/02/2012. Spanish tweets
were identi ed using langid.py
For a given OOV, we de ne its confusion set to be all IV words with Levenshtein distance 2 in terms of characters or 1 in terms of Double Metaphone code. We rank the items in the confusion set according to their distributional similarity to the OOV. Han, Cook, and Baldwin (2012) considered many con gurations of distributional similarity for normalisation of English tweets. We 2http://goo.gl/wgCFSs and http://goo.gl/ xsYkDe, both accessed on 26/06/2013
3https://en.wikipedia.org/wiki/Spanish_ naming_customs 4https://dev.twitter.com use the same settings they selected: context is represented by positionally-indexed bigrams using a window size of 2 tokens; similarity is measured using KL divergence. An entry in the normalisation dictionary then consists of the OOV and its top-ranked IV.
From development data, we observe that in many cases when a correct normalisation is identi ed, there is a large di erence in KL divergence between the rst- and secondranked IVs. Conversely, if the KL divergence of the rst- and second-ranked normalisation candidates is similar, the normalisation is often less reliable. As shown in Table 3.2, callendo (\cayendo") is a correctly-derived (OOV, IV) pair, but guau (\y") is not.
Motivated by this observation, we lter the derived (OOV, IV) pairs by the KL divergence ratio of the rst- and second-ranked IV words for the OOV. Setting a high threshold on this KL divergence ratio increases the reliability of the derived lexicon, but reduces its coverage. This ratio was tested for values from 1.0 to 3.0 with a step size of 0.1 over the development data and Slang Lexicon. As shown in Figure 1, the best precision (94.0%) is achieved when the ratio is 1.9.5 We directly use this setting to derive the nal lexicon, instead of further re-ranking the (OOV,IV) pairs using string similarity.
5Here precision is de ned as #correct normalisations . #normalisations Lexicon Combined Lexicon
Slang Lexicon Dev Lexicon Dist Lexicon
Name Lexicon + Edit distance Baseline Accuracy We set the case of each token that was normalised in the previous step (which is down-cased at the current stage) to its mostfrequent casing in our corpus of Spanish tweets. We also capitalise all normalised tokens occurring at the beginning of a tweet, or following a period or question mark. 4
We evaluated the lexicons using classi cation accuracy, the o cial metric for this shared task, on the tweet-norm test data. This metric divides the number of correct proposals | OOVs correctly normalised or left unchanged | by the number of OOVs in the collection. This is termed \precision" by the task organisers, but a true measure of precision would be based on the number of OOVs that were actually normalised. We therefore use the term \accuracy" here.
We submitted two runs for the task. The rst, Combined Lexicon (Table 4), uses only the combination of lexicons from Section 3, and achieves an accuracy of 0.52. The second run builds on Combined Lexicon but incorporates normalisation based on character edit distance for words with many repeated characters. We observed that such words are often non-standard, and tend not to occur in the lexicons because of their relatively low frequency. For words with 3 repeated characters, we remove all but one of the repeated characters, and then select the most similar IV word according to characterbased Levenshtein distance. The accuracy of this run is 0.54 (+ Edit distance, Table 4).
We further consider an ablative analysis of
the component lexicons of Combined
Lexicon. As shown in Table 4, when Slang
Lexicon ( Slang Lexicon) or Name
Lexicon ( Name Lexicon) are excluded,
accuracy declines only slightly. Although this
suggests that existing resources play only a
minor role in the normalisation of Spanish
tweets, this is likely due in part to the
relatively small size of Slang Lexicon, which is
much smaller than similar English resources
that have been e ectively exploited in
normalisation | i.e., 145 Spanish entries versus
5k English entries used by
On the other hand, normalisation
lexicons derived from tweets | whether based
on the development data (Dev Lexicon) or
automatically learnt (Dist Lexicon) |
substantially impact on accuracy ( Dev
Lexicon and Dist Lexicon). These ndings
for the automatically derived Dist Lexicon
are in line with previous ndings for English
Twitter normalisation
We considered an experiment in which we used Combined Lexicon, but ignored case in the evaluation; the accuracy was 0.56. This corresponds to the upper-bound on accuracy if our system performed case restoration perfectly, and suggests that improving the case restoration of our system would not lead to substantial gains in accuracy.
In the nal row of Table 4 we show results for a baseline method which makes no attempt to normalise the input. All lexiconbased methods improve substantially over this baseline.
To further analyse our lexicon-based normalisation approach, we categorise the errors for both false positives (OOVs that were normalised, but incorrectly so) and false negatives (OOVs that were not normalised, but should have been). As shown in Table 4, 37% of false positives are incorrect lexical forms, e.g., algerooo is normalised to \algero" and not its correct form \alegra". Further examination shows that 23% of these cases are incorrectly normalised to \que", suggesting that distributional similarity alone is insu cient to capture normalisations for some nonstandard words.
Surprisingly, we found some OOVs included in the test data, but excluded from the gold-standard annotations (due to tweet Error type Incorrect lexical form Not available Accent error Case error One to many Annotation error Number
Percentage deletions), or present in the test data, but not found in the tweets, and excluded in the gold standard. These error types are denoted as \Not available" in Table 4, and account for the second largest source of false positives.
Incorrect accents and casing account for 17% and 8% of false positives, respectively. In both of these cases, contextual information, which is not incorporated in the proposed approach, could be helpful. Finally, we identi ed two one-to-many normalisations (which are outside the scope of our normalisation system), and one case we judged to be an annotation error.
We analysed a random sample of 20 of the 280 false negatives, and found irregular character repetitions and named entities to be the main sources of errors, e.g., uajajajaa (\ja") and Pedroo (\Pedro").6 The lexicon-based approach could be improved, for example, by using additional regular expressions to capture repetitions of character sequences. Errors involving named entities reveal the limitations of using the Freeling 3.0 Spanish dictionary as the IV lexicon, as it has limited coverage of named entities. A corpus-derived lexicon (e.g., from Wikipedia) could help improve the coverage. 5
In this paper, we applied a lexicon-based approach to normalise non-standard words in Spanish tweets. Our analysis suggests that the corpus-derived lexicon based on distributional similarity improves accuracy, but that this approach is limited in terms of exibility (e.g., to capture accent variation) and lexicon coverage (e.g., of named entities). In future work, we plan to expand the IV lexicon, and incorporate contextual information to improve normalisation involving accents and casing.
6Pedro is not in our collected list of Spanish names.
NICTA is funded by the Australian government as represented by Department of Broadband, Communication and Digital Economy, and the Australian Research Council through the ICT centre of Excellence programme. The authors would like to thank the anonymous reviewers for their valuable feedback and language expertise.