This paper proposes a resource-based lexical approach for addressing the TWEET-NORM task. The proposed system exposes a simple but extensible modular architecture in which each analysis module independently proposes correction candidates for each OOV word. Each one of these analysis modules tries to address a specic problem and each one works in a very dierent way. The resources are used as the main component for the OOV detection system and they works as support for the validation and ltering of candidates.
Introduction and objectives One of the most important challenges facing us today is how to process and analyze the large amount of information on the Internet, and especially social networking sites like Twitter, where millions of people daily express ideas and opinions on any topic of interest. These texts, called tweets, are characterized by having a short length (140 characters) that is too small compared with the size of traditional genres.
Consequently, users of these networks have developed a new form of expression that includes SMS-style abbreviations, lexical variants, letters repetitions, use of emoticons, etc. The result is that current NLP tools can have problems to process and understand these short and noisy texts unless they are normalized rst.
The TWEET-NORM lexical normalization task proposes the automatic cleansing of a set amount of tweets by identifying and normalizing, abbreviations, words with repeated letters, and generally any out of the vocabulary (OOV) words, regardless of syntactic or stylistic variants.
Before doing any normalization process, we previously did a characterization of the existing phenomena, being easier trying to address the underlying causes of OOVs. To perform this characterization we used a previously collected dataset, composed of 3.1 millions of tweets related with the 2012 UEFA European Football Championship .
The table 1 shows that characterization and provides examples for each phenomenon.The gure 1 also shows the phenomena ratio in a clearer way. It is observed that most of errors t into 5 major categories and most of errors are associated with the fast and informal writing in Twitter, usually done from a mobile device.
The categories proposed for this task are coarser than our characterization. Orthographic errors, Texting language and Character reptitions t into the Variation category, Free Inections and correct words t into the Correct category. Other Language and Ascii Art would t into NS/NC category.
The system proposed in this paper is based on lexical approaches only and it is mainly
Ortographic errors Texting Language Character repetition Ascii Art Free inections Other errors Other Language Multiple phenomena
Examples sacalo ! sÆcalo, trapirar ! transpirar, . . . x2 ! por dos, q ! que, aro ! claro, . . . siiiiiiiii ! si, quiiiiieeeeroooo ! quiero, . . . « ¤ oO._.Oo . . . besote, gatino, bonico, . . . htt, asdafawecas, engoriles, . . . ow, ftw, great, lol, . . . diass ! das, artooo ! rato, . . .
In essence, our system works straightforwardly: it examines each word at lexical scope, determine if it is an OOV using the knowledge, generate possible correction candidates and select the best one. 2
Architecture and components of the system The architechture of our system proposed for this task is pretty straightforward. It is composed by several main components:
Candidate scoring and selection module The gure 2 shows all the process explained before and how the components are interconnected in a single diagram.
The preprocessing module performs the typical initial processing step done in lexical analysis, generating a stream of tokens from tweets taking into account things like hashtags and usernames, numerals, dates and preserving emoticons during the splitting. The detection module tries to determine if a token is an OOV or not. This module performs that detection using resources and checking if a token belongs to any resource. We used a set of lexicons, each one providing known forms used in Twitter, the Spanish language, well known emoticons or even colloquial inections.
Given an OOV Token, an analyzer module perform some error guessing process and try to estimate corrections from it. The specic process varies for each analyzer. Every analyzer provides some kind of basic scoring providing some degree of condence for each correction proposed. The analyzers used for this task were the following:
Tranformation rules : This analyzer holds a collection of hand-crafted rules, each one representing some kind of well dened error and transforms that token into candidate of correction. It is possible to generate more than a candidate due multiple rule matching, but the number usually is limited to a few.
These rules are intended to address phenomena that the edit distance module does not correctly address like Character Repetition or Texting Language. The
Edit distance: This module works very similar to distance-based suggestion scheme commonly found in spell checkers. The main dierence is that it takes into account multiple lexicons instead a monolithic one.
Language: This module tries to identify whether language the OOV token actually belongs to.
Notice that the language analyzer module
does not actually perform any correction,
because if the token comes from another
language only has to be marked not corrected.
This module uses a trigram language guessing
module Python 3 implementation
The candidate generation module asks for candidates to each analyzer, performing a validation and ltering step, thus removing some incorrectly generated candidates from transformation rules according to validation rules and used language resources. Also removes duplicates.
The candidate selector module applies a normalizing scoring function from the condence values provides for each candidates,
kn, cn, ! con xaval, coxe ! chaval, coche sisisisisisi, nonononono ! si, no tkm, tqm ! te quiero mucho sorts them and selects the best one.
The system generates a token stream from the Tweet using the preprocessing module. For each token, determines if a token is an OOV or not using the detector module. If the token is an IV, no further processing is done because is a valid form. Otherwise, the token is an OOV, the candidate generator module creates a tentative list using the analyzers previously described. As nal step, the candidate selector module selects the best candidate for correction. 3
Resources employed We have used several lexicons for the detection and analyzing stages in our system. All of them are in raw text format and one entry per line.
The table 3 shows stats about all the lexicons used.
Spanish
40 320
Description Common forms from Spanish. Based on LibreOffice dictionaries.
Common forms related to Twitter.
Handcrafted.
Commonly used emoticons. Handcrafted.
For the transformation rule module, we crafted a ruleset of 71 rules. The syntax used in the ruleset le vaguely resembles a CSV format, being a rule per line and each line holds information about the matching and the transformation process.
The language detector module uses a trigram character language model implementation as backend and uses dictionaries as backo just in case of insucient data for language estimation. The table 4 shows performance values of the system against the provided corpus and activating dierent analyzer modules. It is observed that the accuracy of the system improves signicantly as more modules are activated.
However, our implementation performance is hindered due high dierences between the preprocessing used to build the corpus for this task and our detection and preprocessing system. Our preprocessing module leads to dierent sets of OOVs to be considered, often leading to outputs that dier in length respect the ones provided with the task. These discrepancies result in a high rate of what the provided test script counts as align errors.
Distance module Rule module Distance + Rule Full system
0:2036 0:3307 0:3905 0:5053
0:2526 0:3231 0:2281 0:1684
It is worth mentioning that we used threshold of k 2 for the edit distance module because most of correct candidates are within that threshold. Though is true that selecting a higher k includes more candidates, most of newly included candidates are not a valid solution and usually they will have a low condence score and will not be selected. 5
Conclusions and future work We provide a resource-aided lexical solution for the proposed task using an extensible architecture made of independent modules. Our system has much room for improvement like adding a ngram segmenter, including context during analysis or using automatic methods for improving the candidate scoring and selection.