The task of normalising user-generated content is a crucial step before analysing social media posts, particularly on Twitter. This paper presents a method for the morphological of tweets by the use of on-line and collectively developed resources, including Wikipedia and a SMS lexicon. The results obtained demonstrate that these resources are a valuable source of knowledge for generating the dictionaries used in the normalisation task.
Microposts published on social media are
characterised by informality, brevity,
frequent grammatical errors and misspellings,
and by the use of abbreviations, acronyms,
and emoticons. These features add
additional di culties in text mining processes
that frequently make use tools designed for
dealing with texts which conform to the
canons of standard grammar and spelling
The micropost normalisation task
enhances the accuracy of NLP tools when
applied to short fragments of texts published
in social media, e.g., the syntactic
normalisation of tweets may improve the accuracy of
existing part-of-speech taggers
The collective knowledge freely available
on the Web, and particularly Wikipedia, has
been used in di erent NLP tasks, such as text
categorization
This paper presents a technique for
morphological normalisation of microposts by the
use of two open data sources namely,
Wikipedia and the SMS dictionary of the Spanish
Association of Internet Users
The paper is structured as follows. Section 2 describes the architecture and the components of the system. Section 3 describes the linguistic resources that we have reused for constructing the normalisation tool. Section 4 presents the evaluation results. Finally, Section 5 presents the conclusions and future lines of work. 2
Figure 1 shows the process followed by the micropost normaliser proposed. The speci c components involved in the overall process are described below.
Standard vocabulary
Normalise Twitter Metalanguage
Element Correct OOV words
SMS Dictionary
OOV word
Twitter metalanguage
element In vocabulary
word Classify OOV word
Spell Checker Dictionary
Correct
Variation OOV word Check & Correct
Spell OOV word
Normalised forms
Concatenate
Normalised Forms Variation
NoES
Correct
Normalised Micropost Micropost
Tokenize
Classify Tokens This component receives the text to be normalised and breaks it into words, Twitter metalanguage elements (e.g., hash-tags, user IDs), emoticons, URLs etc. The output (i.e., the list of tokens) is sent to the Token Classi er component. 2.2
The input of this component is the list of tokens generated by the Tokeniser. It classi es each of them into one of the following categories:
Twitter metalanguage elements (i.e., hash-tags, user IDs, RTs and URLs). Such elements are detected by matching regular expressions against the token (e.g., if a token starts by the symbol \#", then it is a hash-tag). Each token classi ed in this category is sent to the Twitter Metalanguage Normaliser component.
Words contained in a standard language dictionary, excluding proper nouns. Each token classi ed in this category is sent to the Normalised Forms Concatenator component.
Out-Of-Vocabulary (OOV) words. They are words not found in a standard dictionary, neither are Twitter metalanguage elements. Each token classi ed in this category is sent to the OOV Word Classi er component. 2.3
This component receives every token previously classi ed as OOV by the Token Classier and detects if it is correct, wrong, or unknown. If the token is wrong, the component returns the correct form of the token. The OOV Word Classi er Component executes the following process: 1. Firstly, the token is looked up in a dictionary of correct OOV words. The search disregards both case and accents. (a) If an exact match of the token is found in the dictionary (e.g., both forms are capitalised), then the token is classi ed as Correct and sent to the Normalised Forms Concatenator component with no variation. (b) If the token is found with variations of case or accentuation, then the token is classi ed as Variation and its correct form is sent to Normalised Forms Concatenator component. (c) If the token is not found in the dictionary, then the process continues in step 2. 2. The token is looked up in a SMS dictionary which contains tuples with the SMS term and its corresponding correct form. The search is case-unsensitive, and does not consider accent marks. (a) If the token is found in the SMS dictionary, then it is classi ed as Variation and its correct form is retrieved and sent to Normalised
Forms Concatenator component. (b) If the token is not found in the dictionary, then it is sent to the Spell
Checker and Corrector component. 2.4
This component checks the spelling of the token received and returns its correct form when possible. To do so, it executes the following process: 1. Firstly, the token is matched against regular expressions to nd whether it contains characters (or sequences of characters) repeated more than twice (e.g., \loooooollll" and \jajaja"). (a) If the token contains repeated characters (or sequences of characters), the repeated ones are removed (e.g., \lol", and \ja"), and the resulting form is sent back to the OOV Word Classi er, since the new form may be included into the correct words set. (b) If the token does not contain repeated characters (or sequences of characters), then the process continues in step 2. 2. The token is sent to an existing spell checking and correction implementation reused by this component. (a) If the spell is correct, the token is classi ed as Correct and sent to the Normalised Forms Concatenator component without a variation. (b) If the spell is not correct, the token is classi ed as Variation, and the
rst correct form returned by the spelling corrector is sent to Normalised Forms Concatenator. (c) If the spell checker is not able to propose a correct form, the token is classi ed as Unknown and sent to the Normalised Forms Concatenator without a variation. 2.5
This component performs a syntactic
normalisation of Twitter meta-language elements.
Speci cally, it executes a set of rules,
previously proposed by
(1) Remove the sequence of characters \RT" followed by a mention to a Twitter user (marked by the symbol \@") and, optionally, by a colon punctuation mark; (2) Remove user IDs that are not preceded by a coordinating or subordinating conjunction, a preposition, or a verb; (3) Remove the word \via" followed by a user mention at the end of the tweet; (4) Remove all the hash-tags found at the end of the tweet; (5) Remove all the \#" symbol from the hashtags that are maintained; (6) Remove all the hyper-links contained within the tweet; (7) Remove ellipses points that are at the end of the tweet, followed by a hyper-link; (8) Replace underscores with blank spaces; (9) Divide camel-cased words in multiple words (e.g., \BarackObama" is converted to \Barack Obama"). 2.6
This component receives the normalised form of each token, and amends the micropost. 3
The system described makes use of the following resources.
We use Freeling
In addition, we use the POS-tagging module of Freeling within the Token Classi er component. As we deactivate Freeling's probability assignment and unknown word guesser module, all the words which are not contained in Freeling's POS-tagging dictionary are not marked with a tag and considered as OOV words. Our standard vocabulary is, thus, the Freeling dictionary itself.
We have populated the correct OOV
words dictionary (used by the OOV Word
Classi er component) by making use of the
list of articles' titles from Wikipedia
In order to increase the coverage of the
correct OOV words dictionary, we
incorporated into it a list of rst names from the
Spanish National Institute of Statistics
Additionally, we have populated the SMS
dictionary and its corresponding correct
forms, from the SMS dictionary of the
Spanish Association of Internet Users
Finally, the Spell Checker and Corrector
component makes use of Jazzy
The evaluation of the technique previously described was done by using two development corpora and a test corpus provided by the organisation of the Tweet Normalisation Workshop at SEPLN 2013. Speci cally, we evaluated the performance of the OOV identi cation, classi cation and correction tasks. The accuracy of the normalization task for the Twitter metalanguage elements was not evaluated since it was out of the scope of the workshop challenge.
Table 1 shows the results of the evaluation, including the size of each evaluation corpus (column 2), the precision obtained by using either Wikipedia or the SMS dictionary separately (columns 3 and 4 respectively), and the overall precision achieved by exploiting both dictionaries (column 5).
As Table 1 re ects, both dictionaries help to improve the nal precision score, being the SMS dictionary the one which contributes the most. This can be explained with the coverage of OOV words by each of the dictionaries, which is shown in Table 2. The
Devel. 1 Devel. 2 Test
0.336 0.317 0.361
We presented a method for tweet normalisation that relies on existing web resources collectively developed, nding that such resources, useful for many NLP tasks, are also valid for the task of micropost normalisation.
With respect to the future lines of work, we plan to adapt the normaliser to new languages by the incorporation of the corresponding dictionaries and improving the existing lexicons by the use of more available resources, such as the anchor texts from intra wiki links.
Additionally, we plan to improve the
normalization of multiword expressions, as
different words should be transformed in just
one (e.g., \a cerca de" should be
transformed into \acerca de"), as well as cases
where joined words should be splitted (e.g.
\realmadrid") by using existing word
breaking techniques, such as the one described in
Finally, we will study how the normalisation process a ects to di erent opinion mining tasks, including sentiment analysis and topic identi cation.
This research is partially supported by the Spanish Centre for the Development of Industrial Technology under the CENIT program, project CEN-20101037, \Social Media" (http://www.cenitsocialmedia.es). We are very grateful to AUI (Asociacion de Usuarios de Internet) for facilitating the textese dictionary used in this work to us.