In recent years, Sentiment Analysis has become one of the most interesting topics in AI research due to its promising commercial bene ts. An important step in a Sentiment Analysis system for text mining is the preprocessing phase, but it is often underestimated and not extensively covered in literature. In this work, our aim is to highlight the importance of preprocessing techniques and show how they can improve system accuracy. In particular, some di erent preprocessing methods are presented and the accuracy of each of them is compared with the others. The purpose of this comparison is to evaluate which techniques are e ective. In this paper, we also present the reasons why the accuracy improves, by means of a precise analysis of each method.
The subjective analysis of a text is the main task of Sentiment Analysis (SA),
also called Opinion Mining. One of the basic tasks in SA is to predict the
polarity of a given sentence, to nd out if it expresses a positive, negative or neutral
feeling about a certain topic [
Thus, given a corpus of raw data sets, the rst step of SA is the preprocessing of those data. Preprocessing involves a series of techniques which should improve the next phases of elaboration, in order to achieve better performances.
As illustrated in [
Some of these techniques are exposed in the work of A. Balahur [
The work of A. Agarwal et al. [
In the context of SemEval (Task 4)1, for SA in Twitter, N. F. Silva et al. [
In [
The main goal of the present work is to analyse and compare di erent preprocessing steps found in literature, and to actually de ne the best combination of the considered methods. In fact, preprocessing is often seen as a fundamental step for SA, but rarely is it carefully evaluated, thus leaving the open question of why and to what extent does it increase the accuracy of the classi er.
The data set used in this work is the one provided by SemEval [
The paper is structured as follows. In Section 2, a brief introduction to Machine Learning steps is provided. In addition, the section describes the techniques and algorithms used for classi cation and features selection. In Section 3, the considered preprocessing techniques are presented in detail. Section 4 shows the performance of the obtained classi er for each preprocessing method, on two di erent data sets, and discusses the e ectiveness of each of such techniques. In Section 5, a brief discussion of the proposed work and an analysis of the achieved results conclude the paper.
This section describes the di erent preprocessing modules that have been used in this work3. All of them are built in Python4 and they work with version 2.7.
The pipeline of the project is organized in the following way. Firstly, we obtain the 2015 and 2016 data sets (both training and test) of Twitter Sentiment Analysis from SemEval. The training sets are subjected to the various preprocessing techniques analysed in this work. After the text of each instance of a set has been preprocessed, the resulting sentences (the cleaned tweets) become the instances of a new training set. Then, such a data set is used for training a classier and the corresponding test set is classi ed by Weka. Finally, the accuracies of the classi ers obtained from di erent preprocessing modules are compared with each other, in order to evaluate the e ciency and e ectiveness of each technique.
The classi er is made by using Naive-Bayes Multinomial (NBM) method, i.e., a ML algorithm that gives rise to a probabilistic classi er, which works on the basis of the Bayes Theorem, with the strong assumption that features are mutually independent. Let X = (x1; : : : ; xn) be the feature vector of an instance in the data set, that is, a binary vector that takes into account the presence of a feature in that instance, and let C1; : : : ; CK be the possible outputs (classes). The problem is to gain the posterior probability of having the class Ck as output, given the feature vector X, and given the prior probability p(Ck) for each class. Thanks to the Bayes Theorem and the independence between features, the probability that needs to be estimated is the conditional p(XjCk), and then a classi er is trained with a decision rule, such as the Maximum a
Posteriori (MAP) rule. In summary, the probabilistic model of NBM can be expressed in terms of the following formula: p(XjCk) = (Pi xi)! Y pkxii
Qi xi! i where X = (x1; : : : ; xn) is the feature vector, pi is the probability that the feature i appears, Ck is a class and pki is the probability that feature i occurs in the class Ck. Then, Information Gain (IG) is the algorithm used for feature selection [REF]. It evaluates the presence or absence of a feature in a document by measuring its probability of belonging to a class. The amount of information needed to exactly classify an instance D is de ned recursively as follows.
InfoA(D) =
v X jDjj Info(Dj) j=1 jDij when the instance D is divided by some feature attribute A = fa1; : : : ; avg into sub-instances D1; : : : ; Dv. 3 3.1
This rst module manages basic cleaning operations, which consist in removing unimportant or disturbing elements for the next phases of analysis and in the normalization of some misspelled words. In order to provide only signi cant information, in general a clean tweet should not contain URLs, hashtags (i.e. #happy) or mentions (i.e. @BarackObama). Furthermore, tabs and line breaks should be replaced with a blank and quotation marks with apexes. This is useful in order to obtain a correct elaboration by Weka (i.e. not closing a quotation mark causes a wrong read by the data mining software causing a fatal error in the elaboration). After this step, all the punctuation is removed, except for apexes, because they are part of grammar constructs such as the genitive.
The next operation is to remove the vowels repeated in sequence at least three times, because by doing so the words are normalized: for example, two words written in a di erent way (i.e. cooooool and cool ) will become equals. Another substitution is executed on the laughs, which are normally sequences of \a" and \h". These are replaced with a \laugh" tag.
The last step is to convert many types of emoticons into tags that express their sentiment (i.e. :) ! smile happy). The list of emoticons is taken from Wikipedia5.
Finally, all the text is converted to lower case, and extra blank spaces are removed.
All the operations in this module are executed to try to make the text uniform. This is important because during the classi cation process, features are chosen only when they exceed a certain frequency in the data set. Therefore, after the basic preprocessing operations, having di erent words written in the same way helps the classi cation. 3.2
This module reduces the number of emoticons to only two categories: smile positive and smile negative, as shown in Table 1. This module uses the external python library PyEnchant6, which provides a set of functions for the detection and correction of misspelled words using a dictionary.
As an extension, this module allows us to substitute slang with its formal meaning (i.e., l8 ! late), using a list. It also allows us to replace insults with the tag \bad word".
The motivation for the use of these functions is the same as for the basic preprocessing operation, i.e., to reduce the noise in text and improve the overall classi cation performances. Stemming techniques put word variations like \great", \greatly", \greatest", and \greater" all into one bucket, e ectively decreasing entropy and increasing the relevance of the concept of \great". In other words, Stemming allows us to consider in the same way nouns, verbs and adverbs that have the same radix.
This method is already implemented in Weka and the algorithm in use is IteratedLovinsStemmer 7.
As in the case of emoticons, with the use of this technique it is possible to combine features with the same meaning and reduce the entropy of the model. 3.6
Stop words are words which are ltered out in the preprocessing step. These words are, for example, pronouns, articles, etc. It is important to avoid having these words within the classi er model, because they can lead to a less accurate classi cation. 4
The data set is composed of the training and test sets.
The training sets are those provided by SemEval, with a little revision: neutral sentences are removed, in order to focus only on positive and negative ones. Furthermore, in the training set there are more positive sentences than negative ones. Excess positive ones have been eliminated, because they distort the Bayes model.
In the executed tests, the features collected have a minimum presence in the text that is greater than or equal to 5. The Ngrams used are only one-grams and bi-grams. Before starting the simulation with the test set, a 10-fold crossvalidation is carried out. In particular, we searched for the optimal length of N-grams to potentially consider as features. In Figure 2, it can be observed that accuracy nearly peaks at N-gram = 2. Longer sequences increase the complexity of the training phase, without giving a signi cant improvement of the result. Also, we analysed the total number of features to consider. This parameter does not provide a monotonic improvement to the classi er quality. Instead, it peaks out at around 1500 features.
At rst, the executed simulations compare no preprocessed le vs. basic cleaned le. As shown in Table 5, the resulting accuracy is strongly increased. Given the importance of the basic cleaner, we decided to use it in every case,
html together with another preprocessing module, in order to evaluate their contribution together.
Stemming increases the performance, because it groups words reduced to their root form. It allows many words to be selected as useful features for the classi cation phase. In fact, it modi es the weight of a feature, usually increasing it.
Stopword removal enhances the system because it removes words which are useless for the classi cation phase. As a common example, an article does not express a sentiment but it is very present in the sentences.
As a notable result, it is interesting that using a dictionary did not enhance the performance in our tests, but it increased the elaboration-time needed for cleaning raw data.
There is also an improvement in the accuracy of the classi er between the two SemEval test-sets: 2016 and 2015. However, this is only due to there being fewer of sentences in the last test-set, with a corresponding lower probability for the classi er to make mistakes.
Text preprocessing is an important phase in all relevant applications of data mining. In Sentiment Analysis, in particular, it is cited in virtually all available research works. However, few works have been speci cally dedicated to understanding the role of each one of the basic preprocessing techniques, which are often applied to textual data.
Arguably, having a more precise measure of the impact of these basic techniques can improve the knowledge of the whole data mining process. This work adopts a straightforward methodology: it basically applies each one of the most known lters, independently, to the raw data. However, given the importance of the basic cleaner, we decided to use the basic cleaner in every case, together with another single preprocessing module, and then evaluate their joint contribution.
As an interesting result, it is worth noting that using a dictionary did not enhance the performances in our tests, but it increased the elaboration-time needed for cleaning raw data. All other techniques, however, provided signi cant improvements to the classi er performances. Some of the techniques simply removed useless noise in the raw data, while others increased the relevance of some concepts, reducing similar terms and expression forms to their most basic meaning.
This research has been conducted over data which originated from Twitter. In our opinion, a similar analytical work should be performed on di erent kinds of data sets, to have a more comprehensive understanding of the di erent preprocessing lters. The decision to mix some of these lters together is often correct. However, it should be better motivated by empirical data and result evaluations for various application domains and the peculiar nature of their textual data.