The majority of Twitter sentiment analysis systems implicitly assume that the class distribution is balanced while in practice it is usually skewed. We argue that Twitter opinion mining using learning methods should be addressed in the framework of imbalanced learning. In this work, we present a study of synthetic oversampling techniques for tweet-polarity classification. The experiments we conducted on three publicly available datasets show that these methods can improve the recognition of the minority class as well as the geometric mean criterion.
Micro-blogging services are communication tools that are massively used by people to instantaneously share their opinions about any kinds of topics. These opinions are of interest for companies or individuals, like politicians, as they allow them to monitor their online reputation. Twitter has been the most popular micro-blogging service with more than 500 million tweets per day in 20131. Thus, sentiment analysis of tweets2 has received a lot of attention both from academia and industry during the last years.
In this paper, we focus on tweets polarity classification using supervised learning methods. This task is challenging in several respects. Firstly, tweets are limited to 140 characters and they contain irregular lexical units and syntactic patterns. Hence, these data are noisy, sparse and high-dimensional which makes the learning process dicult. Moreover, tweets expressing an opinion about a given topic usually present a skewed polarity distribution. In this case, any classifier would be biased towards the majority class.
In order to cope with these challenges, we propose to use synthetic oversampling techniques. These procedures are designed to deal with the class imbalance issue. We show that not only they enable reducing the bias towards the majority 1 http://www.internetlivestats.com/twitter-statistics/ 2 Short informal messages in a more general perspective. class, but they also alleviate the data sparsity burden commonly encountered in text mining.
The rest of the paper is organized as follows. In section 2, we discuss some related works in order to position and motivate our proposal. In section 3, we present our approach based on three synthetic oversampling methods and two supervised learning methods. Then, in section 4, we detail the experiments we conducted on three datasets including two di↵erent languages and we discuss the obtained results as well. We conclude the paper in section 5. 2 2.1
Twitter sentiment analysis has received a growing interest starting from 2009
[
The class imbalance problem in binary classification occurs when the sizes of the
classes di↵er greatly. In this case, any classifier is biased toward the majority class
(see [
Imbalanced learning for sentiment analysis has been studied by several
researchers in di↵erent learning settings [
Tweets contain slang words and irregular expressions. Thus, linguistic analyses by conventional NLP tools often give poor performances on such texts. To circumvent these diculties, and also to deal with di↵erent languages, we rely on a vectorial representation of tweets based on a bag-of-words approach. We denote by F the resulting feature space, x 2 F is a vector representing a tweet and its coordinates are its words’ frequency. In what follows, we use P and N to designate the subsets of tweets with the minority and the majority class labels respectively (|P| < |N|).
In order to compare tweets, we use the cosine similarity function. Note that all pairwise proximity measures lie between 0 and 1 since the coordinates of vectors are non-negative. Let x be any tweet in P then its neighborhood is denoted NN(x) and it consists of the k nearest neighbors. 3.2
To face the skewed class distribution problem, one straightforward approach is to balance the training set so that |P| = |N|. Undersampling the majority class or oversampling the minority class are two possible strategies. Since the data are very sparse, undersampling the majority class is sub-optimal as we may lose meaningful examples in the learning process. Therefore, oversampling the minority class seems a better solution. In this case, synthetic oversampling creates new examples in P by taking convex combinations of existing points.
We recall three popular synthetic oversampling methods: SMOTE [
Note that y lies in the line segment joining x and x0. It is important to notice that y belongs to the subspace spanned by the union of the underlying subspaces of x and x0. Therefore, synthetic examples are less sparse than original ones.
The main di↵erences between the three oversampling methods concern the random selection of x 2 P in step 1. SMOTE assumes a uniform distribution over P whereas Borderline-SMOTE assumes a uniform distribution over B, a subset of P. B consists of tweets in P whose neighborhoods contain a majority of points in N. These items lie in subspaces where the decision boundary is prone to errors. Thereby, it is expected that oversampling in these parts of the space improves the classifier performances. Regarding ADASYN, it assumes a non uniform distribution over P. It can be seen as a smoothed version of BorderlineSMOTE: the noisier the neighborhood of x, the more synthetic points around x. In other words, the probability to select x in step 1 is proportional to the number of points of N contained in NN(x). 4 4.1
We assess the approach introduced previously on three publicly available
Twitter datasets. The first two are OMD “Obama-McCain Debate” [
Concerning the vectorial representation of tweets, we used unigrams of words and we only removed the hapax.
We give the descriptive statistics3 of these datasets below: – OMD: 1906 tweets (710 positive, 1196 negative) and 1569 features; – HCR: 1922 tweets (541 positive, 1381 negative) and 2066 features; – IW: 4519 tweets (1092 positive, 3427 negative) and 3918 features. 4.2
We experimented with two di↵erent learning models: decision trees and the l1 penalized logistic regression.
Decision trees are well-known symbolic learning techniques and o↵er the
advantages of coping with high-dimensional data as well as providing
humanreadable outputs. In this work, we used CART [
The l1 penalized logistic regression [
We use several performance criteria: overall accuracy (OA), F1-measures of the positive and negative classes (F-P and F-N respectively). OA evaluates the overall performance of a classifier but it does not properly account for the performances on P as compared to N because of the skewed distribution of class labels. Hence, we also use a popular criterion for imbalanced learning: the geometric mean (GM) of both class accuracy rates. Unlike OA, GM is independent of the class distribution (see [9, Chapter 8] for an overview of this topic). Thus we argue that GM should also be a default evaluation criterion in Twitter sentiment analysis tasks. 3 We removed tweets that were labeled as neutral since we are only concerned with polarity detection.
It is important to note that we are not interested in comparing the results of decision trees against l1 penalized logistic regression. Our purpose is rather to illustrate that synthetic oversampling can improve the performances of learning methods on Twitter imbalanced-polarity detection tasks.
We tested the two learning models on the three collections with di↵erent relatively balanced training sets. In what follows, ⌧ is a variable taking its values in {0, 1/4, 1/2, 3/4, 1} which measures how much the training set is balanced with respect to the initial distribution. In fact, ⌧ = 0 is when no oversampling was carried out and we used the initial imbalanced training set (this is our baseline); ⌧ = 1/4 means we generated b(|N| | P|)/4c positive synthetic examples; . . . ; and ⌧ = 1 means we exactly sampled |N| | P| new positive items in order to have a perfectly balanced training set. The neighborhood was set to k = 20 nearest neighbors4. The results we obtained using a 5 fold cross-validation are plotted in Figure 1 for decision trees and in Figure 2 for l1 penalized logistic regression. – For both decision tree and l1 penalized logistic regression, we note quite the same trends: oversampling generally improves the results. Indeed, All 4 We also tested with k = 10, 30 but the trends were similar and the results comparable. three sampling methods globally improve the GM measure5. Thereby, our approach allows alleviating the class imbalance problem e↵ectively. For OMD, when ⌧ = 1, the most important gains for GM measures are given by ADASYN (1st row, 4th column in the figures). Regarding HCR and IW, Borderline-SMOTE performs the best but SMOTE often provides comparable results (2nd and 3rd rows respectively and 4th column in the figures). – All three oversampling strategies generally boosts F-P values6. The minority class is thus better recognized. However, this is at the expense of a reduction of F-N values. Nonetheless, since the increasing rate of F-P is generally much larger than the decreasing rate of F-N, we note the overall increase of GM values as highlighted previously. – For all three sampling techniques, the OA measure tends to diminish as the training set is more and more balanced. In fact, since the class distribution in the test set is skewed towards N, the errors on true negative tweets have more impact on OA than the correct detection of true positive tweets. This illustrates again the fact that OA is not a criterion that properly accounts for imbalanced data. – We cannot conclude on which of the three oversampling strategies is the best. However, we can make the following remarks: • SMOTE and Borderline-SMOTE have quite the same behaviours for the HCR and IW collections. F-P measures are greater than ADASYN whereas F-N values are lower. Both methods allows a much better recognition of the minority class but in doing so they make more mistakes when detecting the majority class. • In contrast, ADASYN presents peculiar properties. The increase of GM values are lower than for the two other methods but this oversampling 5 The only exception is observed for OMD when using a fully balanced training sets (⌧ = 1) generated by Borderline-SMOTE with CART as shown in Figure 1. 6 Except the same particular case mentioned previously. technique shows more stable OA values and even better ones in some cases. For the OMD dataset specifically, this approach not only provides among the best performances for the GM criterion but it also allows improving the OA measures unlike the other methods. 5
Twitter sentiment analysis is confronted with the class imbalance problem and it is important to take this aspect into account when designing opinion mining systems based on machine learning.
A way to address this challenge is to use synthetic oversampling which aims at balancing the training set in a meaningful way. Three state-of-the-art methods have been examined in that regard. We conducted experiments on politicaltweets polarity classification using three datasets and in two di↵erent languages. The obtained results show that our proposal makes it possible to deal with the skewed class distribution issue by providing better recognition of the minority class as well as obtaining large increases of the overall geometric mean criterion.
In future work, we intend to extend our study to multiclass sentiment analysis and also to examine the use of synthetic oversampling methods in other NLP tasks as a general approach to cope with the sparsity problem.
Acknowledgment This work was partly supported by the french national projects Imagiweb ANR-2012-CORD-002-01 and Request PIA/FSN.