Twitter has brought much attention recently as a hot research topic in the domain of sentiment analysis. Training sentiment classifiers from tweets data often faces the data sparsity problem partly due to the large variety of short and irregular forms introduced to tweets because of the 140-character limit. In this work we propose using two different sets of features to alleviate the data sparseness problem. One is the semantic feature set where we extract semantically hidden concepts from tweets and then incorporate them into classifier training through interpolation. Another is the sentiment-topic feature set where we extract latent topics and the associated topic sentiment from tweets, then augment the original feature space with these sentiment-topics. Experimental results on the Stanford Twitter Sentiment Dataset show that both feature sets outperform the baseline model using unigrams only. Moreover, using semantic features rivals the previously reported best result. Using sentimenttopic features achieves 86.3% sentiment classification accuracy, which outperforms existing approaches.
Few years after the explosion of Web 2.0, microblogs and social networks are now considered as one of the most popular forms of communication. Through platforms like Twitter and Facebook, tons of information, which reflect people’s opinions and attitudes, are published and shared among users everyday. Monitoring and analysing opinions from social media provides enormous opportunities for both public and private sectors. for private sectors, it has
Twitter, which is considered now as one of the most popular
microblogging services, has attracted much attention recently as a hot
research topic in sentiment analysis. Previous work on twitter
sentiment analysis [
Figure 1 compares the word frequency statistics of the tweets data we used in our experiments and the movie review data1. X-axis shows the word frequency interval, e.g., words occur up to 10 times 1http://www.cs.cornell.edu/People/pabo/ movie-review-data/ (1-10), more than 10 times but up to 20 times (10-20), etc. Y-axis shows the percentage of words falls within certain word frequency interval. It can be observed that the tweets data are sparser than the movie review data since the former contain more infrequent words, with 93% of the words in the tweets data occurring less than 10 times (cf. 78% in the movie review data).
One possible way to alleviate data sparseness is through word
clustering such that words contributing similarly to sentiment
classification are grouped together. In this paper, we propose two
approaches to realise word clustering, one is through semantic
smoothing [
We propose a semantic interpolation method to incorporate
semantic concepts into sentiment classifier training where we interpolate
the original unigram language model in the Naïve Bayes (NB)
classifier with the generative model of words given semantic concepts.
We show on the Stanford Twitter Sentiment Data [
The rest of the paper is organised as follows. Section 2 outlines existing work on sentiment analysis with focus on twitter sentiment analysis. Section 3 describes the data used in our experiments. Section 4 presents our proposed semantic smoothing method. Section 5 describes how we incorporate sentiment-topics extracted from the JST model into sentiment classifier training. Experimental results are discussed in Section 6. Finally, we conclude our work and outline future directions in Section 7.
Much work has been done in the field of sentiment analysis. Most
of the work follows two basic approaches. The first approach
assumes that semantic orientation of a document is an averaged sum
of the semantic orientations of its words and phrases. The pioneer
work is the point-wise mutual information approach proposed in
Turney [
Twitter sentiment analysis is considered as a much harder problem than sentiment analysis on conventional text such as review documents, mainly due to the short length of tweet messages, the frequent use of informal and irregular words, and the rapid evolution of language in Twitter. Annotated tweets data are impractical to obtain. A large amount of work have been conducted on twitter sentiment analysis using noisy labels (also called distant supervision).
For example, Go et al. [
The best result of 83% was reported by MaxEnt using a
combination of unigrams and bigrams. Barbosa and Feng [
While the aforementioned approaches did not detect neutral
sentiment, Pak and Paroubek [
Speriosu et al. [
There have also been some work in exploring feature engineering to improve the performance of sentiment classification on tweets.
Agarwal et al. [
Kouloumpis et al. [
We built our training set by randomly selecting 60,000 balanced tweets from the original training set in the Stanford Twitter Sentiment Data. Since the original test set only contains a total of 359 tweets which is relatively small, we enlarge this set by manually annotating more tweets. To simplify and speed up the annotation efforts, we have built Tweenator4, a web-based sentiment annotation tool that allows users to easily assign a sentiment label to tweet messages, i.e. assign a negative, positive or neutral label to a certain tweet with regards to its contextual polarity. Using Tweenator, 12 different users have annotated additional 641 tweets from the original remaining training data. Our final test set contains 1,000 tweet messages with 527 negative and 473 positive.
It is worth mentioning that users who participated in the annotation process have reported that using the annotation interface of Tweenator, as shown in Figure 3-a, they were able to annotate 10 tweet messages in 2 to 3 minutes approximately.
Recently, we have added two new modules to Tweenator by implementing our work that will be described in Section 4. The first module (see Figure 3-b) provides a free-form sentiment detection, which allows users to detect the polarity of their textual entries. The second module is the opinionated tweet message retrieval tool (see 2http://www.cs.pitt.edu/mpqa/ 3http://twittersentiment.appspot.com/ 4http://atkmi.com/tweenator/ #MSM2012
Twitter is an open social environment where there are no restrictions on what users can tweet about. Therefore, a huge number of infrequent named entities, such as people, organization, products, etc., can be found in tweet messages. These infrequent entities make the data very sparse and hence hinder the sentiment classification performance. Nevertheless, many of these named entities are semantically related. For example, the entities “ iPad” and “ iPhone” can be mapped to the same semantic concept “ Product/Apple”. Inspired by this observation, we propose using semantic features to alleviate the sparsity problem from tweets data. We first extract named entities from tweets and map them to their corresponding semantic concepts. We then incorporate these semantic concepts into NB classifier training.
We investigated three third-party services to extract entities from tweets data, Zemanta,5 OpenCalais,6 and AlchemyAPI.7 A quick and manual comparison of a randomly selected 100 tweet messages with the extracted entities and their corresponding semantic concepts showed that AlchemyAPI performs better than the others in terms of the quality and the quantity of the extracted entities. Hence, we used AlchemyAPI for the extraction of semantic concepts in our paper.
Using AlchemyAPI, we extracted a total of 15,139 entities from the training set, which are mapped to 30 distinct concepts and extracted 329 entities from the test set, which are mapped to 18 distinct concepts. Table 1 shows the top five extracted concepts from the training data with the number of entities associated with them.
Country Organisation
4954 2815 1575 961 614 4.2 Incorporating Semantic Concepts into NB
Training The extracted semantic concepts can be incorporated into sentiment classifier training in a naive way where entities are simply replaced by their mapped semantic concepts in the tweets data. For example, all the entities such as “ iPhone”, “ iPad”, and “ iPod” are replaced by the semantic concept “ Product/Apple”. A more principled way to incorporate semantic concepts is through interpolation. Here, we propose interpolating the unigram language model with the generative model of words given semantic concepts in NB training. In NB, the assignment of a sentiment class c to a given tweet w can 5http://www.zemanta.com/ 6http://www.opencalais.com/ 7http://www.alchemyapi.com/ (b) Free-Form Sentiment Detector Interface. #MSM2012 be computed as: cˆ = arg max P (c|w)
c∈C = arg max P (c) c∈C
Y 1≤i≤Nw
P (wi|c), where Nw is the total number of words in tweet w, P (c) is the prior probability of a tweet appearing in class c, P (wi|c) is the conditional probability of word wi occurring in a tweet of class c. In multinomial NB, P (c) can be estimated by P (c) = Nc/N Where Nc is the number of tweets in class c and N is the total number of tweets. P (wi|c) can be estimated using maximum likelihood with Laplace smoothing:
P (w|c) = Pw0∈V N (w0|c) + |V |
N (w, c) + 1 Where N (w, c) is the occurrence frequency of word w in all training tweets of class c and |V | is the number of words in the vocabulary. Although using Laplace smoothing helps to prevent zero probabilities of the “unseen” words, it assigns equal prior probabilities to all of these words.
We propose a new smoothing method where we interpolate the unigram language model in NB with the generative model of words given semantic concepts. Thus, the new class model with semantic smoothing has the following formula:
Ps(w|c) =(1 − α)Pu(w|c) + α
X P (w|sj )P (sj |c)
j
Where Ps(w|c) is the unigram class model with semantic
smoothing, Pu(w|c) is the unigram class model with maximum likelihood
estimate, sj is the j-th concept of the word w, P (sj |c) is the
distribution of semantic concepts in training data of a given class and
it can computed via the maximum likelihood estimation. P (w|sj )
is the distribution of words in the training data given a concept and
it can be also computed via the maximum likelihood estimation.
Finally, the coefficientα is used to control the influence of the
semantic mapping in the new class model. By setting α to 0 the class
model becomes a unigram language model without any semantic
interpolation. On the other hand, setting α to 1 reduces the class
model to a semantic mapping model. In this work, α was
empirically set to 0.5.
5. SENTIMENT-TOPIC FEATURES
The joint sentiment-topic (JST) model [
dream bought song eat sweet short listen food e train hair love coffe itv angel love music dinner iso love wear play drink Pgoodnight shirt album yummi free dress band chicken club photo guitar tea feel miss rain exam today sad bike school e hate cry car week tiv sick girl stop tomorrow a eg cold gonna ride luck N suck talk hit suck weather bore drive final headache feel run studi Topic 5 movi show award live night mtv concert vote job hard find hate interview lost kick problem
Inspired by the above observations, grouping words under the same topic and bearing similar sentiment could potentially reduce data sparseness in twitter sentiment classification. Hence, we extract sentiment-topics from tweets data and augment them as additional features into the original feature space for NB training. Algorithm 1 shows how to perform NB training with sentiment-topics extracted from JST. The training set consists of labeled tweets, Dtrain = {(wn; cn) ∈ W × C : 1 ≤ n ≤ N train}, where W is the input space and C is a finite set of class labels. The test set contains tweets without labels, Dtest = {wnt ∈ W : 1 ≤ n ≤ N test}. A JST model is first learned from the training set and then infer sentiment-topic for each tweet in the test set. The original tweets are augmented with those sentiment-topics as shown in Step 4 of Algorithm 1, where li_zi denotes a combination of sentiment label li and topic zi for word wi. Finally, an optional feature selection step can be performed according to the information gain criteria and a classifier is then trained from the training set with the new feature representation.
In this section, we present the results obtained on the twitter sentiment data using both semantic features and sentiment-topic features and compare with the existing approaches.
The raw tweets data are very noisy. There are a large number of irregular words and non-English characters. Tweets data have some unique characteristics which can be used to reduce the feature space through the following pre-processing: #MSM2012 Algorithm 1 NB training with sentiment-topics extracted from JST.
Input: The training set Dtrain and test set Dtest Output: NB sentiment classifier 1: Train a JST model on Dtrain with the document labels discarded 2: Infer sentiment-topic from Dtest 3: for each tweet wn = (w1, w2, ..., wm) ∈ {Dtrain, Dtest} do 4: Augment tweet with sentiment-topics generated from JST, wn0 = (w1, w2, ..., wm, l1_z1, l2_z2, ..., lm_zm) 5: end for 6: Create a new training set Dtrain0 = {(wn0; cn) : 1 ≤ n ≤ N train} 7: Create a new test set Dtest0 = {wn0 : 1 ≤ n ≤ N test} 8: Perform feature selection using IG on Dtrain0 9: Return NB trained on Dtrain0
Pre-processing None Username Hashtag URLS Repeated Letters Digits Symbols All • All URL links in the corpus are replaced with the term “URL”. • Reduce the number of letters that are repeated more than twice in all words. For example the word “loooooveeee” becomes “loovee” after reduction. • Remove all Twitter hashtags which start with the # symbol, all single characters and digits, and non-alphanumeric characters.
We have tested both the NB classifier from WEKA8 and the maximum entropy (MaxEnt) model from MALLET9. Our results show that NB consistently outperforms MaxEnt. Hence, we use NB as our baseline model. Table 4 shows that with NB trained from unigrams only, the sentiment classification accuracy of 80.7% was obtained.
We extracted semantic concepts from tweets data using Alchemy API and then incorporated them into NB training by the following two simple ways. One is to replace all entities in the tweets corpus with their corresponding semantic concepts (semantic replacement). Another is to augment the original feature space with semantic concepts as additional features for NB training (semantic augmentation). With semantic replacement, the feature space shrunk substantially by nearly 20%. However, sentiment classification accuracy drops by 4% compared to the baseline as shown 8http://www.cs.waikato.ac.nz/ml/weka/ 9http://mallet.cs.umass.edu/ in Table 4. The performance degradation can be explained as the mere use of semantic concepts replacement which leads to information loss and subsequently hurts NB performance. Augmenting the original feature space with semantic concepts performs slightly better than sentiment replacement, though it still performs worse than the baseline.
With Semantic interpolation, semantic concepts were incorporated into NB training taking into account the generative probability of words given concepts. The method improves upon the baseline model and gives a sentiment classification accuracy of 84%.
Unigrams Semantic replacement Semantic augmentation Semantic interpolation Sentiment-topic features
To run JST on the tweets data, the only parameter we need to set
is the number of topics T . It is worth noting that the total
number of the sentiment-topics that will be extracted is 3 × T . For
example, when T is set to 50, there are 50 topics under each of
positive, negative and neutral sentiment labels. Hence the total
number of sentiment-topic features is 150. We augment the original
bag-of-words representation of the tweet messages by the extracted
sentiment-topics. Figure 4 shows the classification accuracy of NB
trained from the augmented features by varying the number of
topics from 1 to 65. The initial sentiment classification accuracy is
81.1% with topic number 1. Increasing the number of topics leads
to the increase of classification accuracy with the peak value of
82.3% being reached at topic number 50. Further increasing topic
numbers degrades the classifier performance.
6.4 Comparison with Existing Approaches
In order to compare our proposed methods with the existing
approaches, we also conducted experiments on the original Stanford
Twitter Sentiment test set which consists of 177 negative and 182
positive tweets. The results are shown in Table 5. The sentiment
classification accuracy of 83% reported in [
It can be seen from Table 5 that sentiment replacement performs worse than the baseline. Sentiment augmentation does not result in the significant decrease of the classification accuracy, though it does not lead to the improved performance either. Our semantic interpolation method rivals the best result reported on the Stanford Twitter Sentiment test set. Using the sentiment-topic features, we achieved 86.3% sentiment classification accuracy, which outperforms the existing approaches.
Unigrams Semantic replacement Semantic augmentation Semantic interpolation Sentiment-topic features (Go et al., 2009) (Speriosu et al., 2011)
We have explored incorporating semantic features and sentimenttopic features for twitter sentiment classification. While simplesemantic replacement or augmentation does not lead to the improvement of sentiment classification performance, sentiment interpolation improves upon the baseline NB model trained on unigrams only by 3%. Augmenting feature space with sentiment-topics generated from JST also results in the increase of sentiment classification accuracy compared to the baseline. On the original Stanford Twitter Sentiment test set, NB classifiers learned from sentimenttopic features outperform the existing approaches.
We have a somewhat contradictory observation here. Using sentimenttopic features performs worse than using semantic features on the test set comprising of 1000 tweets. But the reverse is observed on the original Stanford Twitter Sentiment test set with 359 tweets. We therefore conducted further experiments to compare these two approaches.
#MSM2012 We performed feature selection using information gain (IG) on the training set. We calculated the IG value for each feature and sorted them in descending order based on IG. Using each distinct IG value as a threshold, we ended up with different sets of features to train a classifier. Figure 5 shows the sentiment classification accuracy on the 1000-tweet test set versus different number of features. It can be observed that there is an abrupt change in x-axis from around 5600 features jumping to over 30,000 features. Using sentiment-topic features consistently performs better than using semantic features. With as few as 500 features, augmenting the original feature space with sentiment-topics already achieves 80.2% accuracy. Although with all the features included, NB trained with semantic features performs better than that with sentiment-topic features, we can still draw a conclusion that sentiment-topic features should be preferred over semantic features for the sentiment classification task since it gives much better results with far less features. 7. CONCLUSIONS AND FUTURE WORK Twitter is an open social environment where users can tweet about different topics within the 140-character limit. This poses a significant challenge to Twitter sentiment analysis since tweets data are often noisy and contain a large number of irregular words and nonEnglish symbols and characters. Pre-processing by filtering some of the non-standard English words leads to a significant reduction of the original feature space by nearly 61.0% on the Twitter sentiment data. Nevertheless, the pre-processed tweets data still contain a large number of rare words.
In this paper, we have proposed two sets of features to alleviate the data sparsity problem in Twitter sentiment classification, semantic features and sentiment-topic features. Our experimental results on the Twitter sentiment data show that while both methods improve upon the baseline Naïve Bayes model trained from unigram features only, using sentiment-topic features gives much better results than using semantic features with less features.
Compared to the existing approaches to twitter sentiment analysis which either rely on sophisticated feature engineering or complicated learning procedure, our approaches are much more simple and straightforward and yet attain comparable performance. There are a few possible directions we would like to explore as future work. First, in the semantic method all entities where simply replaced by the associated semantic concepts. It is worth to perform a selective statistical replacement, which is determined based on the contribution of each concept towards making a better classification decision. Second, sentiment-topics generated by JST model were simply augmented into the original feature space of tweets data. It could lead to better performance by attaching a weight to each extracted sentiment-topic feature in order to control the impact of the newly added features. Finally, the performance of the NB classifiers learned from semantic features depends on the quality of the entity extraction process and entity-concept mapping method. It is worth to investigate a filtering method which can automatically filter out low-confidence semantic concepts.
Acknowledgement This work is partially funded by the EU project ROBUST (grant number 257859).