English. In this paper we present an approach for joy, anger and neutral emotions detection based on an emerging sentiment language model. We propose an approach that can detect specific emotions from positive, neutral and negative sentiments and which favors the tweets that occur at recent sentiment spikes. Our results suggest that our approach can effectively detect joy, neutral and anger emotions and that it performs better compared to the baselines.
Recent years have seen the emergence of social
media that enable people to share their thoughts
and opinions in an easy and fast way. Opinions
posted on social media are very useful to
understand what people think and how they feel about a
specific entity (e.g. a product, a person, a company
etc.). For example, companies can mine users’
opinions on a product that has just been released
to understand if the users are satisfied or not and
act accordingly. Therefore, the automatic
detection of emotions and sentiments from text has
attracted a lot of research interest
Although sentiment and emotion analysis share
some similarities, they are two different
problems
In addition, public sentiment towards a specific
entity changes over time and in some cases
sentiment spikes may occur. Sentiment spikes occur
when a large amount of documents of a specific
sentiment is posted
In this paper, we focus on Twitter and we
propose the emerging Sentiment Language Model
(emerging-SLM) approach which favors tweets
that occur at recent sentiment spikes with the aim
to predict joy, anger and neutral emotions from
positive, negative and neutral sentiments
respectively. We test our approach on a collection of
tweets that spans over nine days and we show
that the emerging-SLM performs better compared
to both state-of-art Sentiment Language Model
(SLM) and to a random Sentiment Language
Model (random-SLM).
Sentiment and emotion analysis have both
attracted much research attention
Sentiment analysis has attracted a tremendous
research attention over the last years. The
proposed approaches can be roughly classified as
learning and lexicon based. The lexicon based
approaches are typically unsupervised and use lists
of words (e.g. good,bad) whose presence
implies a specific sentiment polarity
With regards to emotion analysis,
Mohammad (2012) considered hashtags that refer to an
emotion (e.g., #anger, #surprise) to create a
collection for emotion analysis and showed that these
hashtag annotations matched with the annotations
of trained judges. Roberts et al. (2012) extended
the list of the six Ekman’s basic emotions (joy,
anger, fear, sadness, surprise, disgust)
In general, language models have been used
for text classification problems
Language Models (LMs) that are widely used in
Information Retrieval (IR) and Natural Language
Processing (NLP) fields assign probabilities to
sequences of words
More formally, let +, , be the LMs for the positive, neutral and negative classes respectively. Given a test tweet d we can detect its emotion class (joy, anger, neutral) c0 as:
jdj p(djc0) = Y p(tij c)
i=1 where jdj is the number of words in tweet d and p(tij c) is a multinomial distribution estimated from the LM of class c (positive, negative, neutral).
To estimate the distributions we use the Maximum Likelihood Estimate (MLE) which computes the probabilities as follows: where n(t; c) is the number of times that the term t appears in the collection of documents of class c and jVcj is the size of the vocabulary of class c.
The emerging-SLM combines two different LMs to estimate the probabilities of the terms. The first LM is based on all the tweets of the collection excluding those that occur at recent sentiment spikes whereas the second is based on tweets that occurred at those recent sentiment spikes. Formally, the distributions of the terms using the emerging-SLM are estimated as: pglobal(tj c)+(1 ) pburst(tj c) where c is the LM for the class c, pburst(tj c) is the probability of that term t appear in the recent sentiment spikes of the class c, pglobal(tj c) is the probability of that term t appear in the class c and is the parameter that determines the importance of each LM for the final estimation. Here we should note that pglobal(tj c) is calculated after we excluded the tweets that occurred at sentiment spikes.
One common issue with the LMs is that they
assign zero probabilities to terms that do not appear
in the training data. To overcome this problem,
we apply Jelinek-Mercer smoothing that assigns
nonzero probabilities to unseen terms
To detect the sentiment spikes, we measure the evolution of each sentiment as rt;s = Nt;s=Nt where Nt;s is the number of documents that express the sentiment s posted at time t and Nt is the total number of documents posted at time t. Figure 1 shows an example of negative spikes that occurred while tracking the sentiment towards Michelle Obama. 4
In this section we describe the experimental details of our study that include the description of the dataset, the baselines we used and the experimental settings. 4.1
Dataset Our collection contains 25,588 tweets about Michelle Obama and spans from June 25, 2015
Time to July 2, 2015. To annotate the collection, we used the Crowdflower platform1. Tweets were annotated with regards to sentiment and emotion. For sentiments, annotators could choose among fpositive, no sentiment, negativeg whereas for emotions they could choose among fanger, fear, sadness, disgust, surprise, happiness, no emotiong. Each tweet was annotated by three different workers.
To optimize the annotation process and obtain
more labels we applied a type of distant
supervision, which is a popular technique for
obtaining more labels for the data
For all the results reported to this study, we used 429 tweets as a test set which were posted on July 2, 2015. We kept the test and training data always separated. We used two different baselines to compare the performance of our approach. The first baseline (SLM) is based on sentiment language models and was built from all the data without favoring tweets that occurred at spikes (i.e. = 1:0). The second baseline is the random-SLM approach. In this case, instead of using tweets from recent sentiment spikes, we randomly chose tweets from the whole collection. To build the random LM we select as many tweets as those used to build the LM of sen1https://www.crowdflower.com/ timent spikes. To evaluate the statistical significance of differences we used the McNemar test. For pre-processing, we removed URLs, mentions, punctuation and the entity-related terms Michelle and Obama. For the experiments we used only unigrams. To overcome the problem of assigning zero probabilities to unseen terms we used JelinekMercer smoothing with = 0:1.
To model the evolution of sentiment and detect any sentiment spikes we split the data hourly. In addition, we defined temporal bins with the size of 8 hours. For the emerging-SLM we detected all the sentiment spikes that occurred in the last two days. To detect the spikes, we used the peakutils2 package setting the threshold to 0.8.
Finally, to tune the parameter, we used crossvalidation on a rolling basis. Following this approach, we used data published on the first temporal bin as training and data of the second temporal bin as test. Next, data from the first and second temporal bins were used for training and data from the third temporal bin as test and so on. In other words, when we set the number of bins to 9, it means that we were using 3 days as training data (i.e. 8 hours * 9 bins = 72 hours) and one bin as test data. The test bin was always the adjacent temporal bin. The first setup included 3 days, since we wanted to have enough data to build the SLMs. After this process we estimated the best parameters using the average performance. 5
Figure 2 shows the performance with regards to the F1-measure for the task of emotion detection using the emerging-SLM on the training data for the different parameters of . We show the results for six different temporal bins for reasons of clarity. From this figure, we observe that there is a performance improvement as the parameter increases.
Table 1 shows the performance with regards to F1-measure for the task of emotion detection using the emerging-SLM, the SLM and the randomSLM approaches. From the results we observe that the emerging-SLM performs better compared to SLM and random-SLM for all the temporal bins. Also, most of the differences are significant. These results are very important because 2https://bitbucket.org/lucashnegri/peakutils they show that favoring tweets that have occurred at recent sentiment spikes is very useful. Also, the improvement over the random-SLM validates further this assumption. The results are also shown on Figure 3 for an easier comparison.
Emerging-SLM SLM Random-SLM In this paper, we proposed the emerging-SLM approach to detect joy, neutral and anger emotions from positive, neutral and negative sentiments respectively. Emerging-SLM favors tweets that occur at recent sentiment spikes. The results showed that our approach performs better compared to both SLM and random-SLM and can be effectively applied to detect specific emotions.
In future we plan to explore if there is any effect of the temporal bins size on the emotion detection performance and if sentiment language models can be used to detect also other emotions such as fear and surprise.
The work of the first author was partially funded by the Swiss National Science Foundation (SNSF) under the project OpiTrack.
The work of the last author was in the framework of the Spanish MINECO research project SomEMBED (TIN2015-71147-C2-1-P).