Microblogging websites like Twitter complement traditional news agencies and have become important sources of information about news events. In particular, aggregated sentiment values from Twitter news messages may tell us about the overall popularity of news entities or people's general perception of news events or entities. On the basis of a Norwegian Twitter news dataset we examine how the sentiments of Twitter news messages can be extracted using Naive Bayes, Support Vector Machines and Maximum Entropy. Our analysis also includes the use of linguistic features and lexical sentiments from SentiWordNet in an attempt to improve the accuracy of the techniques. The results show that there is some gain in including part-of-speech features or predened sentiments from SentiWordNet. Support Vector Machines has the highest accuracy for both subjectivity classication and polarity classication, though the dierences are small and all techniques' performances increase steadily with the size of the dataset. Moreover, our work demonstrates that sudden changes of news entity sentiments tend to be attributed to concrete entity-relevant news events.
Over the last few years Twitter has become an important source of information about unfolding news events that have not yet been properly picked up by news agencies, or where news reporting is dicult or unreliable for some reason. Any owner of a Twitter account may post small messages - tweets - of up to 140 characters that can be widely distributed and read over the internet. In spite of the brevity of these messages, Twitter has been extensively used to report or comment on incidents all over the world, and Twitter now has more than 300 million active users and an excess of 500 million tweets per day.
These tweets may be posted by users that are present at the event and that have no particular experience with traditional news reporting. As a news source Twitter complement the publications of traditional news agencies in several interesting ways: 1) the news may be quickly and directly reported as the event takes place, 2) the news are often reported by people that are physically present or in other ways have a direct link to the event, 3) the aggregation of tweets from many users cancels out individual misconceptions and presents a possibly more reliable collective perception of the event, and 4) Twitter users provide other perspectives than professional journalists and may serve as a corrective to the news agencies that are driven by increasing time pressure and declining revenues.
There are however limitations with Twitter that renders it somewhat unsuitable as a general news source:
The 140-character limitation makes it unfeasible to explain matters in sucient detail.
The lack of any other structural elements than hashtags and user references introduce ambiguities that are dicult to handle manually and very challenging computationally.
Individual tweets may be wrong, contradictory, incomplete or misleading. There are too many tweets to read and no satisfactory way of selecting the most appropriate ones.
Even though individual tweets should not necessarily be trusted, the message formed by collective streams of news tweets carry more weight and may expose other types of information than what is conveyed in reports from conventional news channels. Since people use Twitter for posting opinions on a variety of news topics and express their attitudes towards products or people on a daily basis, it seems interesting to aggregate messages and try to extract a sense of general sentiment over time for particular news entities.
In the SmartMedia project at NTNU in Norway we are developing a news
aggregator in collaboration with one of the largest media houses in Norway [
We have in our project used and compared Naive Bayes, Maximum Entropy
and Support Vector Machine for sentiment analysis of Norwegian news tweets.
An annotated Norwegian data set has been employed, and a variety of feature
sets have been compared for each technique. An important part of the work
has been the extraction of lexical words - or concepts - from the news text
that can be associated with an a priori sentiment from a sentiment ontology
like SentiWordNet [
The rest of paper is structured as follows. In Section 2 we discuss the use of Twitter for posting information about news events. We assess the overall problem of extracting sentiments from Twitter and also present the Twitter dataset that is used in our work. Section 3 introduces related work on news sentiment analysis, with a particular emphasis on the use of Naive Bayes, Support Vector Machines and Maximum Entropy. The whole sentiment analysis process is explained in Section 4. This includes the enrichment of the feature set with part-of-speech tags and predened concept sentiments from SentiWordNet, and the conguration of the machine learning techniques for subjectivity classication and polarity classication. Whereas the overall results are presented and discussed in Section 5, the application of entity sentiments over time is briey discussed in Section 6. The conclusions are given in Section 7. 2
Twitter, like many other social media platforms, allows people to express and
distribute their views across geographical, national and social borders. The service
was founded in 2006 and is today one of the largest microblogging services
available. Apart from making it easier for people to communicate, these social media
networks collect large amounts of data that can be aggregated and analyzed
to identify for example breaking news as they are emerging. As a result social
media networks have provided valuable information in real-time about crisis
situations such as earthquakes and tsunamis [
Often, though, Twitter is used to express very personal attitudes or opinions
about products, companies or people. The tweet in Figure 1 shows an example
of this. The user is Erna Solberg, the prime minister of Norway, and she is
thanking the previous prime minister for winning a prize for the best tweet in
Norway in 2015: ’I would also like to thank you, @jensstoltenberg. If you had not
lost your herring recipe, I would probably not have won this prize. #smd2014.’
@jensstoltenberg is a reference to a particular user Jens Stoltenberg, who is
the previous prime minister of Norway. The hashtag #smd2014 refers to the
Social Media Days conference in Norway in 2014. A tweet may also contain
emoticons, like small smiley faces, but can normally not exceed 140 characters
in total. In comparison, the average non-nance news article in Norwegian online
newspapers is of about 220 words [
Sentiments of Tweets
Sentiments may be associated with individual words, phrases, sentences,
paragraphs or documents. They express opinions of some entities in terms of positive,
negative or neutral attitudes towards the entities. The entities may be like
people or products, though they may also be components or aspects of higher-level
entities. For example, the objective of a sentiment analysis task may be to assess
the sentiments towards a company like Sony (brand reputation), but it may also
be to extract people’s opinions of Sony’s mobile phones or even people’s
perception of the battery life of these phones. In general, [
A quadruple, (g,s,h,t), where g is the sentiment target, s is the sentiment about the target, h is the sentiment holder, and t is the time when the opinion was expressed.
Take a look at Figure 1. Analyzing the sentiments of this tweet, we see that Erna Solberg is the sentiment holder and the opinion was stated on February 5, 2014. It is more complicated to identify the target and estimate the sentiment about the target. The text is generally positive, as she is thanking a political opponent and seems to be happy to win an award. It would be tempting to conclude that the award #smd2014 is the target, but we need to keep in mind that this tweet is posted in a political context, and she is having fun with teasing the previous prime minister (Jens Stoltenberg) that had to move out of the prime minister residence and lost his recipe in the process. She is happy about the election, which is not directly discussed in the tweet, and this award is just mentioned to make the posting a bit childish and funny.
Analyzing the sentiments of news entities from Twitter is notoriously problematic for several reasons. In the rst place, it is often dicult to identify the entity that forms the topic of a particular tweet. As seen above, the entity may not be directly mentioned in the text, or it is referred to indirectly by means of other entities that are somehow related. A second issue is the shortage of sentiment-carrying adjectives and adverbs in Tweets. Adjectives are normally very useful, as their sentiments do not change much from one domain to another and can be retrieved from sentiment lexica to calculate aggregated sentiments. Without sucient adjectives and adverbs, you are left with analyzing contextdependent sentiment values of phrases, which diminishes the value of sentiment lexica in the analysis. Due to the nature of Twitter users, there may also be deliberate ambiguity or irony in tweets that aect the aggregation of sentiment values. 2.2
Norwegian News Data set from Twitter In this work we have built a data set of Norwegian news tweets over a period of 30 days. To ensure that we had enough tweets for particular news entities, we collected the tweets using three separate strategies:
Collect news tweets completely at random (33% of data set) Collect news tweets about the football club Rosenborg (31% of data set) Collect news tweets about prime minister Erna Solberg (36% of data set) A summary of the total data set is given in Table 1. It contains 1847 Norwegian tweets, with an average of 16.1 words per tweet. There are a few emoticons like smiley faces in the dataset, but surprisingly few. The tweets were posted by 1,312 users, who themselves refer to 1,844 users in the texts. Simple word correction was performed on the tweets, as many users were deliberately using improper spellings for stylistic purposes. A parts-of-speech analysis reveals that the average tweet of the dataset contained 1.32 adjectives, 0.75 adverbs, 5.64 nouns and 2.38 verbs.
A group of three annotators were brought in for the manual annotation process. The annotators labeled each tweet as positive, negative or neutral, and each tweet was annotated by two people independently to ensure that the tweets’ sentiments had been correctly understood and annotated. To calculate the reliability of the dataset, the joint probability of agreement and Cohen’s Kappa were calculated. The equation for Cohen’s Kappa is =
where p0 is the relative observed agreement and pe is the hypothetical
probability of chance agreement when the observed data is used to calculate the
probabilities of each annotator randomly saying each category. The overall agreement
for the dataset was 83.9%, with a Kappa value of 0.73, which means that there
is moderate and acceptable agreement among the annotators [
A sentiment analysis system’s objective is to extract the sentiment of a target,
as dened above, on the basis of some textual resource. The task is normally
handled as a natural language processing task at dierent levels of granularity,
and we usually distinguish between unsupervised and supervised approaches [
The highest granularity level is the document level. At this level, one is
concerned with determining the sentiment of each document as a whole [
With respect to sentiment analysis of news content the focus has been on
longer texts, like online nance news [
Sentiment analysis is normally conducted following a two step process. First,
you identify a text to be either objective or subjective. Subsequently, you take
the subjective tweets and determine their polarity, i.e. assess whether they are
negative or positive [
When using machine learning techniques for text classication, feature engineering is an important part of it. The features of a machine learning classier are a selected subset of the measurable properties that dene the documents in the corpus. Selection of the feature set is often performed as a combination of empirical selection by a domain expert and automated methods. The set of feature values for a given document is usually called the feature vector of the document.
In the following we will describe Naive Bayes, Support Vector Machines and Max Entropy in some more detail, as these machine learning techniques are used in our own experiment. 3.1
Naive Bayes
Naive Bayes(NB) is a fast and versatile classication algorithm that is widely
used in supervised text classication systems, though it is often outperformed
by more sophisticated classiers like Support Vector Machines (SVM) [
P (A) P (BjA) P (AjB) =
P (B)
In short, this theorem enables a classier to calculate the posterior probability of B given A, using prior probabilities. In a NB classier for tweets, the formula can be reformulated as follows: (2) P (cpj!dj ) =
P (cp)
P (!dj jcp)
P (!dj ) where P (dj ) is the probability that a randomly selected tweet will be represented by dj , and P (cp) is the probability that a randomly selected tweet belongs to class cp. The classication function then is to nd the class with the largest probability function given by the product of all the feature probabilities, given their class labels. This functionality is described by the equation below. n classif y(f1; :::; fn) = argmaxp(C = c) Y p(Fi = fijC = c) i=1
The equation above shows the intuitive nature of Naive Bayes classiers. In essence, we need to train our classier by counting all the features and which classes they appear in, and use these frequencies to compute their probabilities. When classifying a tweet, we select the class which is given the highest product of the features given by the target feature vector. (4) 3.2
Support Vector Machines
Support Vector Machines(SVM) is a relatively new technique for text
classication and was rst used for this purpose by Joachims in 1999 [
The central idea in SVM is to nd the support vectors which maximize the space - the decision surface - between the two classes, i.e. nding the optimal separation between the features representing the two classes. The two support vectors are dened by the documents that lie closest to the decision surface.
The task of training an SVM classier can be formulated as an optimization
problem of nding the optimal hyperplane. Baeza-Yates & Ribeiro-Neto state
this optimization problem as follows [
Let Hw be a hyperplane that separates all documents in class ca from all documents in cb. Let ma be the distance of Hw to the closest document in class ca and let mb be the distance of Hw to the closes document in class cb, such that ma + mb = m. The distance m is the margin of the SVM. The decision hyperplane Hw maximises the margin m.
When the optimized decision surface has been calculated, any future instance presented to the classier is evaluated using their position in the space as represented by the features of this instance. The instance’s position relative to the separation between the classes determines which class should be linked to the new instance. 3.3
Maximum Entropy
A Maximum Entropy(MaxEnt) classier is a conditional probabilistic classier.
Implementations of it use logistic regression in order to nd the probability
distribution with the largest entropy, which - given by the Theory of Maximum
Entropy [
Unlike the NB classier, MaxEnt assumes no conditional independence for
the features. This means that MaxEnt handles feature overlap better than the
NB classiers [
[
P (cjd; ) =
exp[Pi ifi(c; d)]
Pc0 exp[Pi ifi(c; d)]
where c is the class and d is the tweet. The numerical operations of the task of
optimizing these lambdas are complex and and often lead to non-trivial and
timeconsuming implementations. For text classication tasks, MaxEnt classiers have
been shown to have an accuracy performance comparable to SVM [
Twitter Sentiment Analysis Approaches
Like in our work [
A similar distant learning paradigm is adopted in the work of [
Another interesting approach to sentiment analysis with Twitter is published
by [
Our approach is in many ways similar to [
Given a news tweet x, the task of our sentiment analysis system is to determine whether x expresses a positive, negative, or neutral opinion. The system is split into two separate stages, subjective classication and sentiment classication: The subjective classication component rst decides if the tweet contains a sentiment or not. If there is a sentiment represented, we call the tweet subjective and it is sent to the sentiment classication component for further analysis. Otherwise the analysis is terminated and the tweet is labeled neutral.
The sentiment classication component’s task is to categorize subjective tweets as either positive or negative.
The overall classication process is illustrated in Figure 2. Three machine learning techniques are used and evaluated for both classiers. The tweets themselves are represented as sets of features that refer to both word properties and sentence properties of the tweets. Three dierent feature sets are tested for each machine learning techniques, giving us a total of nine runs for the classiers.
Conguration of Machine Learning Techniques The machine learning techniques were extensively tested with dierent parameter values before the actual evaluation was carried out. Figure 3 lists the relevant parameters with the associated value ranges for each parameter. Two of the parameters were used for text vectorization: the range of N-grams used as features, and the Max document frequency for using the N-grams as features. Three parameters were for TF-IDF vectorizing: Use IDF, Smooth IDF, and Sublinear TF, all three of them boolean values. Finally, four algorithm-specic parameters were used: The Alpha parameter of the NB classier, which is the Laplace/Lidstone smoothing weight, the C parameter in the SVM, which inuences the margin of the SVM hyperplane, and lastly the MaxEnt-specic parameters C and penalty.
The best parameter sets for the two classication tasks and the three machine learning techniques are shown in Figure 4. Even though there are few dierences for the two classication tasks, the small deviations are important to the nal outcome of the classiers. Consequently, the rest of the experiment was conducted using the parameter values from Figure 4. 4.2
Syntactic and Semantic Enrichment of Feature Sets For each of the classication tasks, three dierent feature sets were dened and evaluated. The simplest feature sets, SA for subjective classication and PA for sentiment classication, contain weighted representations of the word tokens of the tweets, without any additional features at the sentence level.
A POS tagger from the TypeCraft project was used to tag the tweets with parts-of-speech information. This gives us additional information about syntactic and morphological properties of the tweets, e.g. the number of adjectives in a sentence or the use of negations. Feature sets SB and PB were enriched with such syntactic sentence level features, as earlier analyses suggested that there is some correlation between frequency of parts-of-speeches and tweet polarity.
An important part of the experiment was to assess the value of combining standard classiers with semantically enriched feature sets. The general idea was to associate identied entities in tweets with concepts (synsets) in WordNet. As WordNet does not exist for Norwegian, this extraction of concepts involved using Bing to translate Norwegian entities into their English counterparts. Having identied the relevant concepts, we used SentiWordNet to retrieve standard sentiments of the concepts.
SentiWordNet is an open sentiment lexicon, in which each synset of Wordnet
is associated to three numerical scores Obj(s), Pos(s) and Neg(s), describing
how objective, positive and negative the synset terms are [
Good#1 synset has an objective score of 0.25, a positive score of 0.75 and a negative score of 0.00. Adding together such scores for all identied concepts of a tweet, we get aggregated scores of the tweet’s subjectivity and polarity.
The semantically enriched feature sets SC and PC include features that reect the generation of aggregated sentiment scores from SentiWordNet concepts. The exact features used in the six sets are listed in Figure 5 and 6. 5
The annotated dataset contained a total of 1847 tweets about Norwegian news entities. There were 9 experimental runs with 10-fold cross validation for each classication task, one for each combination of machine learning technique and dataset. In addition we tested the two classication tasks on datasets of dierent sizes to verify their dependence on large-scale training data. Calculating the quality of the sentiments we used the notions of Accuracy and F1 with the following formulas:
Accuracy =
Tp + Tn
Tp + Fp + Tn + Fn F 1 = 2 precision recall precision + recall (6) (7) where Tp is the number of true positives (actual positive tweet estimated to be positive by classier), Tn is the number of true negatives, Fp is the number of false positives, and Fn is the number of false negatives. The results of the subjective classication component are summed up in Figure 7. As we can see, the F1 values are rather similar across machine learning techniques and datasets for the subjective classication tasks. The Support Vector Machine approach has slightly better results than the other two techniques, with an average F1 score of 0.66 compared to 0.58 for Naive Bayes and 0.61 for Max Entropy. The performance of the sentiment classication task is somewhat better than for subjective classication, as indicated in Figure 8. Again the SVM approach has the highest scores, but all three techniques have F1 scores above 0.7 when SentiWordNet has been used to enrich the feature sets.
It is interesting to analyze the contribution of semantics in some more detail. Whereas the smallest feature sets, SA and PA, only consist of word tokens, the semantically enriched datasets of SC and PC contain information that is calculated from looking up concepts in SentiWordNet. If we compare the results from SA/PA with SC/PC, we can estimate the contribution of semantics in our sentiment analysis system. Figure 9 tells us the changes of F1 values when the simplest dataset is replaced with the semantically enriched dataset, while keeping everything else unchanged. Surprisingly, the addition of semantic features has a negative contribution on SVM and Naive Bayes in the subjective classication task. For the sentiment classication task the use of SentiWordNet improves the F1 scores for SVM by 11.4% and for Max Entropy by 11.9%. The improvement is less for Naive Bayes, though all techniques display a signicantly higher F1 score when semantic features are introduced.
It is also worth noting how the F1 scores improve with the size of the datasets. Figures 10, 11 and 12 show the F1 scores as a function of dataset size for Naive Bayes, SVM and Max Entropy. The scores improve very fast as the dataset is still below 350-400 tweets. With a dataset size between 400 and 1847 tweets the scores improve at a rather slow but steady pace, though they do not seem to have reached their maximum level when the full dataset is employed. Probably we would get even higher scores for all three machine learning techniques if we had a larger dataset available.
Estimating the sentiments of news tweets in general is not necessarily a very useful news service. The tweets span all kinds of topics, and the aggregated sentiment values combine the sentiments of events that probably have very little to do with each other. The sentiments may possibly reveal something about people’s general attitudes or outlook, but sentiments make more sense when they are attributed to a particular target or aspect of this target. Moreover, the absolute sentiment values are of limited value to outsiders, as it is dicult to fully comprehend what a numerical value of sentiment actually mean. Twitter sentiment analysis becomes more useful when it is done at the level of entities for comparative analyses or trend analyses. Below we have extracted the news tweets about Erna Solberg, the Norwegian prime minister, from our initial data set. The subset about Erne Solberg consists of 662 tweets with a total of 10,974 tokens. A manual inspection shows that 307 tweets (46.4%) were neutral, 110 tweets (16.6%) were positive and 245 tweets (37.0%) were negative.
Figure 13 shows the aggregated subjectivity for Erna Solberg tweets in a period spanning 19 non-consecutive days from 20th of September to 28th of October 2014. The red line is our estimated SVM sentiments for Erna Solberg over a period of little more than a month. The real sentiments, as indicated by the annotators, are given by the blue Target line. The grey bars in the background show the actual frequencies of tweets mentioning Erna Solberg. Interestingly, the share of subjective tweets to total number of tweets is fairly constant over time, though there are particular periods with a substantially higher share of subjective tweets.
Figure 14 shows the polarity dierences from the Erna Solberg dataset during the same time period. The red Predictions line shows the aggregated dierences between all positive tweets and all negative tweets per day. When the line is above the grey dotted Neutral line, it means that there are more positive than negative tweets on that particular day. The grey bars in the background show the total number of subjective tweets about Erna Solberg from day to day.
Interestingly, the polarity dierences do not follow the pattern of subjective tweets per day. Clearly, there are periods in which people are strongly negative or strongly positive to the actions or statements of the Norwegian prime minister. The sentiments may change dramatically over just a few days, even though the number of subjective tweets does not change much. A closer look at the news in this time interval reveals some suspicious correlations. Around 21-22 September the positive sentiments of Erna Solberg come at a time when it is announced that she will take part in the World Climate Summit. Similarly, the opinions of the prime minister turn sour when it is revealed that the government referred wrongly to some research results, and even more when Solberg had to go public and defend cuts in well-fare. 7
This paper describes a Twitter sentiment analysis component that is developed as part of NTNU’s SmartMedia program. Three dierent classiers, SVM, Naive Bayes and Max Entropy, have been implemented and evaluated as part of this work. The component has been tested on a manually annotated Norwegian news dataset from Twitter. Additional features from lexical resources and sentiment ontologies have been included to examine the contribution of deeper syntactic or semantic analyses of text.
The results suggest that the three approaches are not very dierent in terms of precision and dependence on data set size, but the choice of feature set is important. In total SVM had the highest precision of the three in sentiment classication and was substantially better for very small data sets or poor feature sets. Maximum Entropy was ecient in subjectivity classication when more informative feature sets were available.
Adding semantic features with the help of SentiWordNet leads to a substantial improvement of the sentiment classier, but not of the subjective classier. Both SVM and Max Entropy see an improvement of more than 11% when SentiWordNet is consulted to enrich the feature sets for the sentiment classier. It is dicult to assess why we do not see a similar improvement in subjective classication, though.
We also analyzed to what extent variations in sentiments coincide with important events dealing with these entities. The analysis reveals that sudden changes of sentiments can usually be attributed to concrete news events that are heavily reported in the media. There is, however, little correlation between the share of subjective tweets and particular news event. It seems that people do not get more emotional when major events take place, but the polarity of their emotional tweets seem to correlate well with their opinions of the underlying news.