This paper describes a supervised approach we have designed for the topic-based message polarity classi cation. Given a message and a topic, we aim at (i) classifying the message on a two point scale, that is positive or negative sentiment toward that topic and (ii) classifying the message on a ve-point scale, that is the message conveyed by that tweet toward the topic on a more ne-grained range. These two tasks have been proposed as subtasks of SemEval-2017 task 4. We have targeted them with the employment of IBM Watson that we leveraged to extract concepts and categories to enrich the vectorial space we have modeled to train our classi ers. We have used di erent classi ers for the two tasks on the provided training set and obtained good accuracy and F1-score values comparable to the SemEval 2017 competitors of those tasks.
Social media platforms are commonly used to share opinions and thoughts about
di erent subjects and topics in any domain. Their huge widespread and
proliferation of content has created opportunities to analyze and study opinions, how
and where emotions are generated, what the current feelings are on a certain
topic and so on. It is straightforward therefore to understand that social media
have more and more interest in identifying sentiment in document, messages or
posts. The common task is to detect whether in a given text there are positive,
negative, neutral opinions expressed, and whether these opinions are general or
focused on a certain person, product, organization or event. A lot of research
has been already performed to address this task and several variations and
extensions of it [
Cognitive computation is a recent kind of technology that is specialized in
the processing and analysis of large unstructured datasets by leveraging arti cial
intelligence, signal processing, reasoning, NLP, speech recognition and vision,
human-computer interaction, dialog and narrative generation. Cognitive
computing systems have earned a lot of attention for guring out relevant insights
from textual data such as classifying biomedical documents [
In this paper we propose a supervised approach for topic-based message
polarity classi cation formulated as follows: given a message and a topic, classify
the message on a two-point scale (Task 1) and on a ve-point scale (Task 2).
These two tasks have been proposed within the task 4: Sentiment Analysis in
Twitter of SemEval 2016 [
We used machine learning approaches to target the two tasks above and leveraged IBM Watson to extract concepts and categories from the input text and to augment the vectorial space using term frequency and TF-IDF. Training and test data consist of tweets and a given topic for each tweet. As for each topic we have several tweets, we created as many classi ers as the overall number of topics in the training set. During the prediction step for a given pair (tweet, topic), two possibilities might occur: 1. the topic was found within the training set and therefore we selected the classi er already trained on the tweets related to that topic; 2. The topic was not found in any tweets of the training set. To solve this case, we used the classi er on the closest topic to the one to predict. We leveraged the semantic features extracted by IBM Watson to nd the closest topic in the training set to the one to predict.
The performance evaluation we have carried out indicates satisfying results for the Task 1 whereas for Task 2 they su er from the low number of tweets per topic present within the training set with respect to the number of tweets in the test set.
The remainder of this paper is organized as follows. Section 2 describes background work on Sentiment Analysis techniques and how Semantics has been employed in that domain. Section 3 introduces the data we have used and how they are organized. Section 4 includes details on the method we have adopted to tackle the tasks and how Cognitive Computing has been leveraged. Section 5 shows results we have obtained and the evaluation we have carried out. Section 6 depicts concluding remarks. 2
Several initiatives (challenges [
Authors in [
The data have been obtained from SemEval7. The have been extracted from Twitter and annotated using CrowdFlower8. The datasets (training and test) for Task 1 included a tweet id, the topic, the tweet text and the tweet classi cation as positive, negative and neutral. The datasets for Task 2 (training and test) had the same structure except for the tweet classi cation that was an integer number ranging in [-2, +2]. Tables 1 and 2 show, respectively, ve records of the dataset related to Task 1 and Task 2.
Moreover, Table 3 indicates the size of training sets and test sets for the two tasks whereas Table 4 and Table 5 show some statistics of the data. 4
In order to prepare the vectorial space, we have augmented the bag of words model resulting from the tweets of the training set with two kind of semantic features extracted using IBM Watson: categories and concepts. As an example, for the third tweet of Table 1 IBM Watson has extracted as categories magic and illusion, football, podcasts and as concepts 2009, singles.
We have employed the augmentation method mentioned in [
675847244747177984 amy schumer 672827854279843840 amy schumer 662755012129529858 amy schumer 679507103346601984 amy schumer -1 -1 -2 2 @MargaretsBelly Amy Schumer is the stereotypical 1st world Laci Green feminazi.
Plus she's unfunny dani pitter I mean I get the hype around JLaw.
I may not like her but I get her hype.
I just don't understand Amy Schumer and her hype Amy Schumer at the #GQmenoftheyear2015 party in a dress we pretty much hate: https://t.co/j5HmmyM99j #GQMOTY2015 https://t.co/V8xzmPmPYX Amy Schumer is on Sky Atlantic doing one of the worst stand up sets I have ever seen.
And I've almost sat through 30 seconds of Millican. "in them to do it. Amy Schumer in EW, October amyschumer is a fucking rock star & I love her & Jesus F'ing
Christ we need more like this" #NFL #Packers our methods. In particular we have employed the vectorial space consisting of: (i) tweets only (what we refer as baseline), (ii) tweets augmented with categories, (iii) tweets augmented with concepts, (iv) and tweets augmented with categories and concepts. We performed a set of cleaning steps to the resulting bag of words which included (i) lower casing the tokens of the input tweets, categories and concepts, (ii) removing of special characters and numbers, (iii) removing of stop words taken from StanfordNLP9.
We employed machine learning classi ers and fed them with the produced vectorial spaces. In particular we used Linear Regression and Naive Bayes for the binary prediction of Task 1 where we have considered the positive/negative classes getting rid of the neutral class (as also suggested in the corresponding SemEval task). As far as the multi class classi cation of the Task 2 is concerned, we employed Decision Trees and Naive Bayes classi ers. To note that, because our data consisted of a set of tweets for each topic, we have trained a classi er for each topic in the training set feeding it with all the tweets with that topic. Both the tasks we targeted are topic-based and, therefore, given a tweet and a topic, we rst had to nd the most similar topic in the training set and then use the related classi er for the prediction step. 4.1
Since the topics in the test set are completely di erent from those in the training set, we had to choose a strategy to associate the most similar topic of the training set (and therefore pick the related classi er) with each topic in the test set. To achieve this we used the categories obtained by IBM Watson. Every tweet in the training set has di erent related categories, thus a set with all the categories for each topic has been prepared. Similarly, for each topic in the test set, we prepared a set of all the categories extracted from each tweet related to that topic. Therefore, each topic in the training set and in the test set corresponded to a vector of categories. During the prediction of a given tweet with a certain topic t, we needed to use the classi er trained on the tweets having the most similar topic to t. To nd the most similar topic in the training set to t, we counted how many categories the two lists (one corresponding to t and the other corresponding to each topic in the training set) had in common and took the one with the highest number. 5
According to SemEval, the evaluation measure for Task 1 was the average recall that we refer as AvgRec:
AvgRec =
where RP and RN refer to the recall with respect to the positive and negative
class. AvgRec ranges in [
F 1 = 2 (P P + P N ) (RP + RN )
P P + P N + RP + RN As the task is topic-based we have computed each metric individually for each topic and then we computed the average value across all the topics to obtain the nal score. Task 2 is a classi cation where we need to classify a tweet in exactly one class among those de ned in C=fhighly negative, negative, neutral, positive, highly positiveg represented in our data by f-2, -1, 0, 1, 2g. We used macro-average mean absolute error (M AEM ) de ned as:
M AEM (h; T e) = 1 jCj X
1 jCj j=1 jT ej j xi2T ej
X jh(xi) yij where yi denotes the true label of item xi, h(xi) is its prediction, T ej represents the set of test documents having cj as true class, jh(xi) yij is the distance between classes h(xi) and yi.
One bene t of the M AEM measure is that it is able to recognize major misclassi cations: for example misclassifying a highly negative tweet in highly positive is worse than misclassifying it as negative. We also used the standard mean absolute error M AE , which is de ned as:
M AE (h; T e) =
1 jT ej xi2T e
X jh(xi) yij The advantage of M AEM with respect to M AE is that it is robust to unbalanced class (as in our case) whereas the two measures are equivalent in presence of balanced datasets. Both M AEM and M AE have been computed for each topic and results averaged across all the topics to obtain one nal score.
Tables 6 and 7 show the results we obtained for our proposed Task 1 whereas Tables 8 and 9 include results for Task 2. Results for both the tasks have been obtained by using the training and test sets of the data released from SemEval and also using a 10-cross validation by merging them. In the latter case, we did not consider the topic information during the learning step and trained one single classi er that used for the test. 5.1
In this section we discuss the obtained results for the two tasks we targeted in this paper. On the one hand, the employment of the semantic features had an impact for the classi cation within Task 1. As the Tables 6 and 7 show, adding the categories to the baseline improved the overall results. The addition of concepts only does not help the classi cation process as with the categories probably because the lower number of concepts ends up adding noise in the used classi ers (Naive Bayes and Linear Regression). Results are con rmed also with the 10-cross-validation.
On the other hand, Task 2 shows important di erences between the baseline and the tweets with the semantic features as Task 1 but in the opposite direction. As Tables 8 and 9 show, adding semantic features never improves the classi cation results, indicating they act like noise. This might be justi ed given the unbalanced nature of the used dataset: typically, each topic contains more tweets for a few classes and much less for the others. This fact generate a lot of error in the classi cation task and produces poor results. Furthermore, one explanation of such a behaviour is that Task 1 only consisted of a binary classi cation whereas Task 2 consisted of the multiclass classi cation where the output class might be assigned to one of ve di erent values. Predicting ve values instead of two is much harder and, given the low number of tweets per topic, the classi ers could not be trained well enough on an appropriate dataset. 6
In this paper we have presented a supervised topic-based message polarity classi cation for two tasks proposed at SemEval. The rst task aims at classifying a tweet on a two point scale (positive or negative) toward a given topic. The second task aims at classifying a tweet on a ve-point scale. We have targeted the two tasks using a machine learning approach where the vectorial space has been created by augmenting the message (tweets) with semantic features (categories and concepts) extracted with IBM Watson, a well known cognitive computing tool. Moreover, categories and concepts have been used to calculate the distances between topics of the training set and test set in order to associate the latter to the former. Although the low number of tweets in the training set, for Task 1 we obtained good results whereas Task 2 su ered from the scarcity of training data. Obtained results showed that with few classes (Task 1), concepts and categories were important for the classi cation task. Conversely, given the strong unbalanced nature of the dataset, in Task 2 concepts and categories were not able to enrich the obtained vectorial space. To address this issue, and as next steps, we would like to further investigate the employment of semantic features extracted from other cognitive computing systems trying to combine and compare them with the results obtained using IBM Watson.
The authors gratefully acknowledge Sardinia Regional Government for the nancial support (Convenzione triennale tra la Fondazione di Sardegna e gli Atenei Sardi Regione Sardegna - L.R. 7/2007 annualita 2016 - DGR 28/21 del 17.05.201, CUP: F72F16003030002).