The remarkable ability to understand the opinion of a user about a speci c topic of discussion allows intelligent systems to provide more speci c and personalized suggestions especially when no other information is available. The strategies for opinion mining, also known as sentiment analysis, are in last years topic of in-depth studies. In this work, we present an approach of text mining for detecting the topic of discussion for textual contents and the emotion that the writer feels while writing it. Conversely to the classic strategies of sentiment analysis, we enrich the standard polarity prediction task with more ne-grained information about user's emotion. By using this information, the nal behavior of the personalized system could be designed by taking into account the view about the topic of the speci c user. For performing this task, we adopted a hybrid approach which uses both lexicons and semantic representation of sentences for the operation of aspect classication. Training data for the aspects detection module have been extracted from already categorized last year world news. The emotional labeling approach is, instead, based on the posts left by users on Facebook, which have been annotated using the emoticon encountered. The evaluation has been conducted on a dataset of tweets opportunely collected using hash-tags which refer both to the topic of discussion and the emotional opinion.
The high di usion of many social platforms has made possible the collection of
information about di erent aspects of the user's life. In this scenario, starting
from user-generated contents, the system should be able to detect the topic of
the discussion and the opinion of the user about it. This operation, recently,
has become popular with the name of opinion mining and sentiment analysis. A
prominent area of application is the analysis of e-commerce user reviews.
Sentiment analysis for the construction of a holistic user pro le can take advantages
of a more ne-grained emotional annotation of relevant aspects of a topic. In a
general domain, such as "travels", it could be interesting to note that the
information "the user is scared by traveling", is completely di erent from "the user
is angry about traveling". In the rst case, a personalized system will never
suggest to the user to take an airplane to far holiday countries. In the second case,
it can start to recommend many o ers and discounts for traveling again maybe
with a di erent airline company. According to this idea, we suggest the
adoption of a general approach for annotating user-generated content with a topic
of discussion (if possible, with a speci c aspect the user is talking about), and
an emotional label. In particular, we focused on eighth general areas of
application: technology, travels, style, sports, politics, music, movies, and art. In Sec.
3, we propose a strategy for automatically annotate a sentence with one topic
of the discussion, by learning a dynamic semantic representation of words from
newspapers. Moreover, in Sec. 5 we use a word2Vec semantic annotation [
The opinion mining is a complex task which commonly merges the results of two sub-tasks which are resolved independently and combined in a single result only at the end of the process. The rst is the aspect opinion extraction; the second is the sentiment annotation. 2.1
The opinion mining on a textual content has the purpose of discovering one or
more aspects or entities which the writer is talking about. A common approach
for dealing with opinion extraction is topic modeling [
The emotional labeling of aspects, detected in spans of text, is commonly
approached by strategies of polarity annotation (positive, negative, neutral /
objective). Most of them are based on the polarity score associated with each word
in a standard lexicon or thesaurus [
The state of the art described in Sec. 2 points out many issues about opinion mining, including the limitations derived by annotating a single review with only a binary polarity label. In some cases, to overcome the limit more ne-grained intervals of polarity have been added such as "almost positive" or "almost negative", but the information about what kind of emotion the user is trying to expose is still missing.
The opinion mining strategy proposed in this work aims to de ne a general
1 http://alt.qcri.org/semeval2014/task4/
model for annotating textual sentences, including textual sources which are not
reviews, with a macro topic of discussion and an emotional label. We decided
to adopt the Ekman model of emotions [
The module of topic detection task usually involves approaches to span analysis or aspect detection. In our approach, we consider only one topic of discussion at a sentence level. Consequently, it is possible to formalize the topic detection as a multi-class classi cation problem. We decided to use a supervised approach based on SVM as consequence of the optimal results obtained by this algorithm in literature. Eight main topics of discussion have been considered relevant: technology, politics, travel, style, sport, music, movies, art; but the set is easily extensible when new data about di erent topics are provided as a training set of the model. In order to collect enough data for training the model, we developed a crawler of web news already classi ed by humans in that categories. In particular, we chose " ipboard.com" 3 as a platform candidate for the task. The website provides a set of thematic news classi ed in a huge number of categories by the user which shares or writes it. They are free to be read, and they are available through "RSS". When invoked, it provides a list of the last ten news about a speci c topic published on the social platform. We collected
all the articles about the de ned topics from the rst of June 2017 to the 31st of December 2017. For each one, we stored the abstract provided by the RSS system, and the content of the article obtained by visiting its published link and stripped by "HTML" tags using Jsoup library 4.
Dataset. The dataset collected contains in total more than 400.000 articles subdivided as follow, and it is available at: http://bit.ly/EOM dataset
Each news collected is formalized through its link, title, summary and textual content in English language. The table 1 describes the distribution of the articles among the categories and the section of the dataset used for training the algorithm. It is important to note that it has been resized for dealing with the computational cost necessary for processing them. In particular, only the 27,69% of the original dataset has been used, keeping intact the original ratio which identi es the sports category as the most discussed on the web. Moreover, the 60% of the resized dataset has been used as a training set of the method. The last 40% is used half as dev dataset for the optimization of the parameters and the last half as a training set.
Methodology. The implementation of the SVM algorithm used in this work
is the one provided by LibSVM 5 [
l X "i i=1 (1) 4 https://jsoup.org/ 5 https://www.csie.ntu.edu.tw/~cjlin/libsvm/ where w are the features vectors for each xi; i = 1; 2; :::l training vectors, is the error and C > 0 is the regularization parameter. This observation points out the necessity to optimize the choice of C for obtaining accurate results. Moreover, changing the dimensionality and the content of the features vectors w it is possible to obtain completely di erent results.
Vectorial representation of text. We modeled the representation of the
content of articles using both a binary n-grams representation and word embeddings
features vectors. The transformation of the plain-text into unigrams and bigrams
have been implemented through the Apache Lucene Standard Analyzer class 6.
The class uses a default English set of stop-words and a grammar-based tokenizer
with a max length of tokens set to 2. For each n-gram, we use its frequency in the
document as a numerical feature. The formalization through a vector of word
embeddings has been approached averaging the singular embedding vectors of the
words encountered in the document into only one per document. In particular,
the word-embedding procedure used is word2vec [
Optimization of parameters. The model has been optimized considering two parameters: the C regularization parameter of the SVM and the features used for representing the information. We started testing four settings for the C value: 1, 2, 4, 8 xing the representation of the data as the concatenation of the all features available. After we found the better C value, we tested the same algorithm on the news represented respectively by bag-of-words, bag-of-words + embeddings news, bag-of-words + embeddings tweets, bag-of-words + embeddings news + embeddings tweets. With this experimental setting, we aim to understand if the inclusion of embeddings can support a better classi cation process. The evaluation, in both the optimization experiments, has been conducted on the "dev" dataset of the news extracted from Flipboard (Tab. 1). We used as evaluation metrics the accuracy of the classi er and the macro Precision, Recall and F1measure. The macro metrics are obtained just averaging the values of evaluation obtained for each class. In particular, for the F1 measure, its macro version is obtained calculating it on averaged precision and recall. On the contrary, micro metrics are calculated summing all the true positives, true negatives, false positives and false negatives obtained for each class and nally calculating a sin6 https://lucene.apache.org/core/ gular metric value. The results show that a C value equal to 2, allows obtaining better results considering accuracy, precision, recall and F1 measure. Moreover, the best formalization of information is obtained using bag-of-words and both news, tweets word2vec embeddings. The nal con guration used for the topic detection module is then the one with a C value equal to 2 and all the three group of features available.
Internal evaluation of performance. The subset of news kept as "test" dataset have been evaluated with this obtained con guration of the classi er in order to be sure about its performance also on the news unrelated to the optimization process. It is interesting to note that the results (Accuracy: 0.9195, P: 0.9137, R: 0.8965, F1: 0.905) preserve the performance obtained in the optimization tests. Moreover, the values of the accuracy obtained on that dataset are statistically better for p < 0:05, using a two-tailored Chi2 test, than the obtainable predicting the topic label randomly (0,125) or using the most frequent class in the dataset (Sport 0,286). The results allow us to con rm the validity of the approach. 5
The emotion detection module aims to detect the emotion expressed by each
piece of text according to what writer wants to communicate. As consequences
of the assumption of self-reliant and mono-thematic made in this work, we detect
the presence of only one main emotion associated with the sentence. The task
has been approached with the methodology of dynamic lexicon describe in [
Methodology. The approach used for the task is articulated in four steps: classi cation of sentences which come from the training dataset with emotional labels using emoji/emoticons; generation of the emotional lexicon from the annotated sentences; creation of word embedding centroids with the most frequent words for each class; use of a similarity function for detecting the most similar centroid for the piece of text to annotate. The approach used for text labeling is based on the idea that it is possible to automatically annotate user's post on Social Media Sites (SMSs) using emoticons. Each smile has been associated with a main emotional sense with a straightforward strategy. We identify the most common emotions7 included in UNICODE standard version 9 and we manually classify them (also considering their main writing variations). Moreover,
we extended this list with most of the new emoji codes 8 which are, as
previously, associable with a main emotional class. Posts which contains discordant
emoji have been excluded. Due to the page limits of the article, the full list
of emoji/emoticons used is not here presented but the list is available in [
Dataset. The datasets used for the approach is myPersonality 9 [
Extension of centroids words with a standard lexicon. In the approach used for this emotion detection module, we manipulated the list of words in each 8 https://unicode.org/emoji/charts/full-emoji-list.html 9 http://mypersonality.org/ meta-document. First of all, we limited the words obtained by the analysis of annotated posts, to the 100 most frequent. This decision has been taken considering that the huge number of words can create centroid vectors too much similar each other, losing part of its semantic information. Moreover, we decided to enrich each list with the 20 words extracted from a standard lexicon most frequent in our training dataset. The lexicon adopted is NRC Word-Emotion Association Lexicon. It is an extensive English dictionary in which each word is linked to one or more emotions among the eight available (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust).
Emoji-based classi er. This simple classi er uses the presence of smiles inside the text as the main element to consider for assigning a primary emotion to the phrase. A counter for the correspondent emotion is incremented every time that an emoji/emoticon, previously manual classi ed in the six xed emotional classes, is found in the text. This counting is used for de ning an order of the most probable emotion associated with the phrase. If two or more emotions obtain the same number of increments, then the component will be not able to classify the text correctly, and the sentence will be processed by the approach previously described. On the contrary, the emotional label provided as output by the emotion detection module will be the class predicted by the Emoji-based classi er.
Internal evaluation of performance. Unlike what presented in [
The complete pipeline of "Emotion-driven Opinion Mining" has been evaluated through a dataset of 4.898 tweets collected using prede ned hash-tags. In particular, messages which contain both hash-tags about emotions and has-tags about the topic of interest have been crawled. For the emotional hash-tags we used: #happy, #sad, #anger, #fear, #disgust, #surprise. For the topic related hash-tags, we de ned for each one a list of words which refers to micro-topic relevant for it. As an example, for the topic technology, we used the hashtags: #tech, #technology, #phone, #sony, #iphone, #nokia, #samsung, #microsoft, #linux, #mac, #ios, #windows, etc. The crawling task started on the 20th of January 2018 and ended the 20th of February 2018.
Observing the Tab. 2 it is immediately evident what is the main problem of the dataset collected. The "joy" tweets are 71.74% of the whole dataset which results very unbalanced. Moreover, messages associated with disgust and surprise are very low and inevitably challenging to detect. For this reason, we decided to do not consider them in the evaluation. Despite the highlighted issues, we decided to use it for the evaluation due to the di culties to nd another dataset already annotated with both topics and emotional labels. Besides, the dataset has been collected directly from the web, and it re ects what is the real behavior of the users on SMSs. They commonly tend to share more often positive things than negative when communicating on that platforms. This characteristic has been considered an added value more than an issue which on the contrary is often present in arti cial or ad-hoc datasets.
The results resumed in Tab. 3 con rm our hypothesis about the di culties of the task in a real scenario. The topic detection module performed an accuracy of 0,6772 that is about the 30% less than the results obtained in internal evaluations, as the same of the other measures. Considering that the classi cation task has to work with eight di erent classes and spontaneous messages, we can hold to be true the result obtained. An issue observed during the annotation is the di culty of the classi er to match domain relevant entities, as consequence of the pre-processing step performed on the text. Entities such as "Barack Obama", often encountered during the training phase, are never matched in tweets because they are often recalled using hash-tags or "@ citations", which are automatically transformed in a unique sequence of characters entirely di erent from the real entity name. In a similar way, the emotion detection module obtained an accuracy of 0,5587 and a macro F1 measure of 0,2650 which, also in this case, is about the 30% less than the internal evaluation. In that case, we note that the module does not work well on emotions with associated few messages which result challenging to recall. We consequently encourage the further studies to use larger datasets for increasing that statistic. The complete pipeline performed on Twitter messages produces results which could be interpreted as lower than expected, but they are obtained considering as "true positive" only the annotation which correctly predicts both the labels. This strong assumption consequently caused a signi cant reduction of the performance measured because the module should be able to detect one pair ftopic,emotiong among the possible 48 (eight aspects six emotions). The accuracy of 0,3498 is in line with the di culties of the task and, undoubtedly better than a classic random strategy which can obtain only a result of 0,0250 and the use of the most frequent pair (joy; music) that can obtain an accuracy of 0,1672. The di erences in accuracy obtained are statistically signi cant using a two-tailored Chi2 test for p < 0:05. The results are consequently very encouraging and can be considered as one of the rst baselines for further studies about Emotional-driven Opinion mining approaches. 7
The proposed approach of Emotion-driven Opinion Mining has demonstrated to be an e ective strategy for: { discovering the topic of discussion of a message from Social Media Sites { detecting the primary emotion related with the social textual content Traditional opinion mining strategies are commonly applied in the domain of reviews where xed sets of aspects and internal structures of the comments can support a more accurate detection. Moreover, the user opinion is annotated using a polarity level which does not provide a ne-grain overview of the emotional state of the user. Producing more detailed emotional annotations is a challenging task, but nowadays they are essential features for enriching the user pro le and to adopt in personalized systems. In this work, we provide a complete pipeline which, starting from the plain text, can accomplish this task. The proposed module of topic detection is based on an SVM algorithm trained on a dataset of more than 400k real news classi ed according to eight topics: technology, politics, travel, style, sport, music, movies, art. They have been processed and formalized through a conjunct vector of unigrams, bigrams, and word embeddings. The emotions detection module is based on word embedding centroids of the most signi cant words of a speci c emotional class dynamically detected by a corpus of spontaneous messages: 2M of the Facebook post and 1.3M of tweets. The result of the nal Emotion-driven Opinion mining has shown baseline results for the challenging task on a dataset of 1k annotated tweets. To encourage further studies in this area of Opinion Mining and to ensure reproducibility of our experiments, we distributed all the datasets which have been collected and used in this work. They are available at: http://bit.ly/EOM_dataset