=Paper=
{{Paper
|id=Vol-1382/paper8
|storemode=property
|title=A Case-Study for Sentiment Analysis on Twitter
|pdfUrl=https://ceur-ws.org/Vol-1382/paper8.pdf
|volume=Vol-1382
|dblpUrl=https://dblp.org/rec/conf/woa/FornacciariMT15
}}
==A Case-Study for Sentiment Analysis on Twitter==
https://ceur-ws.org/Vol-1382/paper8.pdf
Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy
A Case-Study for Sentiment Analysis on Twitter
Paolo Fornacciari, Monica Mordonini and Michele Tomaiuolo
Dipartimento di Ingegneria dell’Informazione
Università degli Studi di Parma
Parma, Italy
e-mail: paolo.fornacciari@studenti.unipr.it, {monica.mordonini, michele.tomaiuolo}@unipr.it
Abstract — Microblogging platforms like Twitter can convey address the risks of online social networks, which are perceived
short messages to direct contacts, but also to other potentially as serious by many users and have led to incidents [13][35].
interested users. They are actively exploited either by individual Ethnicity, religion, sexual orientation, political beliefs are
users or whole organizations and companies. This paper other factors that have led to the establishment of dedicated
describes some results we obtained from the Social Network and social network services, but probably they are also playing an
Sentiment Analysis of a Twitter channel, related to a pop music active role in creating and aggregating online communities
event. Apart from the particular results, a methodology and some leveraging the bigger and most popular social networks. This
guidelines for the automatic classification of Twitter content are
suggests the possibility of new ways to spread information and
discussed.
to influence public opinion. These new scenarios can be better
Keywords—Social Network; Sentiment Analysis; Hierarchical evaluated by a combined observation of the structure and the
Classification actual content of the network. This kind of analysis could
highlight emerging social behaviors. In [6], for example, the
possible differences in the sentiment polarity of female and
I. INTRODUCTION male users, towards the discussed topic, are examined.
In the common meaning of the term, an online community To investigate on the content and on the relations among
(or virtual community) is a group of people interested in a the actors of a network, it could be useful to contextualize the
particular topic, or that share some ways of thinking, or that in network itself. In particular, it could be important to consider
general have some kind of link that brings them together, with and inquiry the content of the messages that guide the
the peculiarity that they interface and connect to each other relationships of the community. It is only through this kind of
through a data communication network (such as Internet). In investigation that we can analyze the semantic meaning of a
this way, they form a social network with unique link, from which we could infer the kind of relationship. This
characteristics: in fact this combination is not necessarily sharpens our description of the social network in many of its
bound to a physical place and anyone can participate wherever facets. A useful tool for such surveys is Sentiment Analysis
he is, with a simple access to networks. (SA). SA is a branch of Opinion Mining, that aims to listen and
The social networking sites (SNSs), as defined by Boyd process the data that users post on social media. It is an
and Ellison in [9], are a collection of web-based services that interdisciplinary field that in recent years has had a significant
allow users to build a profile within the system and define a list growth and that makes an extensive use of machine learning
of other users with whom they have some kind of connection. techniques. A survey of the main techniques and approaches
According to Sunden profiles are unique pages where one can can be found in [26][7][8]. In [33], it is showed how the
“type oneself into being” [32], as the creation of a profile is the information about social relationships can be used to improve
minimum condition for joining an SNSs. What makes the SNSs user-level sentiment analysis. In [25] Sentiment Analysis is
unique is that their purpose is not, in most cases, to allow users mapped on social media with observations and measurable
to make new friends but the emphasis is on making visible data; the results highlight the importance of SNSs (i.e.
their existing social networks and on the chance to describe Facebook) as a platform for online marketing.
them. On the other hand, the specific features of each social
network site may depend also on the possible target (social, II. BACKGROUND
linguistic or geographic) to which the service is directed. The Anthropologist John Barnes was the first to introduce the
architecture of social networking platforms is very concept of social network. In 1954 in [5] he described the
differentiated. While the most popular platforms are build as results of over two years of studies on the composition of
essentially centralized systems, other platforms have a classes and social groups in the town of Bremnes (today
distributed architecture [14][15]. The decentralized systems, in Bomlo) in Norway. James Mitchel in [22] gave a more
particular, often use some notion of trust and cryptography to sociological and analytical interpretation, describing a social
53
Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy
network as “a specific set of linkages among a defined set of knowledge, and detecting emotions [11]. In [10] the correlation
actors, with the additional property that the characteristics of among topics and the positive or negative opinions are
these linkages as a whole may be used to interpret the social investigated, to automatically classify the topics themselves.
behavior of the actors involved”. Mitchel is a representative of An ontology driven approach is used in [4] to extract rich
the anthropological school of Manchester, formed in the late emotional semantics of tagged texts, by combining available
40s, whose founders were the first to use the concept of computational and sentiment lexicons with an ontology of
network in a systematic way. emotional categories. A similar approach can be taken into
More simply and more generally, in [34] Wasserman and consideration for the detection of feelings in tweets: for
Faust defined a social network as a finite set of actors and the example, a taxonomy of feelings can drive the selection of
relation or relations defined on them. This approach is hashtags for the automatic search of tweets with a prevalent
characterized by the priority interest turned to the shape of the sentiment. Such tweets can be used in the training phase of an
networks, rather than their content. According to the exponents automatic classifier.
of this line of research, the form of social relations largely
determines their content. This theory (developed since the 70s III. SENTIMENT ANALYSIS ON TWITTER
at Harvard) lays the foundation for social network analysis In this research work, we built a system for social network
(SNA). SNA has the objective to model social structures with and sentiment analysis, which can operate on Twitter data.
different properties, starting from the mathematical theory of Twitter is a popular platform for social networking and
graphs and the use of matrix algebra [12]. All these definitions microblogging, counting hundreds of millions of active users
could be summarized by arguing that a social network is a and daily published messages. As a social networking platform,
group of individuals (actors) which are connected to each other Twitter is structured as a directed graph, in which each user can
through different types of social links (relationships), such as choose to follow a number of other users (followees), and can
family ties, employment relationships, superficial knowledge, be similarly followed by other users (followers). Thus, the
common interests. With the development of communication “follow” relationship is asymmetrical, it does not require
technologies and the growth of online communities, the mandatory acknowledgement, and it is essentially used to
importance of social networks has increased. The research in receive all public messages published by any followee user. As
SNA finds application in analytical and predictive models used a microblogging service, Twitter is used to publish short
in sociology, anthropology, psychology, computer science and messages counting a maximum of 140 characters (tweets),
economics [29]. which may contain opinions, thoughts, facts, references to
One of the most popular social networks is Twitter images and other media. Moreover, through the @ symbol it is
(https://twitter.com/). At the end of 2012, the company declared possible to introduce mentions, i.e. references to other users,
in a tweet: «There are now more than 200M monthly active and through the # symbol it is possible to introduce hashtags,
@twitter users. You are the pulse of the planet. We're grateful i.e. references to discussion topics.
for your ongoing support.» In this short message, the company Consequently, in our analysis we collected three types of
announced what many researchers in different domains had data. The User type represents users' profiles; from Twitter we
already noticed: the information and opinions in our society go obtain the following fields: user_id, name, location,
through a social network where everyone can sign up and num_followers, num_tweets. The Tweet type represents posted
participate. So the analysis of this large amount of data is an messages; from Twitter we obtain the following fields:
exciting challenge for researchers, but it is also crucial for all tweet_id, user_id, message, date. Finally, the Friend type
those who work at different levels in the current information represents the “follow” relationships among users. Apart from
society. data obtained directly from Twitter, we added a field to both
Twitter has been the subject of attention from researchers as tweets and users, to associate a sentiment with them, according
early as 2009, for example in [18]. In [24], the authors describe to the result of our analysis.
a recent important application for understanding how public As a communication medium, tweets have a quite peculiar
sentiment is shaped, how it could be tracked and its nature. Some distinguishing features of communication on
polarization with respect to candidates and issues. Another kind Twitter are related to technical aspects; those include length of
of research in the Twitter social network is to combine data text, tags, urls, etc.. Other features may be classified as
source and sentiment analysis. In [2] geo-spatial information idiomatic use of the medium, and create a sort of Twitter
related to tweets is used for estimating happiness in the Italian culture; those features include typical content and most
cities. Twitter is also a microblogging platform, so the discussed topics, idiomatic expressions, abbreviated forms, etc.
techniques used generally in Sentiment Analysis and Text For example, a tweet may have the following form:
Classification must be adapted to the famous 140-character
tweet and this opens the way for new issues. Some example of «RT @richman wow this is the #happiest day of my life.
work in this sector are described in [1][21][20][36]. One of the #happy #glad #icantbelievit :) :D http://t.co/4VEH827bG7»
major problems is how to automatically collect a corpus for
Sentiment Analysis and Opinion Mining purposes; see, for The peculiar nature of tweets requires specialized analysis
example, [28][19]. techniques. As a start, a tweet may contain many elements
Sentiment Analysis is traditionally focused on the which are not significant for our classification, and can thus be
classification of web comments into positive, neutral, and dropped though a filtering process. To polish the message, we
negative categories. But an intelligent and flexible opinion- defined various filters, which we have applied in a
mining system has to incorporate a deeper analysis of affective customizable sequence.
54
Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy
A first filter eliminates useless tokens. Removed tokens our tests, we used the following list: @ABC, @BBCNews,
include: the starting “RT” sequence, which indicates a @BBCSport, @business, @BW, @cnnbrk, @CNNMoney,
republished messages from a different user (i.e. a retweet); the @fox32news, @latimes, @nytimes, @TIME. To obtain
@ character and the whole following user name; the # symbol, subjective content, instead, we gathered comments directed to
but not the following topic name, which is kept in the message. the same list of users.
The topic name is also removed, though, when it coincides About the polarity classifier, we decided to search for
with the name of the channel where tweets are collected from. sources of mostly positive or negative messages, respectively.
A second filter applies the language specific rules. It On the one hand, those sources should fit the particular setting
includes an orthographic correction of the message, which is of Twitter (short messages, idiomatic expressions, smiles, etc.).
used to remove unknown words, which may not appear in any On the other hand, they should not be specific to a particular
other tweet (in the example: “icantbelievit”). Ideally, the filter topic or context (sport, music, etc.). Thus, we dropped the idea
at this level should also support stemming and removal of of collecting messages about particular events, mostly
stopwords. However, those operations can be easily performed generating either positive or negative sentiments. Instead, we
by Weka, which we used for analysis. collected messages, using generic yet polar terms as queried
Finally, another filter separates all punctuation symbols hashtags. In particular, we used the following channels to
from the text, and organizes them as single-character words. gather positive content: #adorable, #awesome, #beautiful,
However, some typical patterns are kept as aggregates, #beauty, #cool, #excellent, #great. We used the following
including smiles sequences, repeated question and exclamation channels to gather negative content: #angry, #awful, #bad,
marks. #corrupt, #pathetic, #sadness, #shame. Actually, such terms
The final result of the filtering process is a word vector, have been chosen quite empirically, taking into account the
which is then submitted to the classifier agents. As we have quality of training sets they generated. But they could be
mentioned, our analysis aims at identifying the following selected from WordNet-Affect [31], SentiWordNet [3], and
classes of messages: undiscriminated, objective, subjective, other affective lexicons, in a more systematic way.
positive, negative. This way, the training set is generated in an automated
The system is organized as a simple hierarchy of agents, fashion, as a list of tweets. Each tweet is associated with its
mimicking the hierarchy of sentiment classes. In fact, since supposed class, in accordance to its source. In fact, the training
objective messages have no polarity by definition, the classifier set is not perfect, as it contains messages gathered from public
for positive and negative sentiments is only applied to channels. However, a training set of this kind can be generated
subjective messages. If a message fails to be classified at the easily and in a methodical way, from real and updated Twitter
first stage, then it simply remains undiscriminated. If it fails to messages. In the next section, we will also discuss the quality
be classified at the second stage, then it is marked as of results that can be obtained, using it as a basis for sentiment
generically subjective. analysis.
The training set can be provided directly to the classifier
agents. In the present form, the system is based on Weka, and
can thus be configured for performing additional preprocessing
steps on the messages, including common TF-IDF
transformations, stemming, elimination of stopwords,
exclusion of infrequent words, etc.
Currently, we analyze tweets for discriminating the basic
classes of objectivity and polarity, at two levels. However, we
designed the system for more complex hierarchical
classification, with the application of various types of
classifiers, as an alternative to current Naive Bayes.
In fact, hierarchical classification has been applied
successfully in a number of studies, for information retrieval
[30]. It has been proven effective especially in the case of
classification over hierarchical taxonomies. Moreover, it has
the advantage of being modular and customizable, with respect
to the classifiers used at different levels. Using the same
probabilistic classifier and a maximum likelihood estimator,
Fig. 1. Hierarchy of basic sentiment classes.
instead, does not provide advantages for the hierarchical
approach over the flat approach. Mitchell [23] has proved that
Currently, the classifier agents apply the Multinomial Naive the same feature sets represent documents in both approaches.
Bayes algorithm, but other methods can be used and different Consequently, the whole hierarchical classifier system is
agents can be plugged in the system. However, instead of equivalent to the corresponding flat system.
generating a training set by hand, we aimed at realizing an Also in the case of sentiment analysis, a hierarchy of
automated (or at least semiautomated) process for obtaining classes can be defined [16][4]. Accordingly, hierarchical
good training sets. classification has already been applied to sentiment analysis,
About the objectivity/subjectivity classifier, we adopted a too [17].
similar strategy to [27]. In fact, to obtain objective content, we
gathered messages generated from popular news agencies. In
55
Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy
Fig. 2. Communities participating in the #SamSmith channel.
IV. A CASE-STUDY: THE #SAMSMITH CHANNEL type of the published tweets and the instances used for training
the classifiers. All data were downloaded between 2015-02-02
This section will show the results of the classifiers and the
and 2015-02-10. The awarding of the Grammy took place on
analysis carried out on a case study.
2015-02-08. The network (shown in Fig. 2) consists of a total
With the above described software, it is possible to obtain
of 5570 nodes and 6886 arcs.
some training sets for the classifiers. In our case study, they
Looking at the figure, it is possible to notice that the
consist of:
network topology is consistent with the nature of the
considered case. In fact, most of the channel consists of
• 86000 instances (polarity) independent users (or small groups of users) that express their
• 32000 instances (subjectivity) opinion about the artist; however, in the central part of the
network there are some major communities.
These instances have been obtained by exploring more than As shown in Fig. 3, the prevailing sentiment detected from
60 channels on the social network. the classifier is the negative one. Performing an analysis on a
In the generated models, the selected features are consistent
with our expectations: the typical expressions of a certain
feeling (such as smileys, or some words that express
appreciation or disgust) show a higher probability of belonging
to the class of that feeling, rather than to the class of the
opposite sentiment.
The obtained results by the classifiers using cross-
validation (with folds = 10) on the training sets showed an
accuracy of:
• 77,45% (polarity classifier)
• 79,50% (subjectivity classifier)
These results show that the model of the classifiers contains
effective features for the recognition of the sentiment of a
message.
The case study which was considered in this work is the
social network of the #SamSmith channel (the singer who won Fig. 3. Sentiment analysis on the #SamSmith channel.
four awards at the Grammy Awards 2015). The choice of this
channel is justified by the strong similarities found between the
56
Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy
actually positive, while the other one is objective. This episode
shows how some errors of assessment can have important
impact on larger communities.
Another kind of analysis we made concerns with the grade
of the users. Fig. 5 shows that two nodes have a key role within
the social network:
• @samsmithworld
• @TheGRAMMYs
Fig. 4. A small community, showing positive sentiment.
These users are the main sources of news about the singer
Sam Smith and the event Grammy Awards 2015. This explains
their importance within the social network which we
sample of tweets in the network, we noticed that many considered.
sentences are actually quotes of songs. These messages contain
melancholic and sad phrases, and are therefore classified as V. CONCLUSION
negative. Considering that a quote is generally an appreciation
for the artist, most users classified as negative are actually In this article, we describe some results obtained from the
positive users. This is a typical example of a classic problem of synthesis of Social Network Analysis and Sentiment Analysis
misunderstanding of the SA: the system, while classifying applied to the channel #SamSmith during the Grammy Awards
correctly the tweet, misses the assessment of the feeling in 2015. Apart from the particular results, a methodology and
because it can not evaluate the tweet together with its context. some guidelines for the automatic classification of Twitter
content have been discussed.
For evaluating the performances of our system, we
conducted a simple survey through a group of persons in our The implemented software allows: (i) to get a training set
department. In this way, we selected and classified 100 for the classifiers that deal with Sentiment Analysis, and (ii) to
messages that show a clear opinion on the singer. Then, we make a thorough study of the topology of the networks.
used those messages as a test. The results of the classifiers The study of the global sentiment within the network has
showed an accuracy of 84% for the polarity and 88% for highlighted the typical problems of Sentiment Analysis (irony,
subjectivity. sarcasm, lack of information, etc.). Additionally, some peculiar
In the network periphery, it is possible to notice a small problems of the considered channel were also detected (such as
group of users whose feeling is completely positive (Fig. 4). the quotes of songs).
After a careful analysis of users' tweets in this small group, it The performances obtained by the classifiers during tests
was found that these posts are mainly retweets and the original conducted on the training set and the analysis of the case
messages are only two. Of these two messages, the first is studies have shown good and promising results.
Fig. 5. The most followed nodes in the #SamSmith channel.
57
Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy
REFERENCES [18] A. Go, L. Huang, and R. Bhayani, “Twitter sentiment analysis,” Final
Projects from CS224N for Spring 2008/2009 at The Stanford Natural
[1] A. Agarwal, B. Xie, I. Vovsha, O. Rambow, and R. Passonneau. Language Processing Group, 2009. [Online].
“Sentiment analysis of Twitter data”, Procs of the Workshop on
[19] S. Hassan, F. Miriam, H. Yulan, and A. Harith, “Evaluation datasets for
Languages in Social Media (LSM '11), pp. 30-38, 2011.
Twitter sentiment analysis: a survey and a new dataset, the STS-Gold,”
[2] L. Allisio, V. Mussa, C. Bosco, V. Patti, and G. Ruffo, “Felicittà: Proc. of ESSEM 2013, Turin, Italy, 2013.
Visualizing and Estimating Happiness in Italian Cities from Geotagged
[20] S. Kiritchenko, X. Zhu, and S. Mohammad, “Sentiment analysis of short
Tweets,” Proc. of ESSEM 2013, Turin, Italy, 2013.
informal texts,” J. Artif. Int. Res., vol. 50(1), 2014.
[3] A. E. S. Baccianella and F. Sebastiani, “Sentiwordnet 3.0: An enhanced
[21] A Kowcika, A. Gupta, K. Sondhi, N. Shivhre, R. Kumar, "Sentiment
lexical resource for sentiment analysis and opinion mining,” Proc. of the
Analysis for Social Media", Int. J. of Advanced Research in Computer
7th Conf. on Inter. Language Resources and Evaluation (LREC’10),
Science and Software Engineering, vol. 3(7), 2013.
ELRA, 2010.
[22] J.C. Mitchell, Social Networks in Urban Situations: Analysis of Personal
[4] M. Baldoni, C. Baroglio, V. Patti, and P. Rena, P. “From tags to
Relationships in Central African Towns, Manchester Univ. Press, 1969.
emotions: Ontology-driven sentiment analysis in the social semantic
web,” Intelligenza Artificiale, vol. 6(1), pp. 41-54, 2012. [23] T. Mitchell, “Conditions for the Equivalence of Hierarchical and Flat
Bayesian Classifiers”. Technical report, Center for Automated Learning
[5] J. A. Barnes, “Class and Committees in a Norwegian Island Parish,
and Discovery, Carnegie- Mellon University, 1998.
Human Relations”, Human Relations, vol 7(1), pp. 39-58, 1954.
[24] S. Mohammad, X. Zhu, S. Kiritchenko, J. Martin, “Sentiment, emotion,
[6] A. Bermingham, M. Conway, L. McInerney., N. O'Hare, and F.
purpose, and style in electoral tweets”, Information Processing &
Smeaton, “Combining Social Network Analysis and Sentiment Analysis
Management, Elsevier, vol 50 (1), 2014.
to Explore the Potential for Online Radicalisation”, in Procs of the 2009
Int. Conf. on Advances in Social Network Analysis and Mining [25] F. Neri, C. Aliprandi, F. Capeci, M. Cuadros., and T. By, “Sentiment
(ASONAM '09). IEEE Computer Society, pp 231-236, 2009. Analysis on Social Media”, Proces of the 2012 Int. Conf. on Advances
in Social Networks Analysis and Mining (ASONAM '12), IEEE
[7] L. Bing, “Web Data Mining: Exploring Hyperlinks, Contents, and Usage
Computer Society, pp: 919-926, 2012.
Data,” Data-Centric Systems and Applications, 2nd ed., Springer, pp. 1-
603, 2011. [26] B. Pang, and L. Lee, “Opinion Mining and Sentiment Analysis,” Found
Trends Inf. Retr., vol. 2 (1-2), pp. 1-135, 2008.
[8] L. Bing. Sentiment Analysis and Opinion Mining, Morgan & Claypool
Publishers, 2012. [27] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up?: sentiment
classification using machine learning techniques”. Procs of the ACL-02
[9] D. Boyd, N. Ellison, “Social Network Sites: Definition, History and
conf. on Empirical methods in natural language processing (EMNLP
Scholarship,” Journal of Computed-Mediated Communication, vol. 13
'02), vol. 10, USA, pp. 79-86, 2002.
(1), pp. 210-230, 2008.
[28] A. Pak and P. Paroubek, “Twitter as a corpus for sentiment analysis and
[10] K. Ca, S. Spangler. Y. Chen; L. Zhang, “Leveraging Sentiment Analysis
opinion mining,” in Proc. of the 7th conf. on International Language
for Topic Detection,” Web Intelligence and Intelligent Agent Technology
Resources and Evaluation (LREC’10). Valletta, Malta, May 2010.
(WI-IAT '08), IEEE/WIC/ACM Int. Conf. on, vol.1, pp. 265-271, 2008.
[29] J. Scott, Social Network Analysis, 3rd ed., London and Beverley Hills
[11] E. Cambria, B. Schuller, Y. Xia, C. Havasi , “New Avenues in Opinion
Sage Publications, 2012.
Mining and Sentiment Analysis,” IEEE Intelligent Systems, vol.28, no.
2, 2013. [30] Jr. Silla, and A. Freitas, “A survey of hierarchical classification across
different application domains,” Data Mining and Knowledge Discovery,
[12] P. Carrington, J. Scott, S. Wasserman, Models and Methods in Social
vol. 22(1-2), pp. 31-72, 2011.
Network Analysis, Cambridge University Press, 2005.
[31] C. Strapparava and A. Valitutti. “WordNet-Affect: an affective extension
[13] E. Franchi, A. Poggi, and M. Tomaiuolo. “Information and password
of WordNet,” Procs of 4th Int. Conf. on Language Resources and
attacks on social networks: An argument for cryptography,” Journal of
Evaluation (LREC’04), vol. 4, pp 1083–1086, 2004.
Information Technology Research (JITR), 8(1), 25-42, 2015.
doi:10.4018/jitr.2014070104. [32] J. Sunden, Material Virtualities. Approaching Online Textual
Embodiement, New York: Peter Lang., 2003.
[14] E. Franchi, A. Poggi, and M. Tomaiuolo, “Open Social Networking for
Online Collaboration,” International Journal of e-Collaboration (IJeC), [33] C. Tan, L. Lee, J. Tang, L. Jiang, M. Zhou, and P. Li, “User-level
9(3), 50-68, 2013. doi:10.4018/JITR.2015010103. sentiment analysis incorporating social networks”, Procs of the 17th
ACM SIGKDD Int.. Conf. on Knowledge Discovery and Data Mining
[15] E. Franchi, and M. Tomaiuolo, “Distributed social platforms for
(KDD '11), ACM, pp 1397-1405. 2011.
confidentiality and resilience,” Social Network Engineering for Secure
Web Data and Services, IGI Global Publisher, pp. 114-136, 2013. [34] S. Wasserman, K. Faust, Social Network Analysis: Methods and
doi:10.4018/978-1-4666-3926-3.ch006. Applications, Cambridge University Press, 1994.
[16] V. Francisco, P. Gervas, and F. Peinado., “Ontological reasoning to [35] M. Tomaiuolo, “dDelega: Trust Management for Web Services,”
configure emotional voice synthesis,” Procs of Web Reasoning and Rule International Journal of Information Security and Privacy (IJISP), 7(3),
Systems, vol. 4524 of LNCS, pp. 88–102. Springer, 2007. 53-67, 2013. doi:10.4018/jisp.2013070104
[17] D. Ghazi, D. Inkpen, and S. Szpakowicz, “Hierarchical approach to [36] X. Zhu, S. Kiritchenko, and S. Mohammad, “NRC-Canada-2014: Recent
emotion recognition and classification in texts,” Advances in Artificial improvements in sentiment analysis of tweets,” Procs of the Int.
Intelligence, vol. 6085 of LNCS, Springer Berlin, pp 40-50 2010. Workshop on Semantic Evaluation, Dublin, Ireland, 2014.
58