-Online social networking communities usually exhibit complex collective behaviors. Since emotions play a relevant role in human decision making, understanding how online networks drive human mood states become a task of considerable interest. One of the most relevant task in Sentiment Analysis is Polarity Classification, aimed at classifying the sentiment behind texts. We formulated different assumptions regarding which patterns within a message can be relevant sentiment indicators. Differently from well-formed texts, messages on social networks contain emoticons which could be strong sentiment indicators. For this, the first assumption states that the occurrences of emoticons representing a certain polarity could strongly agree with the overall message polarity. We then expanded the feature space including initialisms for emphatic and onomatopoeic expressions (e.g. bleh, wow, etc.) and “stretched words” (words with a letter repeated several times to emphasize a mood), extensively used in social media messages, because they could be useful information to help in determining the sentiment. Detailed analyses have been performed in order to support our assumptions. Four Machine Learning (supervised) classifiers are applied upon the expanded feature space model. Several experiments show that the considered features lead to increments of accuracy up to 5%.
According to the definition reported in [
Sentiment Analysis is a growing area of Natural
Language Processing with research ranging from document level
classification [
Given the common characteristic of posts on social media to be short (e.g. the limit imposed by Twitter - a popular microblogging social networking web site - is 140 characters per post), classifying the sentiment of posts is most similar to sentence-level Sentiment Analysis.
However, the informal, specialized and length constrained language makes SA on social media a complex task. How well the features and techniques used on more well-formed data will transfer to the social media domain is an open question.
Characteristics that distinguish social media contents from
well-formed contents (e.g. movie reviews [
However, SA on social media leads towards new and more complex scenarios. In a post, a sentiment is conveyed in one or two sentence passages, which are rather informal and usually filled with abbreviations and typos. These messages are less consistent in terms of language, and usually cover a much wider array of topics. Since 2001, several studies based on polarity classification for well-formed scenarios have been proposed [4], [8], while polarity classification on usergenerated content has rapidly grown only the last few years. For instance, Barbosa and Feng [10] explored the linguistic characteristics of how tweets are written and the metainformation of words for polarity classification. In the study of Davidov et al. [11] four different feature types (punctuation, words, n-grams and patterns) are used for polarity classification and the contribution of each feature type evaluated for this task. Celikyilmaz et al. [12] proposed a new method for text normalization and investigated its effect when used for polarity classification. In particular, they used pronunciations of words to map alternative and shorter spellings into the intended words (reducing the sparseness caused by the noise in tweets).
SA is a multidisciplinary field that affects different
branches of Computer Science, Social and Management
Sciences. During the last years, several intersections between
Sentiment Analysis and the Multi-Agent system technology
are emerging [13]–[15]. For instance, Almashraee et al. [
By using a stochastic multi-agent based approach, the system proposed by Gatti et al. [17] models and simulates user behavior on real-world social networks, taking into account what users (agents) post. Several challenges are faced: sampling the networks from the real-world social networks, performing text classication (Natural Language Processing) to predict topic and sentiment from posts, modeling the user behavior to predict his/her actions (pattern recognition), and large-scale simulation.
In this work, we propose different approaches for text normalization and feature expansion to improve classification performance: after that messages are analyzed and normalized/preprocessed, different additional features (initialisms for emphatic and onomatopoeic expressions, emoticons, adjectives and “stretched words”1) are integrated within the bag-of-words model and used by common Machine Learning (supervised) classifiers. Further details are reported in Sect. II.
Although adaptation of this framework to other microblogs is straightforward, experiments are addressed on Twitter (tweets are short status updates of 140 characters or less) since it is an increasingly popular platform able to convey opinions and thoughts. Polarity classification approaches based on Twitter could provide unprecedented utility for different parties (e.g. marketing and financial purposes). For instance, an industry could gauge its recent marketing campaign by aggregating user opinions regarding their products. Moreover, it might be possible to identify the sentiment of financial news to forecast returns of markets [18] or sound out public opinion during political campaigns [19].
We propose a system that is composed of three main modules: the first deals with preprocessing techniques, such as Text Normalization and Spelling Corrections, the second with Feature Expansion and the last with supervised classification techniques. Data are stored in a database to be subsequently easily reused and plotted. The system architecture is reported in Fig. 1.
Since tweets are similar to SMS messages, the writing style and the lexicon is widely varied. Moreover, tweets are often highly ungrammatical, and filled with spelling errors.
a) Text Normalization: In order to clean the dataset, we captured a set of patterns which are detected using dictionaries a priori defined and regular expressions. The applied filters are: URLs: All tokens matching the REGEXP (https?jftpjfile)://[-a-zA-Z0-9+&@#/% ?=˜_j!:,.;]*[-a-zA-Z0-9+&@#/%=˜_j] are transformed in its form without punctuation to avoid URL segmentation during tokenization (e.g. http://www.mind.disco.unimib.it becomes httpwwwminddiscounimibit); Hashtags: The symbol # is removed from all the tokens; Mention Tags: The tokens corresponding to a mention tag, identified through the REGEXP @(.+?), are removed; Retweet Symbols: All the tokens matching the expression RT @(.+?): are removed.
1Words that have a letter repeated several times to emphasize a mood, e.g. “I’m so happyyyyyyyyyy!” 2https://code.google.com/p/google-api-spelling-java/ Note that the adaptation and modifications of the REGEXPs adopted in these filters to other microblogs is straightforward.
b) Spell-Checker: In addition to filters, misspelled tokens have been corrected using the Google’s Spell Checker API2. Since the Google’s algorithm takes the neighbourhood (context) of a misspelled token into account in suggesting the correction, the whole previously filtered tweet is considered as a query rather than the single token.
Once the text normalization step has been performed, some additional features have to be extracted:
Emoticons: in order to detect positive, neutral and negative emoticons, three dictionaries have been defined. For instance, positive emoticons are ’:-)’, ’:)’, ’=)’, ’:D’, neutral emoticons are ’:-j’, ’:j’, ’=j’, ’;j’ and negative emoticons are ’:-(’, ’:(’, ’=(’, ’;(’.
If a token appears in the dictionary of positive emoticons then it is replaced with POSEXPRESSIONS, if it appears in the dictionary of neutral emoticons it is replaced with NEUEXPRESSIONS, otherwise with NEGEXPRESSIONS; Initialisms for emphatic expressions: several emphatic expressions are used in English. For instance, expressions such as ’ROFL’,’LMAOL’,’LMAO’,’LMAONF’ represent positive expressions. They are replaced with POSEXPRESSIONS, NEUEXPRESSIONS or NEGEXPRESSIONS; Slang correction: in order to aggregate terms with the same meaning but different slangs, a dictionary of a priori defined slang expressions with their meaning, such as ’btw’ (by the way), ’thx’ (thanks), ’any1’ (anyone) and u (you) has been built; Onomatopoeic expressions: as the previous point, a mapping dictionary has been defined for onomatopoeic expressions, such as ’bleh’ (NEGEXPRESSIONS) and ’wow’ (POSEXPRESSIONS). Also laughs are considered as onomatopoeic expressions: if a token has a sub-pattern matching ((ajejijoju)hjh(ajejijoju))nn1+j (ahhajehhejihhijohhojuhhu)+, then the whole token is replaced with POSEXPRESSIONS; Stretched words: a specific procedure has been defined to detect weather a term is a stretched word or not: 1: if Term has a lengthening then 2: root Extract term root 3: correction list GoogleSpellChecker (root) 4: if correction list = ; then 5: return isStretched 6: end if 7: else 8: return isNOTStretched 9: end if A particular REGEXP has been defined to detect the presence of a lengthening (or stretching) in a term. Whether there is a match, the term root is extracted, otherwise the term is discarded and the next token is analyzed. The term root is analyzed by the Google’s Spell Checker3. The spell checker’s output is a list of possible corrections, ordered with respect to their probability. If the list is empty, the term is declared to be a stretch word.
Adjectives: a Part-Of-Speech (POS) tagging process has been performed in order to tag each term with respect to its verbal form, to subsequently extract the adjectives (tagged as JJ, JJR, JJS) and determine their polarity depending on the fact that the term is in the dictionary of positive or negative terms. If the adjective is neither in the positive nor in the negative dictionary, its polarity is assumed to be neutral. The Stanford Log-linear Part-Of-Speech Tagger library4 of Stanford University has been used for this task.
Let d~ = (t1; :::; tn) a traditional feature vector composed only of terms, the new representing feature vector is defined as: d~new = (t1; :::; tn; epos; eneu; eneg; se; adjpos; adjneu; adjneg; strw; class) where fpos; neu; negg 2 pol is the polarity, epol represents the emoticons, initialisms for emphatic and onomatopoeic expressions according to polarity pol, adjpol represents the adjectives according to polarity pol, strw represents the stretched words and class is the ground truth polarity. According to the used term weighting method, boolean (0/1) or Term-Frequency (TF), adjpol and strw represents the presence or absence of 3https://code.google.com/p/google-api-spelling-java/ 4http://nlp.stanford.edu/software/tagger.shtml the feature or how many times the feature occurs, respectively. Considering the boolean weighting schema, epol is zero if all the three atomic features involved (emoticons, initialisms for emphatic and onomatopoeic expressions) are zero and one if at least one of them exists, while considering the TF method it assumes the sum of how many times each atomic feature occurs. Experiments on the datasets have returned higher results using the 0/1 weighting schema for terms and TF for the additional external features.
The supervised classifiers used and compared in the system are: Naive Bayes (NB), K-Nearest Neighbors (K-NN), Support Vector Machine (SVM) and Decision Trees (DT). Several classifier configurations have been tested (linear, polynomial and gaussian kernel for SVM and Naive Bayes Multinomial) and the most performing are used. For SVM, the linear kernel is used, while the Naive Bayes Multinomial overperforms the Naive Bayes classifier. K-NN has been tested for k = 1; 3; 5; 10 and the most performing value is k = 3. A 10folds cross validation has been adopted as evaluation criteria. In order to obtain more statistically significant results, each experiment has been performed 10 times. The final performance are obtained by the arithmetic mean among the experiments. Classification experiments have been performed using Java. In particular, the WEKA5 libraries have been adopted as tools for classification.
III.
In order to evaluate the proposed method, we performed our experiments on three datasets. The first is called Gold Standard Person [20], the second Gold Standard Movie [20] and the third is a concatenation of the two plus 3258 additional posts (that we called ’merged’).
Each gold standard dataset originally contains 1,500 manually labeled Twitter data for target-specific sentiment (i.e., 5http://www.cs.waikato.ac.nz/ml/weka/ the sentiment towards a specific target such as a movie or a person).
Each line of the data entry follows the format (id, topic, content, polarity), where ’id’ is the id of the tweet, ’topic’ is the name of the movie/person talked in the tweet, ’content’ is tweet content and ’polarity’ is the sentiment polarity about the topic expressed in the tweet, which can be ’pos’ (positive), ’neg’ (negative), ’neu’ (neutral), or ’no sentiment’ (not considered in this work). Polarity distributions for each of the studied dataset are reported in Figure 2.
We report in the following the descriptive analysis for adjectives, emoticons and stretched words. Similar statistics have been obtained (but omitted) for initialisms for emphatic and onomatopoeic expressions.
The assumptions we formulated regard which patterns behind messages can be relevant sentiment indicators. Wilson et al. [21] shows that using emoticons for the learning phase of a classifier can lead to performance improvements. Moreover, Marchetti-Bowick and Chambers [22] present an approach that instead uses distant supervision (using emoticons as ground truth for labels) to train a classifier on a dataset of tweets, achieving higher performance in polarity classification. For this reason we argue that the occurrences of emoticons representing a certain polarity could strongly agree with the overall message polarity.
In addition to emoticons, we expect that also adjectives could be relevant sentiment indicators: in human interactions, the use of adjectives offers to the author the possibility to describe, in the best possible way, the own subjectivity within the discourse. Moghaddam and Popowich [23] demonstrated that the inclusion of adjectives as features in the bag-of-words model improves the classification accuracy.
Finally, we assume that also stretched words, extensively used in social media posts, could be useful information to help in determining the sentiment. To the best of our knowledge, no studies consider the combination of adjectives, initialisms for emphatic and onomatopoeic expressions, emoticons and stretched words as possible additional features.
In order to verify whether the proposed preprocessing techniques and generated features improve classification performance, four experiment configurations have been considered for each studied dataset (Table II).
An analysis on the adjective distribution has been performed on the studied datasets (Table III).
First of all, as expected, positive and negative messages have a high percentage of adjectives. In order to verify that adjectives could be an important source of information for polarity classification, a further and detailed analysis has been conducted calculating conditional probabilities (conditioning the adjective presence to the overall message polarity) and with inverse conditional probabilities (viceversa). Conditional probabilities give us information about how much posts p classified with a contain polarity pol contain emoticons with the same polarity: P (adj = pol 2 p j p = pol) = #(p = pol ^ adj = pol 2 p) #(p = pol) where P stands for Probability.
Results (Table VI) generally show that the polarity of adjectives in messages agrees with the overall message polarity: this leads a positive message to have positive adjectives with a higher probability. Moreover, also inverse conditional probabilities have been calculated: P (p = pol j adj = pol 2 p) = #(p = pol ^ adj = pol 2 p) #(adj = pol 2 p) as the ratio between the number of messages p with polarity pol that contain adjectives with polarity pol and the number of messages that contain adjectives with polarity pol. The estimation of conditional probabilities and inverse conditional probabilities leads us to state the agreement between the adjective and message polarities.
Table IV shows the emoticons distribution on the three studied datasets. Positive and negative messages, as expected, have a high percentage of emoticons.
In order to verify that emoticons could be an important source of information for polarity classification (as well as adjectives), a further and detailed analysis has been conducted conditioning the emoticons presence to the message polarity and viceversa with inverse conditional probabilities. Conditional probabilities give us information about how much posts p classified with a contain polarity pol contain emoticons e with the same polarity:
P (e = pol 2 p j p = pol) = #(p = pol ^ e = pol 2 p) #(p = pol) Results generally show that the polarity of emoticons in messages agrees with the message polarity: this leads a positive message to have positive emoticons with a higher probability. Moreover, as well as for adjectives, the inverse conditional probabilities have been calculated:
P (p = pol j e = pol 2 p) = #(p = pol ^ e = pol 2 p) #(e = pol 2 p) as the ratio between the number of messages p with polarity pol that contain emoticons e with polarity pol and the number of messages that contain emoticons with polarity pol. Both probabilities further confirm the agreement between emoticons and message probabilities.
An further analysis has been performed on the stretched word distribution for the three studied datasets (Table V). First of all, as expected, positive and negative messages have higher percentages of stretched words than neutral messages (even if messages which contain stretched words are very few). In order to verify that stretched words could be an important source of information for polarity classification, a further and detailed analysis has been conducted conditioning the stretched words presence to the message polarity and viceversa with inverse conditional probabilities.
Conditional probabilities (and inverse conditional probabilities) are calculated as shown above for adjectives and emoticons. Supported from the analyzed data, we can conclude that stretched words have a high correspondence with positive and negative polarities.
In this section the performance achieved from the studied classifiers on the configurations of the Merged dataset are presented (Table I), since the Movie and Person datasets present few instances that could be not statistically significant (Figure 2). To this purpose, we measured Precision (P ), Recall (R) and F1-measure, defined as
P =
T P T P +F P
R
T P = T P +F N
F 1 = 2 P R
P +R for the positive, neutral and negative labels. We also measured Accuracy, defined as
Acc =
T P + T N
T P + F P + F N + T N
Figure 3 shows the final classification accuracy of each studied classifier on the Merged dataset6. The configurations C-FE and PC-FE lead to increments of approximately 2% of accuracy. In some cases, increments achieve 5%. Moreover, results of the two configurations are usually statistically significant (i.e. results are not randomly achieved). However, text normalization does not lead to significant improvements and results are not statistically significant. Regarding classifier performance, Multinominal Naive Bayes and SVM achieve the highest results.
VI.
CONCLUSION
In this work, we proposed a system aimed at classify the polarity of messages on social media. We formulated different assumptions regarding what elements within a message can be relevant sentiment indicators. The first assumption states that the occurrences of emoticons representing a certain polarity could strongly agree with the overall message polarity. As 6Results on the other datasets are similar. well as expanding the feature space including emoticons, we assumed that also adjectives and stretched words, extensively used in social media messages, could be useful information to help in determining the sentiment. To the best of our knowledge, no studies consider the combination of adjectives, initialisms for emphatic and onomatopoeic expressions, emoticons and stretched words as possible additional features. Subsequently, detailed analyses have been performed in order to verify our assumptions. For each studied dataset, four different configurations have been considered to measure the improvements led from each component (not preprocessed content and no additional features, not preprocessed content but additional features, preprocessed content but not additional features and preprocessed content with additional features). The supervised classifiers used in the system are Naive Bayes (NB), K-Nearest Neighbors (K-NN), Support Vector Machine (SVM) and Decision Trees (DT). Several experiments show that text normalization does not lead to significant improvements but expanding the feature space of the traditional bagof-words model with the considered features lead to accuracy increments up to 5%. Regarding classifier performance, Multinominal Naive Bayes and SVM achieve the highest results.
This work has been partially supported by the I-ShErPA project. K. Fujimoto, “A computational account of potency differences in ewom messages involving subjective rank expressions,” in Web Intelligence