Online Reputation Management is a novel and active area in Computational Linguistics. Closely related to opinion mining and sentiment analysis, it incorporates new features to traditional tasks like polarity detection. In this paper, we study the feasibility of applying complex sentiment analysis methods to classifying polarity for reputation. We adapt an existing emotional concept-based system for sentiment analysis to determine polarity of tweets with reputational information about companies. The original system has been extended to work with texts in English and in Spanish, and to include a module for ltering tweets according to their relevance to each company. The resulting UNED system for pro ling task participated in the rst RepLab campaign. The experimental results prove that sentiment analysis techniques are a good starting point for creating systems for automatic detection of polarity for reputation.
Part of eMarketing, Online Reputation Management (ORM) has already become
an essential component of corporate communication for public gures and large
companies [
In order to study a brand image, reputation management consultancies perform two main tasks: monitoring and pro ling. As its name suggests, the former consists in a constant (daily) monitoring of online media, seeking and analysing information related to the entity, with the objective of detecting any topic that might damage its image. In contrast, pro ling refers to a single or periodic (e.g., monthly) revision of a company's reputation as it distils from news, opinions and comments expressed in social media or online press. Unlike monitoring, which is ? This research was supported by the European Unions (FP7-ICT-2011-7 - Language technologies - nr 288024 (LiMoSINe).) essentially a real-time problem, pro ling is a static study of opinions and polar facts concerning a certain entity and extracted for a given period. Normally, this information is contrasted with what has been said in the same period of time about the company's potential competitors, and with the opinions about the entity in earlier periods of time. These practical scenarios have been adopted as tasks for RepLab 2012, the rst evaluation campaign for ORM systems1. In this paper, we will focus exclusively on the pro ling task.
Although for reputation analysts pro ling implies a complex of subtasks such as identifying the dimension of the entity's activity a ected by a given content, detecting opinion targets and determining the type of opinion holder2, the basis of adequate pro ling is undoubtedly an e ective named entity disambiguation and detection of polarity for reputation. The system described in the present paper centres precisely on the problem of classifying tweets into related and unrelated with respect to a given company ( ltering) and on determining the polarity of tweets based on the emotion concepts they contain.
Some tasks considered in pro ling are similar to the research problems in
opinion mining and sentiment analysis that comprise subjectivity detection [
Identifying polarity of a given content, ORM system should assess if it has negative, positive or neutral e ect on the company's image. Again, this problem is related to sentiment analysis and opinion mining, but signi cantly di ers from the mainstream research in these areas. First of all, what is analysed are not only opinions, subjective content, but also facts, and more to the point, polar facts, i.e. objective information that might have negative or positive implications for the company's reputation. This means that ORM systems have to be able to detect polarity also in non-opinionated texts. On the second place, focus or perspective plays sometimes a decisive role, since the same information may be negative from the point of view of clients and positive from the point of view of investors. We will refer to this complex notion of polarity with the term polarity for reputation.
In this paper, we study the feasibility of using complex sentiment analysis approaches to classifying polarity for reputation. To this end, as exposed in Section 2, we adapt an existing emotional concept-based system for sentiment analysis to classify tweets with reputation information about companies. The system is also extended to lter tweets according to their relevance to each
2 Both companies' dimensions and the types of opinion holder are standard parameters of RepTrak System http://www.reputationinstitute.com/thought-leadership/ the-reptrak-system. company and works with texts both in English and in Spanish. It is worthy to mention that one of the applied approaches combines our runs with the algorithm provided by Barcelona Media (see Section 3). Our experimental results, described in detail in Section 4, demonstrate that sentiment analysis techniques are a good starting point to process and classify online reputation information. Finally, after a brief discussion of the obtained results (see Section 5), we outline some lines for future work in Section 6. 2
An emotional concept-based approach for polarity for
reputation
As mentioned above, our main concern is to analyze the applicability of sentiment
analysis techniques for classifying the polarity of reputation information. To this
aim, we have adapted the approach presented in [
In order to determine if a text is related to a given entity we have implemented a simple vote mechanism that calculates a score depending on how many entity context words are found in the input text. The entity context is obtained from the entity website and from the entity entry in Wikipedia, as provided by RepLab. An input text is determined as related or not to a given entity when the nal score is upper a certain threshold. Di erent thresholds have been evaluated, changing from the simple presence of the search query to mentions of the complete entity name or to the presence of more entity context words. Our main objective for this approach is to determine if a simple method of word presence is able to correctly determine the relatedness of the text to a given entity. It is also important to highlight the complexity of the task, since even for humans it is often di cult to decide if a text is ambiguous or not with the respect to a given entity due to the lack of context in the input text. This problem is more evident in microblogging systems such as Twitter, where the text of the post is limited to 140 characters.
As a rst step, the system pre-processes each input text splitting it in
sentences and isolating the tokens of each sentence using the GATE architecture [
The score of each input text is then compared to the threshold to determine if the text is related to the entity or not, ltering all the input texts that are not related to the entity. 2.2
The original method presented in [
Emotion Identi cation Once the concepts are identi ed, the next step maps
each WordNet synset to its corresponding emotional category in the SentiSense
a ective lexicon, if any. The emotional categories of the hypernyms are also
retrieved. We hypothesize that the hypernyms of a concept entail the same
emotions than the concept itself, but decreasing the intensity of the emotion as we
move up in the hierarchy. So, when no entry is found in the SentiSense lexicon
for a given concept, the system retrieves the emotional category associated to its
nearest hypernym, if any. However, only a certain level of hypernymy is accepted,
since an excessive generalization introduces some noise in the emotion identi
cation. This parameter has been empirically set to 3. In order to accomplish this
step for Spanish texts we have automatically translated the SentiSense lexicon
to the Spanish language. To do this, we have automatically updated the synsets
in SentiSense to their WordNet 3.0 version using the WordNet mappings. In
particular, for nouns and verbs we use the mappings provided by the WordNet
team [
To this end, our system rst identi es the presence of modi ers using a list of common negation and intensi cation tokens. In such a list, each intensi er is assigned a value that represents its weight or strength. The scope of each modi er is determined using the syntax tree of the sentence in which the modi er arises. We assume as scope all descendant leaf nodes of the common ancestor between the modi er and the word immediately after it, and to the right of the modi er. However, this process may introduce errors in special cases, such as subordinate sentences or those containing punctuation marks. In order to avoid this, our method includes a set of rules to delimit the scope in such cases. These rules are based on speci c tokens that usually mark the beginning of a di erent clause (e.g., because, until, why, which, etc.). Since some of these delimiters are ambiguous, their POS is used to disambiguate them. Once the modi ers and their scope are identi ed, the system solves their e ect over the emotions that they a ect in the text. The e ect of negation is addressed by substituting the emotions assigned to the concepts by their antonyms. In the case of the intensi ers, the concepts that fall into the scope of an intensi er are tagged with the corresponding percentage weight in order to increase or diminish the intensity of the emotions assigned to the concepts.
In particular, for English texts we use the original list of negation signals
from [
Classi cation In the last step, all the information generated in the previous steps is used to translate each text into a Vector of Emotional Intensities (VEI), which will be the input to a machine learning algorithm. The VEI is a vector of 14 positions, each of them representing one of the emotional categories of the SentiSense a ective lexicon. The values of the vector are generated as follows: { For each concept, Ci, labeled with an emotional category, Ej , the weight of the concept for that emotional category, weight(Ci; Ej ), is set to 1.0. { If no emotional category was found for the concept, and it was assigned the category of its rst labeled hypernym, hyperi, then the weight of the concept is computed as: weight(Ci; Ej ) = 1=(depth(hyperi) + 1) (1) { If the concept is a ected by a negation and the antonym emotional category, Eantonj , was used to label the concept, then the weight of the concept is multiplied by = 0:6. This value has been empirically determined in previous studies. It is worth mentioning that the experiments have shown that values below 0.5 decrease performance sharply, while it drops gradually for values above 0.6. { If the concept is a ected by an intensi er, then the weight of the concept is increased/decreased by the intensi er percentage, as shown in Equation 2. weight(Ci; Ej ) = weight(Ci; Ej ) (100 + intensif ier percentage)=100 (2) { Finally, the position in the VEI of the emotional category assigned to the concept is incremented by the weight previously calculated. 3
Heterogeneity based ranking approach for polarity for
reputation
Our last approach consists in combining our runs (Uned 1..4) with the runs
provided by Barcelona Media (BMedia 1..5) [
We propose a voting algorithm that directly considers the diversity of
classiers instead of the amount of classi ers that corroborate a certain decision. As
far as we know, this perspective has not been applied before due to the lack of a
diversity measure to be applied over classi er sets rather than pairwise measures.
Our approach is inspired in the Heterogeneity Based Ranking [
We de ne the Heterogeneity of a set of classi ers F = ff1::fng as the probability over decision cases C that there exists at least two classi ers contradicting each other.
H(F ) = PC(9fi; fj 2 F =fi(c) 6= fj (c))
The approach basically consists in selecting the label (e.g. positive, neutral) that maximizes the heterogeneity of classi ers corroborating this decision.
The data set used for evaluation consists of a training set of 1800 tweets crawled for six companies (300 per company) and a test set of 6243 tweets for 31 companies. However, since Twitter's Terms of Service do not allow redistribution of tweets, some of them have been removed from the release version of the data set. So, the training set that has been nally used contains 1662 tweets manually labeled by experts, 1287 for English and 375 for Spanish. For the ltering task, the training set comprises 1504 tweets related to any of the companies and 158 that were not related to any company. In polarity for reputation, the distribution of tweets between classes is: positive=885, neutral=550, and negative=81. The test set contains tweets manually labeled by experts, 3521 of them are in English and 2722 in Spanish. Besides, this set contains 4354 related tweets, and 1889 unrelated tweets, while the distribution of tweets in polarity for reputation classes is 1625, 1488 and 1241 tweets for positive, neutral and negative, respectively. For the pro ling task (i.e., combining systems of ltering and polarity), the performance is evaluated as accuracy on a four-class classi cation problem: irrelevant, relevant negative, relevant neutral and relevant positive. For the individual evaluation of the ltering and polarity tasks, performance is evaluated in terms of reliability (R) and sensitivity (S)3. Accuracy (% of correctly annotated cases) is also included for comparison purposes. Overall scores (R, S, F(R,S), accuracy) are computed as the average of individual scores per entity, assigning the same weight to all entities.
As the system for polarity for reputation is a supervised method, we have
tested di erent machine learning algorithms with the training set in order select
the best classi er for the task. To determine the best machine learning algorithm
for the task, 20 classi ers currently implemented in Weka [
Based on these considerations, ve systems have been presented to RepLab 2012: Uned 1 (Logistic + threshold=0.5 ), Uned 2 (RandomForest + threshold=0.5 ), Uned 3 (Logistic + threshold=0.75 ), Uned 4 (RandomForest + threshold=0.75 ), and Uned 5 or Uned-BMedia, which is the system described in the sec
tion 3 that combines the outputs of the algorithms Uned 1..4 and BMedia 1..5 using the heterogeneity-based ranking approach.
Table 1 shows the results of our systems for the ltering task when evaluated
over the test set. As can be seen, the best performance in terms of F(R,S) is
obtained by the two approaches that use the 0.75 threshold (Uned 3 and Uned 4),
followed by the combined version Uned-BMedia and the two approaches that use
the 0.5 threshold (Uned 1 and Uned 2). In terms of accuracy, the best result is
obtained by the combined version Uned-BMedia, only 2.0 percentage points more
than the two approaches that use the 0.75 threshold and 3.1 percentage points
more than the 0.5 threshold approaches. Comparing these results with the All
relevant baseline, it is evident that the performance of our approaches is quite
acceptable but may be easily improved. It is important to recall the di culty of
the task even for humans. In fact, the best results obtained in the challenge by
our systems, Uned 3 and Uned 4, are placed 13th and 14th, respectively, out of
33 participants [
Table 2 summarizes the results obtained by our systems in terms of accuracy, reliability and sensitivity over the test set. The best performance in terms of F(R,S) is achieved by the combined version Uned-BMedia, while the systems Uned 2 and Uned 4 that use the RandomForest classi er are only 4.0 percentage points lower. The systems Uned 1 and Uned 3 that use the Logistic classi er achieve 8.0 percentage points less than the Uned-BMedia, which is an important di erence in performance. In contrast, in terms of accuracy the best performance is obtained by the RandomForest classi er (Uned 2 and Uned 4), followed by the combined version Uned-BMedia and the Logistic classi er. As can be seen in the table, all the systems improved over the All positives baseline. It is worth noticing that the results achieved for the polarity for reputation task are much better than the obtained for ltering task. This is evidenced by the fact that the combined system Uned-BMedia obtained the 2nd position within the 35 participating systems in term of F(R,S), and the two approaches that use the RandomForest classi er obtained the 5th and 6th position, respectively. However, in terms of accuracy the results reported by our systems are even better, the two approaches that use the RandomForest classi er have achieved the 1st and 2nd position in the ranking, while the combined version Uned-BMedia has achieved the 4th position.
The results of our systems for the pro ling task, i.e. combining ltering and polarity for reputation, are shown on Table 3. The best result is achieved by the system Uned 4. However, the di erence is not so marked with respect to the Uned 3 and the combined version Uned-BMedia, 0.6 and 1.9 percentage points of accuracy, respectively. The di erence becomes higher comparing to the systems Uned 2 and Uned 1, 5.6 and 7.2 percentage points of accuracy. Moreover, all systems considerably improve the performance over the All relevant and positives baseline. As in the polarity for reputation task, the results achieved by our systems compare favorably to those of other participants. In particular, three of our systems are among six best systems out of 28 systems that participated in the task (the 3rd, the 5th and the 6th for Uned 4, Uned 3 and Uned-BMedia, respectively), which proves that the proposed approaches are a good starting point for a more complex pro ling system. Analysing and discussing the results obtained by the systems proposed for RepLab 2012, rst of all and regarding the ltering task, we have to say that our system correctly classi ed 3352 out of 4354 related tweets (76.9%) and 1019 out of 1831 non-related tweets (55.6%). These results suggest that the 0.75 threshold in the vote mechanism is a good choice for discriminating between related and unrelated tweets. Most of the related tweets were above this threshold, and nearly half of the unrelated tweets were below it. However, a number of unrelated tweets obtained a score far above 0.75, which seems to suggest that the ltering method should take into account not only positive terms but also negative ones, i.e. terms that decrease the score. We also have analyzed which are the most frequent combination of rules for assigning scores, as well as their adequacy. We have found that rule II (i.e. the one that considers the presence in the input of the individual tokens within the search query) and rule III (i.e. the one that looks for the search query without blanks), are frequently launched together producing satisfactory results. The combination of rule III and rule IV (i.e. the presence of entity context words) is the second most frequent one. However, it is also the one that introduces more noise. Finally, the presence of the complete entity name (rule I) is the less launched rule, but the one that produces the best performance, as expected.
Regarding the polarity for reputation task, our best systems in terms of accuracy (Uned 2 and Uned 4) correctly classi ed 822 tweets as positive, 810 tweets as neutral and only 179 tweets as negative. These results show the good performance of these systems at identifying the positive and neutral classes. In contrast, the systems only classify 15% of negative tweets correctly. This di erence may be due to the fact that the number of negative instances in the training set is not big enough for the classi ers to correctly learn this class.
Even if the systems achieve a good performance in polarity for reputation, it is worth mentioning that these results are lower than those reported by the same systems when evaluated with other datasets containing product reviews. In order to determine the reason of this, we analyzed the datasets and found that an important number of tweets are not labeled with any emotion. In particular, the coverage of the vocabulary in SentiSense for the training set is 12.5% for English and 12.6% for Spanish, while the coverage in the test set is 11.3% and 11.2% for English and Spanish, respectively. This was expected, since SentiSense is specially designed for processing product reviews. Therefore, taking this low coverage into account, we expect that expanding SentiSense with reputationrelated vocabulary will allow us to signi cantly improve the classi cation results.
However, as we suspected, another important source of classi cation errors is that many of the tweets labeled with polarity in the dataset (positive and negative) do not contain any emotional meaning per se. These are factual expressions that, from a reputation perspective, entail a polarity, and therefore must be classi ed as negative or positive. For example, the tweet I bough a Lufthansa ticket to Berlin yesterday does not express any clear emotion, but is considered as positive for reputation purposes. To this aim, reputation management knowledge must be used to allow the system to correctly interpret these expressions.
The importance of the opinion holder and, specially for microblogging, the importance of the external links in the input text are other two important ndings of our analysis. On the one hand, we nd some examples of tweets that, from our point of view, should be classi ed as neutral, but were classi ed by the experts as polar due to the relevance or popularity of the the tweet's author. So, we can conclude that correctly determining the opinion holder is important to weight the nal polarity for reputation. On the other hand, we nd that many of the tweets that have external links obtain their polarity from the linked documents. This suggests that studying polarity for reputation in social media, especially in such microblogging services as Twitter, needs more context than the posts themselves.
Finally, the results obtained in the pro ling task show the good adequacy of complex sentiment analysis approaches as a starting point to analyze texts according to the reputation of a given entity. Even if the performance of the combined system, ltering + polarity for reputation, is quite acceptable, there is still room for improvement. It is important to notice that the evaluation of this task includes, rst, ltering relevant tweets, and next, the polarity classi cation of these relevant tweets, so that the error of the ltering step is propagated to the polarity for reputation classi cation. 6
In this paper, we have presented a system to address the task of classifying polarity for reputation, based on sentiment analysis techniques. We also face the problem of determining whether a text is related or not to a given entity (e.g., company or brand). The results showed that, even if the task is more complex than classifying polarity in opinionated texts, the use of complex sentiment analysis approaches seems a good basis for classifying polarity for reputation. However, it is still necessary to incorporate a high level of expert knowledge to correctly analyze the reputation of a company.
As future work, we plan to develop a system for separating objective from subjective statements. This separation will allow us to study objective texts from a reputation perspective, while subjective opinions will be analyzed using sentiment analysis approaches. Our analysis has also revealed the importance of detecting the opinion holder, which may in uence the polarity of the text. To address this, we plan to study existing approaches in the area and evaluate them for the polarity for reputation task. Finally, we would like to extend the SentiSense a ective lexicon to cover more domain speci c vocabulary.