Sentiment analysis aims at detecting sentiment polarities in unstructured Internet information. A relevant part of this information for that purpose, emojis, whose use in Twitter has grown considerably in these years, deserves attention. However, every time a new version of Unicode is released, nding out the sentiment users wish to express with a new emoji is challenging. In [KNSSM15], an Emoji Sentiment Ranking lexicon from manual annotations of messages in di erent languages was presented. The quality of these annotations a ects directly the quality of possible generated emoji sentiment lexica (high quality corresponds to high self-agreement and inter-agreement). In many cases, the creators of the datasets do not provide any quality metrics, so it is necessary to use another strategy to detect this issue. Therefore, we propose an automatic approach to identify and manage inconsistent manual sentiment annotations. Then, relying on a new approach to generate emoji sentiment lexica of good quality, we compare two such lexica with lexica created from manually annotated datasets with poor and high qualities.
Following a trend in the last years, emojis are being increasingly used in social applications. For example, 1% of the messages in a random sample of 22.14 billion tweets taken between July 2013 and March 2018 contained at least one emoji1.
Emojis allow users to express feelings and emotions. Thus, it is interesting to try to extract from them useful knowledge on user opinions [HTGL13]. Natural Language Processing (nlp) allows us to analyze opinions, feelings, assessments, etc. on products, services or organizations [Liu12]. Until very recent times, researchers in the eld of sentiment analysis (sa) only considered the information contributed by emoticons [BFMP13, DTR10, HBF+15]. Nevertheless, nowadays emojis are attracting considerable attention [GOB16, HGS+17]. For this reason, some recent studies have tried to obtain the sentiment expressed by emojis in the form of a lexicon [KNSSM15, LAL+16, KK17]. In many cases, however, the expected meaning of an emoji (in terms of positivity, neutrality or negativity), which is assumed to be universal, may changes among languages and cultures [BKRS16].
Following this line, in [KNSSM15] the authors presented an Emoji Sentiment Ranking (esr)2, resulting from texts in 15 di erent languages containing emojis, whose sentiments were labeled manually by different human annotators over three months. However, the quality of manual labeling, measured in terms of self-agreement and inter-agreement as explained in [MGS16], may be poor.
1http://www.emojitracker.com/api/stats 2Available at https://goo.gl/XEkJhZ
We can suppose that, if an emoji sentiment lexicon is generated from one of these single-language datasets, the most popular emojis should be highly correlated with those obtained from the overall esr when the quality of manual labeling, measured in terms of selfagreement and inter-agreement, is acceptable (di erences would be mainly due to emojis with di erent interpretations among languages). On the contrary, if at least one of these metrics is low, inconsistencies in manual sentiment annotations should be suspected, and the correlation would be seriously a ected (the di erences in emoji interpretations would be much greater). When these measurements are not provided by the dataset creators or they are unknown, an alternative should be sought to identify the inconsistencies. The nal objective should be to create an emoji sentiment lexicon with the highest possible quality.
In this paper, we propose an approach to detect low-quality dataset annotations. In case of inconsistent annotations, we also present a fully automated approach to obtain emoji lexica with good quality.
The rest of the paper is organized as follows: Section 2 reviews related work on emoji sentiment analysis. Section 3 discusses the issue of labeling quality. Section 4 describes the proposed method. Section 5 presents experimental results. Finally, Section 6 summarizes the main contributions and conclusions. 2
Even though emoji sentiment interpretation (where sentiment is expressed as a positive, neutral or negative polarity) has already been studied in the eld of nlp, a common practice in the case of Twitter was to lter Unicode symbols during message preprocessing, so that emojis' information was lost [TK16]. But, for example, in the message \Today I have to go to the supermarket ", the obvious negativity is given by the emoji.
Focusing on methods to guess the real sentiment of emojis, they can be classi ed in three types: manual, semi-automatic and automatic. Regarding manual methods, in [MTSC+16] the most popular Unicode emoji characters were manually labeled by multiple annotators, taking into account sentiment (positivity, neutrality and negativity) variance as well as semantics (meaning). In [KNSSM15], 83 native speakers of di erent languages labeled by hand the sentiment (positive, neutral or negative) of texts containing 751 di erent emojis. The authors calculated their sentiment based on their occurrences and the manual labels of the tweets containing them, by applying a discrete probability distribution. Finally, in [ELW+16] 78 strongly and 34 weakly subjective emojis were extracted from the list [KNSSM15] and given polarity values of +2, +1, -1 and -2 (strongly positive, weakly positive, weakly negative and strongly negative, respectively).
Currently, few approaches assign polarities to emojis with semi-automatic or automatic methods. In [HTAAAK16] the most used emojis in a dataset of Arabic tweets were classi ed into four categories: anger, disgust, joy and sadness. Subsequently, they were weighted with scores between -5 and +5 (most negative and most positive, respectively), according to those categories. The weights were obtained from the afinn lexicon [Nie11], in which some entries are emojis. The Unicode short Common Locale Data Repository3 (cldr) names of the missing emojis were obtained and the words composing them were searched in afinn (one by one, independently). Finally, weights were also manually assigned according to the category of each emoji.
Regarding the approaches that obtain emoji sentiment lexica in a fully unsupervised way, we are only aware of the following examples. In [LAL+16], the authors analyzed emoji usage in text messages by country. In total, the sentiment of 199 emojis was obtained from their short cldr names processed with the liwc4 tool (which counts words that express positive, neutral or negative sentiment). This analysis did not exploit their real descriptions or their usage contexts. In [KK17], the authors extracted, for each word of a tweet that co-occurred with a target emoji, the set of synonyms or synsets available in WordNet 5. Then they recovered the most frequent a ective label from WordNet-A ect 6. Five sentiment categories were di erentiated: happiness, disgust, sadness, anger and fear, following a hierarchical structure. Finally they calculated a sentiment score vector for 236 emojis based on the mentioned co-occurrences. Again, this analysis also ignored the real descriptions or the usage contexts of the emojis. Finally, in [FGJMGM+18], a lexicon of 840 emojis was created using an unsupervised sa system, taking only into account emoji de nitions in Emojipedia7. This lexicon was then improved in di erent variants that took advantage of the sentiment distribution of informal texts including emojis. 3
In general, a given emoji should have the same emotional meaning in di erent datasets written in the same language. This implies that emoji sentiment interpretation for each of them should be very close to 3http://unicode.org/emoji/charts/emoji-list.html 4https://liwc.wpengine.com/ 5https://wordnet.princeton.edu/ 6http://wndomains.fbk.eu/wnaffect.html 7https://emojipedia.org/ the interpretation for all datasets together. The problem arises when an emoji sentiment lexicon is created from multilingual datasets with manual sentiment annotations that are inconsistent for a language or some languages.
On the other hand, it seems logical to think that an emoji should have di erent emotional meanings across di erent languages and cultures. Nevertheless, according to [BKRS16], the semantics of the most popular emojis are strongly correlated most of the time in most languages in that regard. This was an interest nding, because both the vocabularies of the languages and the context words modeled by the semantic spaces are different. The authors stated that English and Spanish speakers interpret emojis in the most universal way, with a high correlation with all other languages, although strong di erences may persist for some emojis. In this way, the sentiment of the most popular emojis in a particular language may di er from the \universal sentiment", but they should be close in most cases.
Our main contributions are a method to detect anomalies in emoji sentiment lexicon due to inconsistent annotations and an alternative automatic approach to predict emoji sentiments with applications in emoji sentiment lexica generation. 4
We rst present a method for constructing automatically two emoji sentiment lexica [FGJMGM+18] (Figure 1). Summing up, (1) emojis are extracted from a set of informal texts and their descriptions are acquired from the Emojipedia repository. Then, (2) nlp techniques capture their linguistic peculiarities from both the descriptions and the informal texts, which are exploited independently by an unsupervised sa system with sentiment propagation across dependencies (usspad) described in [FGALJM+16]. Depending on the combination of the polarities obtained from the sa, (3) two emoji sentiment lexica variants are created. In this regards, we remark that our aim is not a novel sa approach.
In Figure 1, the dotted arrow in the upper left corner represents the actions to gather a set of informal texts with emojis. The solid arrows represent the processes carried out on these texts to obtain the rst emoji sentiment lexicon from an emoji sentiment ranking, from automatically labeled texts where they occur. The dashed arrows refer to the case in which a similar process is previously applied on each individual emoji description (extracted from Emojipedia), to obtain an initial emoji sentiment lexicon from the universal de nitions by emoji creators. This lexicon, unsupervisedemojiDef, is later applied as extra information into each particular informal text, to assign sentiment labels automatically and then obtain the second lexicon through the same emoji sentiment ranking. Next, we explain the method in more detail. 4.1
In order to extract emoji de nitions, messages must be converted to a Unicode representation and regular expressions must be used for the extraction8. Then, each emoji Unicode codepoint in hexadecimal notation is converted to UTF-8 hex bytes and submitted via a get request9 to the Emojipedia resource to retrieve its 8This process was carried out using the Emoji-java library, available at https://github.com/vdurmont/emoji-java. 9http://emojipedia.org/search/?q=.
English description, which is parsed through jsoup10.
5.1 At this point, the method performs sa on both the informal texts containing the emojis and their de nitions. This consists of two main tasks: preliminary data treatment with lexical and syntactic analysis; and capturing linguistic peculiarities and applying usspad sa [FGALJM+16]. In it, the nal sentiment results from the propagation of sentiment term values (included in a sentiment lexicon) from the leaves to the parent nodes of each dependencies tree. Once these steps are completed, a polarity score is assigned to each informal text and emoji description, and emoji sentiment lexica can be created. Once all previous steps have been performed on informal texts and descriptions, we are in a position to apply two di erent approaches to exploit polarity scores of texts and de nitions, and create two emoji sentiment lexica. In the rst variant (E1), the lexicon is created considering the ranking of polarity scores assigned to texts with emojis, applying the estimations in [KNSSM15]. That is, following the solid arrows in Figure 1 we obtain Runsupervised. The second variant (E2) considers extra information. Lexicon unsupervisedemojiDef is created from sentiment scores obtained through automatic sentiment propagation on emoji de nitions. These values are then included in the sentiment lexicon used in Section 4.2 to improve the sa of informal texts and obtain new polarity scores for them. Finally, the same estimations in [KNSSM15] are applied to the resulting unsupervised sets. That is, following the dashed arrows in the gure, to obtain
4.4
Given the hypothesis that the sentiments of the most popular emojis are preserved across di erent languages, and that only a small percentage of them show language-speci c usage patterns [BKRS16], we assume that the correlation between the entries of an emoji sentiment lexicon created for a particular language and the entries of a multilingual emoji sentiment lexicon (ideally a universal lexicon) should be high. This is the base for the experiments in Section 5.2. We used the annotated datasets in [KNSSM15] in 15 di erent languages including Albanian, English, Polish and Spanish, among others. These datasets are available at the public clarin11 language resource repository. The entry for each labeled tweet consists of a tweet ID, a sentiment label (negative, neutral or positive) and an anonymized annotator ID. We focused on the four datasets in Table 1, discarding tweets without emojis and tweets with ambiguities among annotators. The authors reported good self-agreement (Alphas) and inter-agreement (Alphai) values for English and Polish and worse values for Albanian and Spanish.
Dataset
Albanian Alphas = 0:447 Alphai = 0:126
English Alphas = 0:739 Alphai = 0:613
Polish Alphas = 0:757 Alphai = 0:571
Spanish Alphas = 0:245 Alphai = 0:121 #emojis 48 624 369 613
Label Negative Neutral Positive Negative Neutral Positive Negative Neutral Positive Negative Neutral Positive Table 2 shows the correlations for positive, negative and neutral labels between the conventional esr lexicon (Rannotatedall , created using the method in [KNSSM15] from messages in 15 languages annotated by hand) and each emoji sentiment lexicon, which was created in the same way for a single language (Rannotateden for English, for instance). For a fair analysis, given the detection criterion, to calculate the correlation we considered the top 100 occurring emojis in each language lexicon as the most popular.
Looking at Table 2, score and ranking level correlations are high for English and Polish (Rannotatedpo ). Moreover, looking at Figures 2a and 2b, the associated linear regressions (represented with solid lines) have slightly less slope than the regression for the overall case that serves as gold-standard (represented with a dotted line). This suggests that the English and Polish datasets have consistent annotations, as 10Available at https://jsoup.org/ (b) Plot for top 100 emoji sentiment scores comparing
(c) Plot for top 100 emoji sentiment scores comparing
(d) Plot for 48 emoji sentiment scores comparing
evidenced by their good Alphas and Alphai values in [KNSSM15, MGS16].
However, when we compared the overall Rannotatedall lexicon with the Spanish and Albanian lexica (Rannotatedes and Rannotatedal ), score and ranking correlations were worse. Indeed, in Figures 2c and 2d, the linear regression slopes are very at, and therefore they move far from the overall case. This suggests that the Spanish and Albanian datasets have inconsistent manual annotations (as shown by Alphai=0:121 and Alphas=0:245 for Spanish and Alphai=0:126 for Albanian) [KNSSM15, MGS16]. In addition, if we focus on Figure 2c, a vast majority of emoji dots have positive polarity in the Spanish lexicon (X axis) while, for the overall case, polarities vary between positive and negative. Once we are able to detect annotation anomalies, we also have a methodology to validate an alternative solution to generate lexica automatically. We veri ed it on English and Spanish datasets as representative cases of which we have good and bad manual annotations, respectively. Two sentiment emoji lexica were created per language, corresponding to variants E1, which only considers the automatic usspad annotation (E1es and E1en), and E2, which also considers Emojipedia de nitions (E2es and E2en). Subindex's es and en denote Spanish and English, respectively.
Lexicon x
Lexicon y E1en E2en E1es E2es
Rannotateden
Rannotateden
Rannotatedes
Rannotatedes Rannotatedall rscore(x, y) rrank(x, y) 82.91% 76.20% 79.70% 75.25% 83.72% 86.90% 47.19% 74.93% 30.06% 81.32% 79.37% 80.71% 47.18% 74.78% 44.09% 79.07% In Table 3, if we compare the English variants, we (a) Correlation between E1en and Rannotateden (b) Correlation between E1en and Rannotatedall observe that the lexica are highly correlated. Introducing the e ect of emoji de nitions, correlation increases from E1en to E2en compared both with Rannotatedall and Rannotateden . This is clear in Figures 3a and 3c, where the line that serves as gold-standard and the regressions intersect at neutral emoji sentiments. However, in Figures 3b and 3d these lines intersect respectively at positive and neutral emoji sentiments. This shows that the de nitions balance sentiments in the second variant.
On the other hand, given the fact that the emoji sentiment lexicon obtained from a manually annotated Spanish dataset Rannotatedes has poor quality due to annotation inconsistencies [MGS16], as con rmed by their authors and by Table 2 and Figure 2c in Section 5.2, its correlation with the automatic variants should also be low. This is veri ed in Table 3 for E1es and E2es. The better behavior of E1es in this case is not relevant, due to the anomalies in Rannotatedes . However, in the comparisons with Rannotatedall , the correlation with E2es is higher both for ranking and score, as shown in Figures 4a and 4b, which is coherent with the observations for English. (c) Correlation between E2en and Rannotateden (a) Correlation between E1es and Rannotatedall (d) Correlation between E2en and Rannotatedall (b) Correlation between E2es and Rannotatedall Rannotatedall is biased by typical emoji usage worldwide and, to a lesser extent, by the vision of the annotator, who writes in a particular language. For this reason, we might worry about the in uence of particular language subsets in the overall lexicon. Therefore, an independent evaluation of the generated emoji sentiment lexica is necessary.
Our objective here is to determine if our lexica variants for Spanish and English are good enough in a real-world scenario, by evaluating their impact with sa metrics (precision (Pmacro), recall (Rmacro) and F (Fmacro) macroaverages on the positive and negative classes).
In principle, in the Spanish subset this is impeded by bad labeling. We assumed that only a small percentage of the most popular emojis had signi cant sentiment di erences between languages. For most properly annotated messages containing the top popular emojis, we could thus assume that any lexica should provide similar results. Therefore, we decided to restrict the sa test to the 100 most popular emojis in Spanish and English. Then we only selected the messages in the English dataset where those emojis occurred (English B). This new dataset had a distribution with 3552 positive, 1998 negative and 1601 neutral messages. Table 4 shows the results.
Our assumptions are validated by these results, sorted by Pmacro. The ordering is coherent with our expectations. Rannotateden was created from consistent manual annotations, but E2en only performs a bit worse. If we compare E1en with E1es, on the one hand, and E2en with E2es, on the other, their performances are comparable bit for small percentages that can be explained by the the small percentage of \top" emojis whose sentiment is not preserved across languages. An important nding is that our automatic approach performs satisfactorily compared to a lexicon produced from a well-annotated dataset. [ELW+16]
A poorly labeled dataset (yielding low self-agreement and inter-agreement) may a ect directly the quality of emoji lexica. In many cases the annotators do not publish any quality metrics, so it is di cult to determine beforehand if bad sa performance is due to the supporting lexicon or to the sa technique itself. In this paper we have proposed a method to detect low-quality annotations of tweet datasets written in particular languages containing emojis. We have also proposed a fully automated unsupervised approach to generate lexica with good quality. They have been validated on di erent datasets taken from [KNSSM15].
This work was partially supported by Mineco grant TEC2016-76465-C2-2-R and Xunta de Galicia grants ED341D R2016/012 and GRC2014/046, Spain. [BFMP13] [BKRS16]
Marina Boia, Boi Faltings, Claudiu Cristian Musat, and Pearl Pu. A : ) is worth a thousand words: How people attach sentiment to emoticons and words in tweets. In Social Computing, pages 345{350.
IEEE Computer Society, 2013.
Francesco Barbieri, German
Kruszewski, Francesco Ronzano,
and Horacio Saggion. How
cosmopolitan are emojis?: Exploring
emojis usage and meaning over
different languages with distributional
semantics. In Proc. of
Dmitry Davidov, Oren Tsur, and Ari Rappoport. Enhanced sentiment learning using Twitter hashtags and smileys. In Proc. of the 23rd Int.
Conf. on Computational Linguistics: Posters, COLING '10, pages 241{249, Stroudsburg, PA, USA, 2010. ACL.
Meng Joo Er, Fan Liu, Ning Wang, Yong Zhang, and Mahardhika Pratama. User-level Twitter sentiment analysis with a hybrid approach. In Int. Symposium on Neural Networks, pages 426{433. Springer, 2016. [HTAAAK16] [HTGL13]
Gael Guibon, Magalie Ochs, and Patrice Bellot. From Emojis to Sentiment Analysis. In WACAI 2016, Brest, France, June 2016. Lab-STICC and ENIB and LITIS.
Alexander Hogenboom, Danella Bal, Flavius Frasincar, Malissa Bal, Franciska De Jong, and Uzay Kaymak.
Exploiting emoticons in polarity classi cation of text. J. Web Eng., 14(12):22{40, March 2015.
Tianran Hu, Han Guo, Hao Sun, Thuy-vy Thi Nguyen, and Jiebo Luo.
Spice up your chat: The intentions and sentiment e ects of using emoji.
CoRR, abs/1703.02860, 2017.
Wegdan A. Hussien, Yahya M. Tashtoush, Mahmoud Al-Ayyoub, and Mohammed N Al-Kabi. Are emoticons good enough to train emotion classi ers of arabic tweets? In Computer Science and Information Technology (CSIT), 2016 7th Int. Conf. on, pages 1{6. IEEE, 2016.
Xia Hu, Jiliang Tang, Huiji Gao, and Huan Liu. Unsupervised sentiment analysis with emotional signals.
In Proc. of the 22nd Int. Conf. on WWW, WWW '13, pages 607{618, New York, NY, USA, 2013. ACM.
Mayu Kimura and Marie Katsurai.
Automatic construction of an emoji sentiment lexicon. In Proceedings of the 2017 IEEE/ACM International [KNSSM15] [LAL+16] [Liu12] [MGS16] [MTSC+16] [Nie11] [TK16]