New methodologies to evaluate the consistency of emoji sentiment lexica and alternatives to generate them in a fully automatic unsupervised way Milagros Fernández-Gavilanes Jonathan Juncal-Martı́nez Silvia Garcı́a-Méndez mfgavilanes@gti.uvigo.es jonijm@gti.uvigo.es sgarcia@gti.uvigo.es Enrique Costa-Montenegro Francisco Javier González-Castaño kike@gti.uvigo.es javier@det.uvigo.es GTI Research Group Telematic Engineering Dept., School of Telecommunication Engineering, University of Vigo, Vigo 36310 Spain 1 Introduction Abstract Following a trend in the last years, emojis are being increasingly used in social applications. For example, Sentiment analysis aims at detecting senti- 1% of the messages in a random sample of 22.14 bil- ment polarities in unstructured Internet infor- lion tweets taken between July 2013 and March 2018 mation. A relevant part of this information for contained at least one emoji1 . that purpose, emojis, whose use in Twitter has Emojis allow users to express feelings and emo- grown considerably in these years, deserves at- tions. Thus, it is interesting to try to extract from tention. However, every time a new version of them useful knowledge on user opinions [HTGL13]. Unicode is released, finding out the sentiment Natural Language Processing (nlp) allows us to an- users wish to express with a new emoji is chal- alyze opinions, feelings, assessments, etc. on prod- lenging. In [KNSSM15], an Emoji Sentiment ucts, services or organizations [Liu12]. Until very re- Ranking lexicon from manual annotations of cent times, researchers in the field of sentiment analy- messages in different languages was presented. sis (sa) only considered the information contributed The quality of these annotations affects di- by emoticons [BFMP13, DTR10, HBF+ 15]. Nev- rectly the quality of possible generated emoji ertheless, nowadays emojis are attracting consider- sentiment lexica (high quality corresponds to able attention [GOB16, HGS+ 17]. For this reason, high self-agreement and inter-agreement). In some recent studies have tried to obtain the sen- many cases, the creators of the datasets do timent expressed by emojis in the form of a lexi- not provide any quality metrics, so it is nec- con [KNSSM15, LAL+ 16, KK17]. In many cases, how- essary to use another strategy to detect this ever, the expected meaning of an emoji (in terms of issue. Therefore, we propose an automatic positivity, neutrality or negativity), which is assumed approach to identify and manage inconsistent to be universal, may changes among languages and manual sentiment annotations. Then, relying cultures [BKRS16]. on a new approach to generate emoji senti- Following this line, in [KNSSM15] the authors pre- ment lexica of good quality, we compare two sented an Emoji Sentiment Ranking (esr)2 , resulting such lexica with lexica created from manually from texts in 15 different languages containing emo- annotated datasets with poor and high quali- jis, whose sentiments were labeled manually by dif- ties. ferent human annotators over three months. How- ever, the quality of manual labeling, measured in terms Copyright c 2018 held by the author(s). Copying permitted for private and academic purposes. of self-agreement and inter-agreement as explained In: S. Wijeratne, E. Kiciman, H. Saggion, A. Sheth (eds.): Pro- in [MGS16], may be poor. ceedings of the 1st International Workshop on Emoji Under- standing and Applications in Social Media (Emoji2018), Stan- 1 http://www.emojitracker.com/api/stats ford, CA, USA, 25-JUN-2018, published at http://ceur-ws.org 2 Available at https://goo.gl/XEkJhZ We can suppose that, if an emoji sentiment lexicon values of +2, +1, -1 and -2 (strongly positive, weakly is generated from one of these single-language datasets, positive, weakly negative and strongly negative, re- the most popular emojis should be highly correlated spectively). with those obtained from the overall esr when the Currently, few approaches assign polarities to quality of manual labeling, measured in terms of self- emojis with semi-automatic or automatic methods. agreement and inter-agreement, is acceptable (differ- In [HTAAAK16] the most used emojis in a dataset ences would be mainly due to emojis with different of Arabic tweets were classified into four categories: interpretations among languages). On the contrary, anger, disgust, joy and sadness. Subsequently, they if at least one of these metrics is low, inconsisten- were weighted with scores between -5 and +5 (most cies in manual sentiment annotations should be sus- negative and most positive, respectively), according to pected, and the correlation would be seriously affected those categories. The weights were obtained from the (the differences in emoji interpretations would be much afinn lexicon [Nie11], in which some entries are emo- greater). When these measurements are not provided jis. The Unicode short Common Locale Data Repos- by the dataset creators or they are unknown, an alter- itory3 (cldr) names of the missing emojis were ob- native should be sought to identify the inconsistencies. tained and the words composing them were searched The final objective should be to create an emoji sen- in afinn (one by one, independently). Finally, weights timent lexicon with the highest possible quality. were also manually assigned according to the category In this paper, we propose an approach to detect of each emoji. low-quality dataset annotations. In case of inconsis- Regarding the approaches that obtain emoji senti- tent annotations, we also present a fully automated ment lexica in a fully unsupervised way, we are only approach to obtain emoji lexica with good quality. aware of the following examples. In [LAL+ 16], the au- The rest of the paper is organized as follows: Sec- thors analyzed emoji usage in text messages by coun- tion 2 reviews related work on emoji sentiment anal- try. In total, the sentiment of 199 emojis was ob- ysis. Section 3 discusses the issue of labeling quality. tained from their short cldr names processed with Section 4 describes the proposed method. Section 5 the liwc4 tool (which counts words that express pos- presents experimental results. Finally, Section 6 sum- itive, neutral or negative sentiment). This analysis marizes the main contributions and conclusions. did not exploit their real descriptions or their usage contexts. In [KK17], the authors extracted, for each 2 Related work word of a tweet that co-occurred with a target emoji, the set of synonyms or synsets available in WordNet 5 . Even though emoji sentiment interpretation (where Then they recovered the most frequent affective la- sentiment is expressed as a positive, neutral or neg- bel from WordNet-Affect 6 . Five sentiment categories ative polarity) has already been studied in the field of were differentiated: happiness, disgust, sadness, anger nlp, a common practice in the case of Twitter was to and fear, following a hierarchical structure. Finally filter Unicode symbols during message preprocessing, they calculated a sentiment score vector for 236 emo- so that emojis’ information was lost [TK16]. But, for jis based on the mentioned co-occurrences. Again, this example, in the message “Today I have to go to the su- analysis also ignored the real descriptions or the usage permarket ”, the obvious negativity is given by the contexts of the emojis. Finally, in [FGJMGM+ 18], emoji. a lexicon of 840 emojis was created using an unsuper- Focusing on methods to guess the real sentiment vised sa system, taking only into account emoji defini- of emojis, they can be classified in three types: man- tions in Emojipedia7 . This lexicon was then improved ual, semi-automatic and automatic. Regarding man- in different variants that took advantage of the senti- ual methods, in [MTSC+ 16] the most popular Uni- ment distribution of informal texts including emojis. code emoji characters were manually labeled by mul- tiple annotators, taking into account sentiment (posi- 3 Description of the problem tivity, neutrality and negativity) variance as well as se- mantics (meaning). In [KNSSM15], 83 native speakers In general, a given emoji should have the same emo- of different languages labeled by hand the sentiment tional meaning in different datasets written in the (positive, neutral or negative) of texts containing 751 same language. This implies that emoji sentiment in- different emojis. The authors calculated their senti- terpretation for each of them should be very close to ment based on their occurrences and the manual la- 3 http://unicode.org/emoji/charts/emoji-list.html bels of the tweets containing them, by applying a dis- 4 https://liwc.wpengine.com/ crete probability distribution. Finally, in [ELW+ 16] 5 https://wordnet.princeton.edu/ 78 strongly and 34 weakly subjective emojis were ex- 6 http://wndomains.fbk.eu/wnaffect.html tracted from the list [KNSSM15] and given polarity 7 https://emojipedia.org/ Figure 1: Method to produce two emoji sentiment lexica the interpretation for all datasets together. The prob- ploited independently by an unsupervised sa system lem arises when an emoji sentiment lexicon is created with sentiment propagation across dependencies (uss- from multilingual datasets with manual sentiment an- pad) described in [FGALJM+ 16]. Depending on the notations that are inconsistent for a language or some combination of the polarities obtained from the sa, languages. (3) two emoji sentiment lexica variants are created. In On the other hand, it seems logical to think that an this regards, we remark that our aim is not a novel sa emoji should have different emotional meanings across approach. different languages and cultures. Nevertheless, accord- In Figure 1, the dotted arrow in the upper left cor- ing to [BKRS16], the semantics of the most popular ner represents the actions to gather a set of informal emojis are strongly correlated most of the time in most texts with emojis. The solid arrows represent the pro- languages in that regard. This was an interest finding, cesses carried out on these texts to obtain the first because both the vocabularies of the languages and the emoji sentiment lexicon from an emoji sentiment rank- context words modeled by the semantic spaces are dif- ing, from automatically labeled texts where they oc- ferent. The authors stated that English and Spanish cur. The dashed arrows refer to the case in which speakers interpret emojis in the most universal way, a similar process is previously applied on each indi- with a high correlation with all other languages, al- vidual emoji description (extracted from Emojipedia), though strong differences may persist for some emojis. to obtain an initial emoji sentiment lexicon from the In this way, the sentiment of the most popular emojis universal definitions by emoji creators. This lexicon, in a particular language may differ from the “universal unsupervisedemojiDef , is later applied as extra informa- sentiment”, but they should be close in most cases. tion into each particular informal text, to assign sen- Our main contributions are a method to detect timent labels automatically and then obtain the sec- anomalies in emoji sentiment lexicon due to incon- ond lexicon through the same emoji sentiment ranking. sistent annotations and an alternative automatic ap- Next, we explain the method in more detail. proach to predict emoji sentiments with applications in emoji sentiment lexica generation. 4.1 Acquiring emoji definitions In order to extract emoji definitions, messages must 4 Proposed methods be converted to a Unicode representation and regular We first present a method for constructing automati- expressions must be used for the extraction8 . Then, cally two emoji sentiment lexica [FGJMGM+ 18] (Fig- each emoji Unicode codepoint in hexadecimal notation ure 1). Summing up, (1) emojis are extracted from a is converted to UTF-8 hex bytes and submitted via a set of informal texts and their descriptions are acquired get request9 to the Emojipedia resource to retrieve its from the Emojipedia repository. Then, (2) nlp tech- 8 This process was carried out using the Emoji-java library, niques capture their linguistic peculiarities from both available at https://github.com/vdurmont/emoji-java. the descriptions and the informal texts, which are ex- 9 http://emojipedia.org/search/?q=. English description, which is parsed through jsoup10 . 5 Evaluation and experimental results 5.1 Dataset 4.2 sa on texts and emoji definitions We used the annotated datasets in [KNSSM15] in 15 different languages including Albanian, English, Polish At this point, the method performs sa on both the and Spanish, among others. These datasets are avail- informal texts containing the emojis and their defi- able at the public clarin11 language resource repos- nitions. This consists of two main tasks: preliminary itory. The entry for each labeled tweet consists of a data treatment with lexical and syntactic analysis; and tweet ID, a sentiment label (negative, neutral or posi- capturing linguistic peculiarities and applying usspad tive) and an anonymized annotator ID. We focused on sa [FGALJM+ 16]. In it, the final sentiment results the four datasets in Table 1, discarding tweets without from the propagation of sentiment term values (in- emojis and tweets with ambiguities among annotators. cluded in a sentiment lexicon) from the leaves to the The authors reported good self-agreement (Alphas ) parent nodes of each dependencies tree. Once these and inter-agreement (Alphai ) values for English and steps are completed, a polarity score is assigned to Polish and worse values for Albanian and Spanish. each informal text and emoji description, and emoji sentiment lexica can be created. Dataset #emojis Label #Tweets % Albanian Negative 17 14.53% Alphas = 0.447 48 Neutral 40 34.19% 4.3 Creation of emoji sentiment lexica Alphai = 0.126 Positive 60 51.28% English Negative 2,935 27.59% Once all previous steps have been performed on in- Alphas = 0.739 624 Neutral 2,677 25.16% Alphai = 0.613 Positive 5,027 47.25% formal texts and descriptions, we are in a position Polish Negative 638 27.59% to apply two different approaches to exploit polarity Alphas = 0.757 369 Neutral 919 24.27% scores of texts and definitions, and create two emoji Alphai = 0.571 Positive 2,229 58.87% sentiment lexica. In the first variant (E1), the lexicon Spanish Negative 1,022 16.85% Alphas = 0.245 613 Neutral 3,431 26.89% is created considering the ranking of polarity scores Alphai = 0.121 Positive 8,306 65.10% assigned to texts with emojis, applying the estima- tions in [KNSSM15]. That is, following the solid ar- Table 1: Distribution of negative, positive, and neu- rows in Figure 1 we obtain Runsupervised . The sec- tral tweets containing emojis for the datasets in the ond variant (E2) considers extra information. Lexicon experiments unsupervisedemojiDef is created from sentiment scores obtained through automatic sentiment propagation on emoji definitions. These values are then included in 5.2 Practical case for detecting anomalies in the sentiment lexicon used in Section 4.2 to improve annotations the sa of informal texts and obtain new polarity scores Table 2 shows the correlations for positive, neg- for them. Finally, the same estimations in [KNSSM15] ative and neutral labels between the conventional are applied to the resulting unsupervised sets. That esr lexicon (Rannotated , created using the method is, following the dashed arrows in the figure, to obtain all Runsupervised+unsupervised . in [KNSSM15] from messages in 15 languages anno- emojiDef tated by hand) and each emoji sentiment lexicon, which was created in the same way for a single lan- guage (Rannotateden for English, for instance). For a 4.4 Detecting inconsistent annotations fair analysis, given the detection criterion, to calculate Given the hypothesis that the sentiments of the most the correlation we considered the top 100 occurring popular emojis are preserved across different lan- emojis in each language lexicon as the most popular. guages, and that only a small percentage of them show Looking at Table 2, score and ranking level corre- language-specific usage patterns [BKRS16], we assume lations are high for English and Polish (Rannotatedpo ). that the correlation between the entries of an emoji Moreover, looking at Figures 2a and 2b, the associ- sentiment lexicon created for a particular language and ated linear regressions (represented with solid lines) the entries of a multilingual emoji sentiment lexicon have slightly less slope than the regression for the (ideally a universal lexicon) should be high. This is overall case that serves as gold-standard (represented the base for the experiments in Section 5.2. with a dotted line). This suggests that the English and Polish datasets have consistent annotations, as 10 Available at https://jsoup.org/ 11 http://hdl.handle.net/11356/1054. Lexicon x Lexicon y rscore (x, y) rrank (x, y) Rannotated Rannotateden 93.57% 89.46% all Rannotatedpo 88.74% 86.40% Rannotatedes 34.07% 37.35% Rannotated 36.37% 39.30% al Table 2: Score and rank correlations considering top 100 emojis ranked by score and occurrence (a) Plot for top 100 emoji sentiment scores comparing evidenced by their good Alphas and Alphai values Rannotated with Rannotateden in [KNSSM15, MGS16]. all However, when we compared the overall Rannotated lexicon with the Spanish and Alba- all nian lexica (Rannotatedes and Rannotated ), score and al ranking correlations were worse. Indeed, in Figures 2c and 2d, the linear regression slopes are very flat, and therefore they move far from the overall case. This suggests that the Spanish and Albanian datasets have inconsistent manual annotations (as shown by Alphai =0.121 and Alphas =0.245 for Spanish and Alphai =0.126 for Albanian) [KNSSM15, MGS16]. In addition, if we focus on Figure 2c, a vast majority (b) Plot for top 100 emoji sentiment scores comparing of emoji dots have positive polarity in the Spanish Rannotated with Rannotatedpo lexicon (X axis) while, for the overall case, polarities all vary between positive and negative. 5.3 Alternative solution for lexica generation Once we are able to detect annotation anomalies, we also have a methodology to validate an alternative so- lution to generate lexica automatically. We verified it on English and Spanish datasets as representative cases of which we have good and bad manual annota- tions, respectively. Two sentiment emoji lexica were created per language, corresponding to variants E1, (c) Plot for top 100 emoji sentiment scores comparing which only considers the automatic usspad annota- Rannotated with Rannotatedes tion (E1es and E1en ), and E2, which also considers all Emojipedia definitions (E2es and E2en ). Subindex’s es and en denote Spanish and English, respectively. Lexicon x Lexicon y rscore (x, y) rrank (x, y) E1en Rannotateden 82.91% 76.20% Rannotated 79.70% 75.25% all E2en Rannotateden 83.72% 79.37% Rannotated 86.90% 80.71% all E1es Rannotatedes 47.19% 47.18% Rannotated 74.93% 74.78% all (d) Plot for 48 emoji sentiment scores comparing E2es Rannotatedes 30.06% 44.09% Rannotated with Rannotated Rannotated 81.32% 79.07% all al all Figure 2: Top 100 emoji sentiment scores comparing Table 3: Score and rank correlations considering top the general emoji lexicon with the lexicon of a partic- 100 occurrent emojis in English and Spanish ular language In Table 3, if we compare the English variants, we observe that the lexica are highly correlated. Introduc- ing the effect of emoji definitions, correlation increases from E1en to E2en compared both with Rannotated all and Rannotateden . This is clear in Figures 3a and 3c, where the line that serves as gold-standard and the re- gressions intersect at neutral emoji sentiments. How- ever, in Figures 3b and 3d these lines intersect respec- tively at positive and neutral emoji sentiments. This shows that the definitions balance sentiments in the second variant. (a) Correlation between E1en and Rannotateden On the other hand, given the fact that the emoji sentiment lexicon obtained from a manually annotated Spanish dataset Rannotatedes has poor quality due to annotation inconsistencies [MGS16], as confirmed by their authors and by Table 2 and Figure 2c in Section 5.2, its correlation with the automatic variants should also be low. This is verified in Table 3 for E1es and E2es . The better behavior of E1es in this case is not relevant, due to the anomalies in Rannotatedes . How- ever, in the comparisons with Rannotated , the corre- all lation with E2es is higher both for ranking and score, as shown in Figures 4a and 4b, which is coherent with (b) Correlation between E1en and Rannotated the observations for English. all (a) Correlation between E1es and Rannotated (c) Correlation between E2en and Rannotateden all (d) Correlation between E2en and Rannotated (b) Correlation between E2es and Rannotated all all Figure 3: Top 100 emoji sentiment scores in English Figure 4: Top 100 emoji sentiment scores in Spanish 5.4 Checking with sa the new approaches 6 Conclusions A poorly labeled dataset (yielding low self-agreement Rannotated is biased by typical emoji usage world- and inter-agreement) may affect directly the quality all wide and, to a lesser extent, by the vision of the an- of emoji lexica. In many cases the annotators do not notator, who writes in a particular language. For this publish any quality metrics, so it is difficult to de- reason, we might worry about the influence of particu- termine beforehand if bad sa performance is due to lar language subsets in the overall lexicon. Therefore, the supporting lexicon or to the sa technique itself. an independent evaluation of the generated emoji sen- In this paper we have proposed a method to detect timent lexica is necessary. low-quality annotations of tweet datasets written in particular languages containing emojis. We have also Our objective here is to determine if our lexica vari- proposed a fully automated unsupervised approach to ants for Spanish and English are good enough in a generate lexica with good quality. They have been val- real-world scenario, by evaluating their impact with idated on different datasets taken from [KNSSM15]. sa metrics (precision (Pmacro ), recall (Rmacro ) and F (Fmacro ) macroaverages on the positive and negative Acknowledgements classes). In principle, in the Spanish subset this is impeded This work was partially supported by Mineco grant by bad labeling. We assumed that only a small per- TEC2016-76465-C2-2-R and Xunta de Galicia grants centage of the most popular emojis had significant sen- ED341D R2016/012 and GRC2014/046, Spain. timent differences between languages. For most prop- erly annotated messages containing the top popular References emojis, we could thus assume that any lexica should [BFMP13] Marina Boia, Boi Faltings, provide similar results. Therefore, we decided to re- Claudiu Cristian Musat, and Pearl strict the sa test to the 100 most popular emojis in Pu. A : ) is worth a thousand words: Spanish and English. Then we only selected the mes- How people attach sentiment to sages in the English dataset where those emojis oc- emoticons and words in tweets. In curred (English B). This new dataset had a distribu- Social Computing, pages 345–350. tion with 3552 positive, 1998 negative and 1601 neutral IEEE Computer Society, 2013. messages. Table 4 shows the results. [BKRS16] Francesco Barbieri, Germán Kruszewski, Francesco Ronzano, Dataset Lexicon Pmacro Rmacro Fmacro and Horacio Saggion. How cos- English B Rannotateden 76.16% 69.45% 72.65% mopolitan are emojis?: Exploring E2en 75.49% 69.20% 72.21% emojis usage and meaning over dif- E1en 67.95% 67.74% 67.85% ferent languages with distributional E2es 73.01% 67.84% 70.33% semantics. In Proc. of the 2016 ACM E1es 66.98% 67.89% 67.43% Conf. on Multimedia Conference, Rannotatedes 56.42% 62.04% 59.10% MM 2016, Amsterdam, Oct. 15-19, 2016, pages 531–535, 2016. Table 4: Macroaveraging sa metrics of English dataset for the most popular emojis in English and Spanish [DTR10] Dmitry Davidov, Oren Tsur, and Ari Rappoport. Enhanced sentiment learning using Twitter hashtags and Our assumptions are validated by these results, smileys. In Proc. of the 23rd Int. sorted by Pmacro . The ordering is coherent with our Conf. on Computational Linguistics: expectations. Rannotateden was created from consis- Posters, COLING ’10, pages 241–249, tent manual annotations, but E2en only performs a Stroudsburg, PA, USA, 2010. ACL. bit worse. If we compare E1en with E1es , on the one hand, and E2en with E2es , on the other, their perfor- [ELW+ 16] Meng Joo Er, Fan Liu, Ning mances are comparable bit for small percentages that Wang, Yong Zhang, and Mahard- can be explained by the the small percentage of “top” hika Pratama. User-level Twitter sen- emojis whose sentiment is not preserved across lan- timent analysis with a hybrid ap- guages. An important finding is that our automatic proach. In Int. Symposium on Neural approach performs satisfactorily compared to a lexi- Networks, pages 426–433. Springer, con produced from a well-annotated dataset. 2016. [FGALJM+ 16] Milagros Fernández-Gavilanes, Conference on Advances in Social Tamara Álvarez López, Jonathan Networks Analysis and Mining 2017, Juncal-Martı́nez, Enrique Costa- ASONAM ’17, pages 1033–1036, New Montenegro, and Francisco Javier York, NY, USA, 2017. ACM. González-Castaño. Unsupervised method for sentiment analysis in [KNSSM15] Petra Kralj Novak, Jasmina online texts. Expert Syst. Appl., Smailović, Borut Sluban, and 58(C):57–75, October 2016. Igor Mozetič. Sentiment of emojis. PLOS ONE, 10(12):1–22, 12 2015. [FGJMGM+ 18] Milagros Fernández-Gavilanes, Jonathan Juncal-Martı́nez, Silvia [LAL+ 16] Xuan Lu, Wei Ai, Xuanzhe Liu, Qian Garcı́a-Méndez, Enrique Costa- Li, Ning Wang, Gang Huang, and Montenegro, and Francisco Javier Qiaozhu Mei. Learning from the ubiq- González-Castaño. Creating emoji uitous language: an empirical anal- lexica from unsupervised sentiment ysis of emoji usage of smartphone analysis of their descriptions. Expert users. In Proceedings of the 2016 Syst. Appl., 103:74 – 91, 2018. ACM International Joint Conference on Pervasive and Ubiquitous Com- [GOB16] Gaël Guibon, Magalie Ochs, and puting, pages 770–780. ACM, 2016. Patrice Bellot. From Emojis to Sen- [Liu12] Bing Liu. Sentiment analysis and timent Analysis. In WACAI 2016, opinion mining. Synthesis lectures on Brest, France, June 2016. Lab-STICC human language technologies, 5(1):1– and ENIB and LITIS. 167, 2012. [HBF+ 15] Alexander Hogenboom, Danella Bal, [MGS16] Igor Mozetič, Miha Grčar, and Jas- Flavius Frasincar, Malissa Bal, Fran- mina Smailović. Multilingual Twit- ciska De Jong, and Uzay Kaymak. ter sentiment classification: The role Exploiting emoticons in polarity clas- of human annotators. PloS one, sification of text. J. Web Eng., 14(1- 11(5)(5):e0155036, 2016. 2):22–40, March 2015. [MTSC+ 16] Hannah Miller, Jacob Thebault- [HGS+ 17] Tianran Hu, Han Guo, Hao Sun, Spieker, Shuo Chang, Isaac John- Thuy-vy Thi Nguyen, and Jiebo Luo. son, Loren Terveen, and Brent Hecht. Spice up your chat: The intentions “Blissfully Happy” or “Ready to and sentiment effects of using emoji. Fight”: Varying interpretations of CoRR, abs/1703.02860, 2017. emoji. ICWSM’16, 2016. [HTAAAK16] Wegdan A. Hussien, Yahya M. Tash- [Nie11] Finn Årup Nielsen. A new anew: toush, Mahmoud Al-Ayyoub, and Evaluation of a word list for senti- Mohammed N Al-Kabi. Are emoti- ment analysis in microblogs. arXiv cons good enough to train emotion preprint arXiv:1103.2903, 2011. classifiers of arabic tweets? In Com- puter Science and Information Tech- [TK16] Channary Tauch and Eiman Kanjo. nology (CSIT), 2016 7th Int. Conf. The roles of emojis in mobile phone on, pages 1–6. IEEE, 2016. notifications. In Proc. of the ACM Int. Joint Conf. on Pervasive [HTGL13] Xia Hu, Jiliang Tang, Huiji Gao, and Ubiquitous Computing: Adjunct, and Huan Liu. Unsupervised senti- pages 1560–1565. ACM, 2016. ment analysis with emotional signals. In Proc. of the 22nd Int. Conf. on WWW, WWW ’13, pages 607–618, New York, NY, USA, 2013. ACM. [KK17] Mayu Kimura and Marie Katsurai. Automatic construction of an emoji sentiment lexicon. In Proceedings of the 2017 IEEE/ACM International