Increasing number of research in the area of distant supervision for emotion detection task requires a reliable mapping between noisy labels and emotion classes. We propose a method for an experimental creation of such a reliable mapping based on manually annotated data and quantitative relations between labels and classes on example of emojiemotion pair in a form of emoji2emotion mapping.
The japanese word emoji means \picture + character", and has no semantical connection to english emotion as you might have thought. However, emojis indeed very often carry the emotional state of the writer. That is why, no surprises that as a part of the digital text emojis were exploited in various NLP researches related to sentiment analysis or emotion classi cation.
In later works based on machine learning approaches, most of the time emojis are used as a noisy label for a distant supervision task. However, the matching between emoji and sentiment or emotion class is often done manually [WR16]. That approach implies subjectivity and could lead to mismatching. The goal of this work is to propose a method for experimental matching between emoji and classes that should be more reliable. To evaluate our method we apply it to emoji to emotion mapping. Since to our knowledge there is no such experimentally created mapping between them, we introduce the name for it - emoji2emotion.
There exist di erent emotion classi cation models, either discrete or dimensional. In this work we have chosen Plutchik's wheel of emotions [Plu91] that combine characteristics of both discrete and dimensional models. We use main 8 emotions out of it called Plutchik's eight (anger, anticipation, joy, trust, fear, surprise, sadness and disgust) that shown in Figure 1. .
One of the rst attempts to characterize emoji from its sentimental load perspective was a project called Emoji Sentiment Ranking - the rst emoji sentiment lexicon (Figure 2). It was created by [NSSM15] and provides a map between 751 most frequently used emojis and sentiments. The valuable insights from it that we use: the majority of emojis are positive, especially the top popular ones; among tweets with emojis, the inter-annotator agreement tend to be higher.
In [ERA+16] authors release emoji2vec, set of pretrained embeddings for all emojis in Unicode learned from emojisdescription taken from Unicode emoji standard. That is one of the examples of mapping emojis to another forms that are compatible to incorporate into machine learning tasks. And in general, representation learning and usage of pre-trained word embeddings is popular among natural language processing applications focused on social media.
In several works [BFMP13], [HBF+15], [JLL+14], [KZM14] emoticons were used to create a lexicon for a later use in a knowledge-based approach for sentiment analysis or emotion detection. The common thing between these works is a utilization of a high number of emoticon types, usually hundreds. Later works based on machine learning approach in contrast to works in the previous paragraph use emoticons and emojis as noisy labels for distant supervision tasks. Such works are [Rea05], [GBH09], [DTR10], [ZDWX12].
The recent paper [FMS+17] presents a project called DeepMoji and shows that diversi cation of noisy labels set for the distant supervision allows models to learn richer representations. They obtained state-ofthe-art performance on the 8 benchmark datasets according to sentiment, emotion and sarcasm detection, which proves the e ectiveness of the noisy level approach. Furthermore, their analyses con rm the assumption that diversity of emotional labels results in a performance improvement comparing to previous distant supervision methods. 3
In this section, the process of manually annotated corpus creation is described in detail. First, an acquisition of data for further annotation is explained in three steps: emoji list creation, tweets crawling and tweets preprocessing. Second, the annotation course is presented in another three steps: tweets ltering, annotation and averaging of vectors, analysis of resulting corpus. The rst step of an emoji containing tweets corpus creation is to choose the list of emojis. To select most popular emojis in the twitter and in general in text online, we looked into the Emojitracker [etr13] project as well as into Emoji Sentiment Ranking table [NSSM15]. By application of threshold for each ranking (>100 000 000 for Emojitracker and >100 for Emoji Sentiment Ranking) 31 emojis from the rst list and 50 from the second was picked. We selected emojis that were the intersection of both lists and additionally handpicked some emojis that were in the top lists but not in the intersection one. That is how the set of 43 tweets was created. After that, we calculated the distribution percentages for each source and found the average. That average percentage was used to create the same natural balance in our corpus.
The second step of corpus creation is a collection of data using the results of the previous step. In this paper, we use easily accessible Twitter data that we crawl with help of tweepy library. As a result, 84777 tweets containing emojis were crawled. Turned out, the vast majority of them (92.3%) contains only one type of emoji and most of the time its quantity is equal to 1 (average emoji count per tweet is 1.2). That is why we decided to focus on single emoji type tweets and after ltering out tweets with multiple emoji types or with emoji types that are not in our emoji list, the 74670 tweets left for training purposes.
The last step in the creation of corpus for labelling is a tweets preprocessing. On this stage, the raw tweets downloaded in the previous step are processed to the ready tweets . To do so, the number of emoji types in a tweet is counted, as well as the number of occurrences per each emoji type present. The replacement of tags, hashtags and URLs by the placeholders is done. 3.2
Data Annotation In order to start annotation process, we picked 500 tweets with additional requirement in order to enhance the quality of tweets to be annotated. The requirements were:
Tweet does not contain URL-s, TAG-s. That is a common practice in NLP that allows to exclude meaningless parts of the text.
Tweet does not contain HASHTAGS. Even though [DTR10] found hashtags useful for automated sentiment analysis, in our case we decided to eliminate them in order to increase the readability for annotators.
Tweet contains from 5 to 15 words. That way we have not so short and not so long tweets. Tweet contains no more than 2 uppercase words. That is also for readability reasons. Tweet contains no unlemmatizeable words (using spacys lemmatizer). Here it serves the data purity purposes as well as the understandability of the text for annotators.
Tweet contains no certain keywords (the list was manually generated after revision of corpus) in order to eliminate spam tweets.
After choosing these 500 tweets, 3 annotators were asked to go through the procedure of tweets evaluation using an Web Interface created by us. For each tweet they could choose arbitrary number of emotions (including none) out of Plutchik's Eight and set the intensity value for it from 1 to 3. The resulting labels were averaged according to rule of where more than half of annotators should agree on label.
The resulting corpus consists of 500 labeled tweets, where labels are vectors of size 8 containing emotion intensities for 8 emotions. In the annotated set nearly half of tweets has only one emotion type, and the other half the combination of them (up to 4 out of 8 at once), resulting 1.1 emotion per tweet in average. The most prevalent emotion was joy that appeared in 57% of tweets to some extent. Other emotions were not that spread, and appeared in a quarter or less of tweets each.
In Table 1 the statistics of emotion and emotion combination distributions over the dataset is presented. For clarity emotions and emotion combinations are grouped into the positive, negative and neutral groups. Here the grouping was made under the assumption that emotions joy and trust are positive; sadness, anger, disgust, and fear are negative; and no emotions(neutral), anticipation and surprise are neutral. The combinations were determined by the prevailing sentiment, and in case of equality of positive and negative emotions, it was grouped into the neutral category.
The macro distribution shows that tweets with positive emotions are prevailing with about 60%, while the negative and neutral emotion tweets are only the rest. That is predicted that positive tweets will appear more (as stated in [NSSM15]), however, a distribution of classes is quite imbalanced. 4
Using annotated dataset from the previous step the
percentage of emoji occurrences per emotion and vice
verse was done. In order to create a mapping, we
checked each possible pair of emotions and emojis for
the following two conditions. First, emojis percentage
of appearing in the tweets subset of certain emotion
should be at least equal to the median value of
possible percentages. Second, an emotion should appear
in certain emoji tweets at least half of the time. As
a result, the following mapping was done as shown in
Table 3.
To evaluate the quality of mapping, we use them as
noisy labels in emotion
To train our model we turned the news headlines in test set as well as tweet texts in training set into word embeddings using the word2vec methodology and open source code of emoji2vec. Then we fed these word embeddings as well as noisy labels to 4 classi ers (SGD, Naive Bayes, Random Forest and k-NN) from the scikit library. Using the trained model we predicted emotion categories per headline for the 1000 test set mentioned before. The resulting precision, recall and f1 scores are presented in the Table 3. The bold values represent maximum values, while green values are those that outperform the SemEval's best scores.
That is evident that the training data has a imbalance towards certain emotion categories which we link to the number of emojis picked per emotion. That is why the results of training also translate that kind of bias. To avoid that bias we need a more balanced set, and for that, in turn, we need more balanced mapping. To achieve that, more training data will be needed in the next run of the experiment and we leave that for future development of the work. We propose a method of experimental mapping between emoji and sentiment or emotion classes based on a special processing of manually annotated data. The processing includes the nding quantitate relation between emoji and emotion in form of cooccurrence percentage and further thresholding. To implement the method we annotated the corpus of 500 tweets containing emojis with help of 3 human judges. From the average annotation labels we constructed mapping as described above. Due to signi cant imbalance in emotions distribution across the dataset mapping was done only for 4 emotion categories and evaluated by exploiting as noisy labels for emotion detection task on those 4 emotions. The results on emotion detection task show that it is feasible to continue in that direction by increasing the size of the annotated corpus and further tuning the training parameters.
The resulting corpus of manually labeled emoji containing tweets is shared open source online (https://github.com/Aisulu/emoji2emotion) for the bene ts of scienti c society. After the annotation process that was evident to us, that the labelling for 8 classes and 3 intensity levels for each of them require the high cognitive load from the annotators and in average takes 18 seconds per tweet. Even though we knew that the increase in the class numbers leads to the slower labelling [BKT+13], it was higher than we expected and lead to the decrease of the nal corpus size. As a result, not all the emotions were presented in large enough size in the dataset which leads to the convergence of the classes to fewer classes. 7
We aim to nd less time-consuming form of annotation process for users to increase the size of the manually annotated corpus. After that we plan to repeat experimental procedures. [GBH09]