Emojis are very common in social media and understanding their underlying semantics is of great interest from a Natural Language Processing point of view. In this work, we investigate emoji prediction in short text messages using a multi-task pipeline that simultaneously predicts emojis, their categories and sub-categories. The categories are either manually prede ned in the unicode standard or automatically obtained by clustering over word embeddings. We show that using this categorical information adds meaningful information, thus improving the performance of emoji prediction task. We systematically analyze the performance of the emoji prediction task by varying the number of training samples and also do a qualitative analysis by using attention weights from the prediction task.
Emojis are a popular set of ideograms created in the late 1990s to enrich written communication by adding nonverbal expressive power to digital communication. These symbols can be used by human readers to convey emotions and information in a condensed form. As Snapchat, Twitter and other social media platforms have become popular, so has the usage of emojis.
Despite their popularity, there is very little research work in predicting emojis. Over the past few years, the interest in emoji research has increased and several studies has been published in the area of distributional semantics [BRS16, ERA+16, WBSD17b, WBSD17a, BCC18], sentiment analysis [NSSM15, HGS+17, KK17, RPG+18] and multimodal systems [CMS15, CSG+18, BBRS18]. In the past year researchers also focused on the possibility of predicting emojis in a text message [BBS17, FMS+17]. The emoji prediction task consists in predicting the original emoji present in a tweet (or snap caption) given only the non-emoji textual content. Prior explorations of emoji prediction tended to focus on less than 2% of the total number (2653) of emojis in Unicode 6 standard 1 emojis. Another limitation of those papers was that emoji prediction could be ambiguous. For example, when the model predicts the correct label to be , emojis like , , , or can also be valid predictions.
In this work, we extended the emoji prediction task to 300 emojis in order to study a larger number of emojis along with their unicode standard categories, sub-categories, and the new semantic clusters that we created. We are not aware of any previous research work focused on either predicting a large number of emojis (300), or using a multi-task approach to predict emojis or emoji categories. We also do a systematic analysis of how the number of training samples a ect the performance of the emoji prediction task. To mitigate the problem of emoji ambiguity, we concentrate on broad emoji category prediction in addition to that of individual emoji prediction. We grouped emojis in two di erent ways. The rst one was de ned by the Unicode consortium2, which groups emojis into seven categories (e.g., \Smileys & People", \Nature") and 74 sub-categories (e.g., \face-positive", \face-negative"). The main categories are commonly found on mobile phone keyboards as shown in Figure 1). Alternatively 1www.unicode.org/emoji/charts/full-emoji-list.html 2www.unicode.org/emoji/charts/emoji-ordering.html we also created semantic clusters using embeddings.
We use a Multi-task approach to combine the tasks of emoji and category prediction. Multi-task approaches [Car, Car97, Ben12, CW08] improves generalization by transferring information across di erent tasks and improving each task individually. In particular, multi-task learning with simultaneous training on multiple tasks has demonstrated promising results [CW08][FMS+17][Ben12].
Our work performs multi-task learning by training a single model with multiple outputs (the dataset is annotated with multiple labels) and we evaluate using our gold standard created from Twitter and Snap public posts as described in the Datasets section. The subjectivity of emoji interpretation makes emoji prediction a very challenging task. Nevertheless our work shows that simultaneously predicting emojis, their categories, and sub-categories in a multitask framework improves the overall results. It not only improves emoji prediction, but it also helps with the identi cation of emoji categories, which can be particularly more relevant when the emoji prediction model is less precise.
The remainder of this work is organized in the following way: The next section describe the datasets used in our experiments. We then present the Deep Learning Models explored to solve our research problem, Finally, we discuss the experiments, results and then conclude with future research directions. 2
In this study we explore emoji prediction for two different datasets: Twitter and Snapchat captions. We select documents (tweets and snaps) that contain a single emoji, and at least three tokens apart from the emoji. We restrict to documents containing a single emoji so as to minimize the interference of the presence of other emojis in the emoji prediction task. We also consider only the documents that include the most frequent 300 emojis in each dataset. We restrict to the top 300 emojis only due to lack of meaningful number of examples beyond that. A subset of the most frequent emojis for each dataset is reported in Table 1. Note that we remove skin color from the emojis3 to avoid generating very similar labels. Table 2 includes statistics on the two datasets. We can see that Snap captions are shorter than tweets, while average word length is similar. Another important di erence between the two datasets is the most frequent emojis used. Table 1 shows the 60 most frequent emojis in each dataset (on the top Twitter and on the bottom Snap data), along with the number of documents that include each emoji. In both datasets the frequency is very unbalanced: 16% of tweets and 25% of snaps include one of the three most frequent emojis , , . Therefore we use a balanced dataset in our experiments, in order to give same importance to each emoji, independent of the frequency of usage. We subsample the most frequent emojis in order to match the same number of examples of the least represented emoji (1,500 examples for Twitter and 3,000 for Snap data). We show that using fewer than 1,500 examples per emoji leads to a drastic decrease in accuracy of the emoji detection (see Figure 3). We focus our experiments on 300 emojis because we do not have more than 1,500 tweets per emoji beyond the top 300 emojis in our Twitter dataset. For our experiments we randomly chose 80% of the documents for training, 10% for validation and 10% for testing. 2.1
The Twitter dataset contains 50 million tweets retrieved using Twitter API. Tweets were posted between January 2016 and April 2017 and were geolocalized in the United States. We removed hyperlinks from each tweet, and lowercased all textual content in order to reduce noise and sparsity. Since Twitter data includes a large percentage of bot data, we lter noise as much as possible, removing repeated tweets (or very similar ones) and selected a maximum of ve tweets per user. From this dataset, we selected tweets including anyone of the 300 most frequently occuring emojis and at least three tokens (without the emoji), resulting in a nal dataset composed of 2,862,505 tweets. 2.2
SnapCaption is an in-house Snapchat internal dataset containing only Snapchat captions. A caption is the textual overlay component of a snap. These captions were collected exclusively from snaps submitted to public and crowd-sourced stories (as known as Live Stories or Our Stories ). Examples of such public crowd-sourced stories are \New York Story" or \Thanksgiving Story". All captions were posted in one year period and do not contain any image or any 3E.g., are mapped to one single label 2.3
We also consider broader classes of emojis, such as unicode categories and semantic clusters. The unicode consortium de nes a set of 7 categories categories and 74 sub-categories.
The problem with Unicode categories and subcategories is that they fail to accurately capture semantically related emojis. Emojis like and are both in the sub-category neutral faces even though they clearly indicate di erent emotions. Another example is and that are semantically similar, but they appear in di erent categories (\Smiling Faces" and \Emotions") even though they address nearly identical meanings. To overcome this limitation, we propose a second approach to automatically organize emojis by clustering them using pre-trained word embeddings similar to emoji2vec [?]. These clusters have the advantage of better capturing the semantic information of emojis. For example and are in the same cluster. These clusters are an important aspect to consider because they are based on how emojis cooccur in short text messages from tweets and captions of public snaps. We pretrained two di erent sets of skip-gram embeddings [MLS13] for Twitter and Snap. The rst skip-gram model was trained on a dataset of about 70 million tweets and the second skip-gram model was trained on about 100 million Snap captions. Using the embeddings of the 300 most frequent emojis of each dataset, we created two sets of 30 clusters using a k-means algorithm. The number of clusters was de ned based on qualitative analysis (clusters that seemed to better organize emojis by semantics). In addition, the number of clusters was selected such that each cluster has a similar number of emojis that are usually displayed on a mobile keyword. As a result, we would be able to just provide an icon to access directly each cluster in a similar way as the Figure 1 shows for the top categories. The resulting clusters will group semantically similar emojis (like in [BKRS16] where 11 cluster are created for 100 emojis), grouping love, sad faces, hand/gestures, animals, food, drinks, parties, Christmas, and so on. Our main architecture, illustrated in Fig. (2), starts with our character and word embedding modules whose outputs are fused by our feature attention unit and the word attention unit. Finally the fully connected layers and the softmax play the role of the nal multi-task classi er.
Previous approaches [BBS17, FMS+17] have successfully learned LSTM models for emoji prediction tasks. We experimented di erent plain LSTMs, stacked LSTMs [FMS+17], and di erent word representations before solidifying on our nal model architecture Fig. (2). In addition, we explored single task models and multi-task models. In the case of the multi-task models, the entire network is shared and the specialization only occurs at the nal stage to predict speci c labels of each task. This specialization is accomplished through speci c linear transformations. Finally we used a cross entropy loss function for all classi cation tasks. In the case of multitask learning, the nal loss is the sum of each single loss4. In the following subsections, we detail each stage of our main architecture. The word embeddings are learned together with the updates to the model. For out-of-vocabulary words (OOVWs), we used a xed representation that is handled as a separate word. In order to train the xed representation for OOVWs, we stochastically replace (with p = 0.5) each word that occurs only once in the training data. When we use pre-trained word embeddings, that are concatenated with the learned vector.
4We also experimented weighted sum, with various weights, but the best results are obtained with a simple sum of the losses. In addition, we use a character based embedding [LLM+15, ?] stacked with a B-LSTM [GS05], producing a character-based word embedding that focuses on word spelling variants. Indeed, the character-based word embedding learned similar representations for words that are orthographically similar, and thus are expected to handle di erent alternatives of the same word types that normally occur in social media. 3.3
Our bi-directional LSTM modules, [GS05] named BLSTM in Fig. (2), consists of a forward LSTM that processes an input message from left to right, while the backward LSTM processes it in the reverse direction. As a result, the message representation s is based on both the forward and backward LSTM encoding: s = max f0; W[hfw; hbw] + dg where W is a learned parameter matrix, fw is the forward LSTM encoding of the message, bw is the backward LSTM encoding of the message, and d is a bias term, and we use a component-wise ReLU as the nonlinear unit. We use B-LSTM modules for both word and sentence representations, namely Char B-LSTM and Words B-LSTMs in our architecture Fig. (2). Char B-LSTM takes a sequence of characters and outputs a word embedding vector. This output is mixed with another word representation via our feature attention module. Then, the stacked Words LSTMs receive sequences of word representations from the attention module, and output sentence embedding vectors. The feature attention module aims to linearly fuse multiple input signals instead of simply concatenating them. In our architecture, this module learns a unied word representation space, i.e. it produces a single vector representation with aggregated knowledge among our multiple input word representations, based on their weighted importance. We can motivate this module from the following observations.
Prior work, [BBS17] combines both word representation x(w) and character-level representation x(c) by simply concatenating the word and character embeddings at each LSTM decoding step ht = LSTM([xt(w); xt(c)]). However, this naive concatenation results in inaccurate decoding, speci cally for unknown word token embeddings, e.g., an all-zero vector xt(w) = 0 or a random vector xt(w) = U ( ; + ), or even for out-of-vocabulary words. While this concatenation approach does not cause signi cant errors for well-formatted text, we observe that it induces performance degradation for our social media post datasets which contain a signi cant number of slang words, i.e., misspelled or out-of-vocabulary words. As a result, we use a feature attention module, that adaptively emphasizes each feature representation in a global manner at each decoding step t. This process produces a soft-attended context vector xt as an input token for the next stacked B-LSTM that takes care of the sentences embedding. [RCP16] introduced a similar approach, where the character embeddings are weighted with an attention module. We use the following method: [at(w); at(c)] =
Wm [xt(w); xt(c)] + bm (m) = t exp(at(m))
P exp(at(m0)) m02fw;cg 8m 2 fw; cg (1) where t = [ t(w); t(c)] 2 R2 is an attention vector at each decoding step t, and xt is a nal context vector at t that maximizes information gain for xt. Note that this feature attention requires each feature representation to have the same dimension (e.g. xt(w); xt(c) 2 Rp), and that the transformation via Wm essentially enforces each feature representation to be mapped into the same uni ed subspace, with the output of the transformation encoding weighted discriminative features for classi cation of emojis. 3.5
Not all the words have the same importance in the representation of a document. We use the attention mechanism introduced in [YYD+16]: uit = tanh(Wwhit + bw) it =
exp(ui>t uw) Pt exp(ui>t uw) ; di = X
ithit t (2) where the nal document representation di is a weighted average of the hidden representation hit of the LSTM. The weights it are learned by the use of a Multi-Layer Perceptron (linear transformation W and biases b) with tanh as non-linear operation, and a softmax to compute the probability of each word. 4
We use two main variations for experiments: SingleTask Prediction of emojis, unicode categories, and emoji clusters, and Multi-Task Prediction, where we combine the single tasks in one single model. We also evaluate the impact of our di erent modules including the combination of word/char LSTMs and the word attention unit. Finally we investigate the in uence of the number of layers for the LSTMs. 4.1
We explore three di erent tasks: (i) the emoji prediction task proposed by [BBS17], (ii) prediction of unicode emoji categories (the emoji in the text belong to the faces, animal, objects) and sub-categories (positive faces, animal-mammal), and (iii) prediction of automatic clusters that we previously generated using pre-trained word embeddings. 4.1.1
Given a set of documents, each document containing only one emoji class, the task consists of predicting the emoji from the text. For this task, we tested the in uence of the number of emoji classes and the number of examples per class. More precisely, for each experiment, we extract a balanced dataset of Nclass emoji classes, and Ndata examples per class, with Nclass = f20; 50; 100; 200; 300 and Ndata = f100; 500; 1000; 1500; 2000; 2500; 3000g. Nclass and Ndata are tested independently: when we vary Nclass, we x Ndata to 3000, and when we vary Ndata we x Nclass to 300. Figure 3 shows our experiments with the Snapchat dataset. It is clear that using more examples per class improves our model by around 1% absolute point from 1500 to 3000 examples. For >2000 examples the system converges to its optimum.
From Figure 4, we observe that Twitter data is easier to model than Snap data. In the 300 emoji prediction task the best accuracy at top 5 (a@5) on Twitter data is 40.05% while on Snap data it is 34.25% (see lstm lstm + att 2 lstm + att char + 2 lstm + att
char + lstm char + lstm + att 100 500 Table 3). There are several reasons that could explain this di erence in results. One reason is the length of the text messages, since in Twitter there are on average twelve words per message, while on Snap has only ve (see Table 2). Another reason could be the missing visual context of Snap posts5, while only a small percentage of tweets is complemented with a visual content. For this reason, tweets contain typically less semantic ambiguity.
Table 3 highlights the best performing systems on the emoji prediction task. For the two datasets state of the art systems are outperformed by the combination of additional components. For example, adding a word attention module improves the baseline of [BBS17]. Finally, there is an important di erence when predicting 20 and 300 emojis. We plot on the left of Figure 3 the accuracy of same model architecture (Char + 2 LSTM + word attention) on the emoji prediction task for di erent numbers of emojis (20 to 300). Best accuracy at top 5 (a@5) drops from 20 to 100, and then remains constant. We observe the same drop using F1 (that only considers whether an emoji is predicted as rst option), however, having more than 100 classes results in improvement. This is probably due the type of the more rare emoji classes added after the most 100 frequent ones, that are more speci c (like , , or ) hence easier to predict. one emoji as we did in the emoji prediction task.
Table 4 shows the prediction results using macroF1 and a@5 evaluation metrics. In the rst two blocks (main and sub lines), we predict the main category and sub-category respectively. The third block details the clusters' evaluation results, and the last block presents the emoji prediction results. In the rst line of each block are the single-task results and the remaining lines include the ones using a multi-task framework. 4.1.3
Given a text message containing an emoji e we predict the cluster that emoji e belongs to. Cluster creation is described in the dataset section. Cluster results are reported in Table 4, in the lines corresponding to \Semantic Clusters". The results are better on Snap than Twitter for broader classes and our clusters capture better semantics than the categories and In Table 4 we show the multi-task prediction results. We considered multiple multi-task combinations. Learning more than one objective task simultaneously helps in the main category prediction, as macro F1 improves from 46.56% to 48.52% (4.2% relative improvement) when adding also sub-category and emoji losses. Sub-categories prediction also improves when it is learned together with main categories and emojis.
On Snap data, category and sub-category prediction tasks do not improve using a multitask approach in terms of macro F1, but we obtain a relative improvement of 8.67% and 21.14% using a@5.
The clusters prediction tasks also bene t from multi-task learning when combined with the emoji prediction. However, emoji prediction seems not to improve much in a multi-task setting for Twitter. Emoji prediction on Snap improves from 33.86% to 34.27% or 1.21% relative improvement in terms of a@5 when it is learned together with Unicode sub-categories. 4.3
We analyzed in detail our emoji prediction approach (char + 2 LSTM + attention) based on the best performing system described in the previous section. This analysis enumerates the emojis that are easier and harder to predict. We also include some visualization examples of where the attention module obtains more information. These examples provide us with a better understanding of the importance of the character and word features in our results. 4.3.1
Table 5 shows a list of the top and bottom 10 emojis based on the prediction accuracy. We investigated what emojis are di cult to predict, and we found interesting patterns. As expected, the emojis that are easier to predict describe speci c objects without multiple meanings (such as, and ) or topics (e.g., and ). These emojis, as suggested in [BRS16], could easily be replaced by a word, such as by key), or are used when speci c words occur in a text message including Christmas for and ). In both datasets, subjective emojis including and obtained lowest accuracy values. These subjective emojis describe emotional information, and they can be interpreted di erently among di erent users and based on the surrounding context. Hence, these emojis do not seem to have a speci c meaning and become di cult to model. 4.3.2
We previously described the two types of attention explored. The Feature Attention approach gives more importance to either the character or word representation of a word. The Word Attention approach increases the importance of more discriminative words, for example the word \key" to predict the emoji .
Figure 5 visualizes the weights of each of these two attention modules using three example messages. For each of them, we list the gold label (\G") and the predicted labels (\P"), along with their prediction probability. i.e. the output of the softmax layer. The internal weights of the two attention modules are visualized using text highlights. Darker color indicates more attention over word ( it from Formula 2 of each word in the message). In second line of each message the red highlight shows the weights of the feature attention ( of Formula 1). Bold text formatting indicate the out of vocabulary words.
Based on the three examples, and some additional that we manually evaluated, we veri ed how these two attention approaches work. The Word Attention module (blue highlight) give us insights on the recognition of emojis. In the rst example the most important word is \cat" and the predictions are indeed about cats, apart from the fth predicted emoji . This emoji is triggered (probably) because of the presence of the token \135" as the word attention module also focuses on this token. In the second example, the attention goes to the word \pregnant", but in this case this word misleads the network that incorrectly predicts baby emojis. However, the correct emoji is then predicted as fourth option. In the last example, the network correctly classi es the emoji , based on the hashtag \#unicorn".
Regarding the Feature Attention over the word or character representation of each token in a message, we observed that the character representation seems to gain importance on long and less frequent tokens, namely numbers, hashtags, and as expected, out of vocabulary words (\135" and \#austinblueroos"). 5
In this paper, we explored emoji prediction in two social media platforms, Twitter and Snapchat. We extended the emoji prediction task to a large number of emojis and showed that the prediction performance drastically drops between 50 and 100 emojis, while the addition of more emojis keeps the accuracy of the model somehow constant (even if it has to predict more emojis). We attribute these results to the speci city of the less-used emojis. We also proposed a novel task that predicts broader classes of emojis, grouping emojis in automatic clusters or prede ned categories, as dened by the Unicode consortium. These new tasks allow us to better evaluate the predictions of the model, since plain emoji prediction may be ambiguous. We also carried out an extensive qualitative analysis in order to understand the importance of the character encoding of words in noisy social media text, the number of training examples, and the di culties in modeling speci c emojis.
Finally, we proposed a multi-task approach to predict emojis and emoji group a liation at the same time. We showed that the model obtains signi cant improvements in the Twitter dataset, while more investigation is needed for the Snapchat dataset.
This work was done when Francesco B. interned at Snap Inc. Francesco B. acknowledge support also from the TUNER project (TIN2015-65308-C5-5-R, MINECO/FEDER, UE) and the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).
Proceedings of NAACL: Short Papers, New Orleans, US, 2018. Association for Computational Linguistics.
and Horacio Saggion. Are emojis predictable? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 105{ 111, Valencia, Spain, April 2017. Association for Computational Linguistics.
[BBRS18]
Francesco Ronzano, and Horacio Saggion. How Cosmopolitan Are Emojis? Exploring Emojis Usage and Meaning over Di erent Languages with Distributional Semantics. In Proceedings of the 2016 ACM on Multimedia Conference, pages 531{535, Amsterdam, Netherlands, October 2016. ACM.
ICML'93. A In
Cees GM Snoek. Image2emoji: Zero-shot emoji prediction for visual media. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1311{ 1314. ACM, 2015.
Pierre Garrigues, Thomas Mensink, and Cees GM Snoek. The new modality: Emoji challenges in prediction, anticipation, and retrieval. arXiv preprint arXiv:1801.10253, 2018.
In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, pages 48{54, [FMS+17] [GS05] [HGS+17] [KK17] [LLM+15] [RPG+18]