Nowadays, social media has become the essential part of our lives. Pictograms (emoticons/emojis) have been widely used in social media as a medium for visually expressing emotions. In this paper, we propose a emoji-aware attention-based GRU network model for sentiment analysis of Weibo which is the most popular Chinese social media platform. Firstly, we analyzed the usage of 67 emojis with facial expression. By performing a polarity annotation with a new “humorous type” added, we have confirmed that 23 emojis can be considered more as humorous than positive or negative. On this basis, we applied the emojis polarity to a attentionbased GRU network model for sentiment analysis of undersized labelled data. Our experimental results show that the proposed method can significantly improve the performance for predicting sentiment polarity on social media.
Today, many people share their lives with their friends by posting status updates on Facebook, sharing their holiday photos on Instagram or tweeting their views via Twitter or Weibo - the biggest Chinese social media network that was launched in 2009. Social media data contains a vast amount of valuable sentiment information not only for the commercial use, but also for psychology, cognitive linguistics or political science [Li et al., 2018a].
Over the past decade, sentiment analysis of microblogs became an important area of research in the field of Natural Language Processing. Study of sentiment in microblogs in English language has undergone major developments in recent years [Peng et al., 2017]. Chinese sentiment analysis research, on the other hand, is still at early stage [Wang et al., 2013] especially when it comes to utilizing lexicons and considering pictograms.
Recently, emojis have emerged as a new and widespread aspect of digital communication, spanning diverse social networks and spoken language. For example, “face with tears of joy” (an emoji that means that somebody is in an extremely good mood) was regarded as the 2015 word of the year by The Oxford Dictionary [Moschini, 2016]. In our opinion, ignoring pictograms in sentiment research is unjustifiable, because they convey a significant emotional information and play an important role in expressing moods and opinions in social media [Novak et al., 2015; Guibon et al., 2016; Li et al., 2019].
Furthermore, we also noticed that when people use emojis, they tend to express a kind of humorous emotion which is difficult to be easily classified as positive or negative. It seems that some pictograms are used just for fun, self-mockery or jocosity which expresses an implicit humor which might be characteristic to Chinese culture. Figure 1 shows an example of a Weibo microblog posted with emojis. In the third line of the post, (ning meng ren1) is a new word that appeared in early 2019 on Chinese social media and means “lemon man”. Accordingly, to address this new popular phrase, was added to the pictogram repoitoare by social media companies in January 2019. This lemon with a sad face is also called “lemon man” which expresses the same emotion as slang ning meng ren – “sour grapes” or “jealous of someone’s success”. This entry seems to convey a humorous nuance of a pessimistic attitude. Emojis seem to play an important role in expressing this kind of emotions. There is a high possibility that this phenomenon can cause a significant difficulty in sentiment recognition task.
To address this phenomenon, in this paper we focus on the emojis used on Weibo in order to establish if pictograms improve sentiment analysis by recognizing humorous entries which are difficult to polarize. Because the emojis probably play an equal or sometimes even more important role in expressing emotion than textual features, we analyzed the characteristics of emojis, and report on their evaluation while dividing them into three categories: positive, negative and humorous. We also noticed that among the resources of Chinese social media sentiment analysis, the labelled Weibo data sets containing emojis are extremely rare which makes considering them in machine learning approaches difficult. To resolve this problem, we propose a novel attention-based GRU network model using emoji polarity to improve sentiment analysis on smaller annotated data sets. Our experimental results show that the proposed method can significantly improve the 1In this paper we use italic to indicate romanization of Chinese language (pinyin).
Tan and Zhang conducted an empirical study of sentiment categorization on Chinese documents [Tan and Zhang, 2008]. They tested four features – mutual information, information gain, chi-square, and document frequency; and five learning algorithms: centroid classifier, k-Nearest Neighbor, Winnow classifier, Na¨ıve Bayes (NB) and Support Vector Machine (SVM). Their results showed that the information gain and SVM achieved the best results for sentiment classification coupled with domain or topic dependent classifiers. There are also researchers who have combined the machine learning approach with the lexicon-based approach. [Chen et al., 2015] proposed a novel sentiment classification method which incorporated existing Chinese sentiment lexicon and convolutional neural network. The results showed that their approach outperforms the convolutional neural network (CNN) model only with word embedding features [Kim, 2014]. However, all these approaches did not consider emojis.
In 2017, Felbo and collegues [Felbo et al., 2017] proposed a powerful system utilizing emojis in Twitter sentiment analysis model called DeepMoji. They trained 1,246 million tweets containing at least one of 64 common emojis with Bi-directional Long Short-Term Memory (Bi-LSTM) model and applied it to interpret the meaning behind the online messages. DeepMoji is also one of the most advanced sarcasmdetecting models and irony reverses the emotion of the literal text, therefore sarcasm-detecting capability can play a significant role in sentiment analysis, especially in case of social media. Although sarcasm and irony tend to convey negative emotions in general, we found that in Chinese social media (Weibo in our example), in addition to the expression of positive and negative emotions, people tend to express a kind of humorous emotion that escapes the traditional bi-polarity.
In their research, [Li et al., 2018b] analyzed the usage of the emojis with facial expression used on Weibo. They asked 12 Chinese native speakers to label these emojis by applying one of three following categories: positive, negative and humorous. They have confirmed that 23 emojis can be considered more as humorous than positive or negative. On this basis, they used the emoji polarities (see Table 1) in a long shortterm memory recurrent neural network (called EPLSTM) for sentiment analysis also on undersized labelled data. [Chen et al., 2018] proposed a novel scheme for Twitter sentiment analysis with extra attention on emojis. They first learned bipolarity emoji embeddings under positive and negative sentimental tweets individually, and trained a sentiment classifier by attending on these bi-polarity emoji embeddings with an attention-based long short-term memory network (LSTM). Their experiments shown that the bi-polarity embedding was effective for extracting sentiment-aware embeddings of emojis. However, humorous posts of social media were not considered in their paper.
An attention-based mechanism usually has been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. [Luong et al., 2015] examined two simple and effective classes of attentional mechanism: a global approach which always uses all source words and a local one that only looks at a subset of source words at a time. Their proposed model using different attention architectures has established a new state-of-the-art result.
Attention-based neural network has also been applied to classification task. Zhou and the others [Zhou et al., 2016] proposed attention-based bidirectional long short-term memory networks (AttBLSTM) to capture the most important semantic information from a sentence. The experimental results on the SemEval-2010 relation classification task have shown that their method outperforms most of the existing methods. [Yang et al., 2016] proposed hierarchical attention networks (HAN) for classifying documents. Their model progressively builds a document vector by aggregating important words into sentence vectors and then aggregating important sentences vectors to document vectors. Experimental results demonstrate that proposed model performs significantly better than previous methods. Results illustrate that this model is effective in choosing out important words also in our study and we decided to adopt it. 3
Inspired by the above-mentioned works, in this paper, we applied emoji polarity to an attention-based bi-directional GRU network model (EAGRU, where “E” stands for Emojis) for sentiment classification of Weibo undersized labelled data. The architecture of the proposed method for sentiment classification is shown in Figure 2. 3.1
The Gated Recurrent Unit [Bahdanau et al., 2014] is a gating mechanism to track the state of sequences without using separate memory cells. There are two types of gates: the reset gate rt and the update gate zt. They together control how information is updated to the state. At time t, the GRU computes the new state as: where xt is the sequence vector at time t. The candidate state eht is computed in a way similar to a traditional recurrent neural network (RNN): eht = tanh(Whxt + rt (Uhht 1) + bh)
Here rt is the reset gate which controls how much the past state contributes to the candidate state. If rt is zero, then it forgets the previous state. The reset gate is updated as follows: rt =
(Wrxt + Urht 1 + br) 3.2
Considering that the entries of Weibo are sentences of less than 140 words, in contrast to related work of [Yang et al., 2016], in our research we focus on sentence-level social media sentiment classification. Assuming that a sentence si contains Ti words, wit with t 2 [1; T ] represents the words in the ith sentence. Our proposed model projects a raw Weibo post into a vector representation, on which we build a classifier to (1) (2) (3) (4) perform sentiment classification. In the below, we introduce how we build the sentence level vector progressively from word vectors by using the attention structure.
Given a post with words wit; t 2 [0; T ], we first vectorize the words through an embedding matrix We; xij = Wewij . We use a bidirectional GRU [Bahdanau et al., 2014] to address word annotations by summarizing information from both directions from words, and therefore incorporate the contextual information in the annotation. The bidirectional GRU contains the forward GRU !f which reads the sentence si from wi1 to wiT and a backward GRU f which reads from wiT to wi1:
xit = Wewit; t 2 [1; T ] ! h it = GR!U (xit); t 2 [1; T ] h it = GRU (xit); t 2 [T; 1] (7) We obtain an annotation for a given word wit by concate! nating the forward hidden state h it and backward hidden ! state h it, for example, hit = [ h it; h it], which summarizes the information of the whole sentence centered around wit. Not all words contribute equally to the representation of the Weibo entry meaning. Hence, we introduce attention mechanism to extract words which are important to the meaning of the post and show how we calculate the total of the representation of those informative words to form a sentence vector. Specifically, (5) (6) (8) (9) (10) (11) 3.3
In order to predict sentiment category of Weibo posts considering the influence of emojis for Chinese social media sentiment analysis, we assign a hyper-parameter 1 to the probability of the deep learning model’s softmax output S(zi). At the same time, we apply the labelled emojis from the work of [Li et al., 2018b] as polarity P e, and assign a hyper-parameter 2. P becomes the final probability output of the classification:
P = 1S(zi) + 2Pe (13) where the summation of 1 and 2 is equal to 1.
As a result, we can obtain the sentiment probability of a Weibo post which considers the effect of emojis. 4
In order to verify the validity of our proposed method, we performed series of experiments described below. 4.1
Initializing word vectors with those obtained from an unsupervised neural language model is a popular method to improve performance in the absence of a large supervised training set. For our experiment we collected a large dataset (7.6 million posts) from Weibo API from May 2015 to July 2017 to be used in calculating word embeddings. Firstly, we deleted images and videos, treating them as noise. Secondly, we used Python Chinese word segmentation module Jieba2 to segment the sentences of the microblogs, and fed the segmentation results into the word2vec model [Mikolov et al., 2013] for training word vectors. The vectors have dimensionality of 300 and were trained using the continuous skip-gram model.
When we collected microblog data, we discovered that Weibo emojis are converted by API into textual tags, for example, will be convert into (“smile”). This provided us with the possibility of representing emojis in word embedding. Therefore, we transformed the 109 Weibo emojis (see Figure 3) into Chinese characters, and converted them into textual features for word embedding. Several examples are shown in Table 2.
Next, we collected 4,000 Weibo posts containing ambiguous eight emojis ( , , , , , , , ), ensuring each entry has only one pictogram of a given type (cases with more emojis of the same type were allowed). To use these posts as our training data, we asked three Chinese native speakers to annotate them into three categories: “positive”, “negative”, and “humorous”. After one annotator labelled polarities of all posts, two other native speakers confirmed correctness of his annotations. Whenever there was a disagreement, all decided the final polarity through discussion.
2https://github.com/fxsjy/jieba uit = tanh(Wwhit + bw) it =
exp(uiTtuw) Pt exp(uiTtuw) si = X
ithit t
We first feed the word annotation hit through a one-layer MLP to get uit as a hidden representation of hit, then we measure the importance of the word as the similarity of uit with a word level context vector uw and get a normalized importance weight it through a softmax function. Secondly, we compute the sentence vector si as a weighted sum of the word annotations based on the weights. The context vector uw can be seen as a high level representation of a fixed query the informative word over the words like those used in memory networks [Sukhbaatar et al., 2015]. The word context vector uw is randomly initialized and jointly learned during the training process. The outputs of softmax layer S(zi) are the probabilities of each category. The softmax function is defined as follows [Bridle, 1990; Merity et al., 2016]:
S(zi) =
ezi
Pij=1 ezj where the input of softmax layer zi is defined as: zi = wix + bi (12) and where w is the weight and b is bias, both of them calculated during the model training process. We trained our EAGRU model with 10 epochs and the performance achieved the highest value when the dropout rate was 0.5. The validity of the model was examined by holdout method (90%/10%, training/validation). In general tanh was used as the activation function and softmax was the network output activation function. 4.3
We compare our EAGRU method with several baseline methods, including traditional deep learning approaches such as convolutional neural network and long short-term memory recurrent neural network.
Convolutional neural networks (CNN) utilize layers with convolving filters that are applied to local features [LeCun et al., 1998]. Originally invented for computer vision, CNN models have subsequently shown to be effective for NLP and have achieved superior results in semantic parsing [Yih et al., 2014], search query retrieval [Shen et al., 2014], sentence modeling [Kalchbrenner et al., 2014], and other traditional NLP tasks.
We experimented with the CNN architecture proposed in [Kim, 2014] and applied our emoji polarities to this model.
The CNN model considering Emoji Polarities (EPCNN) was trained with 10 epochs and the dropout rate was 0.5 (the same as in the proposed method), the filter size was 32 and number of strides was 2. As the activation functions, we used RELU in general, and the network output activation function was softmax.
Long short-term memory recurrent neural network (LSTM) [Hochreiter and Schmidhuber, 1997] is well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series [Eyben et al., 2010].
We utilized EPLSTM proposed in [Li et al., 2018b] trained with 10 epochs and the dropout rate was 0.5 identical with our proposed method. The validity of the model was examined by holdout method (90%/10%, training/validation). The network output activation function was also softmax. 4.4
Using a trained word2vec model, we passed word vectors of training data into the three deep learning models for training. We collected and annotated 180 Weibo entries with the eight emojis mentioned above as a testing set, deleting images and videos. Then we used the proposed method to calculate probabilities of each category and confirmed the precision, recall and F1-score. Because we assumed that in emotion expression emojis might play an equal or greater role than text, in our experiment we set the hyper-parameters 1 and 2 to 0.4 and 0.6 respectively.
We compared the results of sentiment classification by deep learning approaches with and without considering emoji polarities. Results of deep learning models without emojis are shown in Table 3. Table 4 introduces results of two traditional deep learning approaches where emoji polarities were considered, and the results of our proposed method. Table 5 describes the comparison of F1-scores of the above-mentioned methods.
The results proved that our proposed method is more effective than traditional neural network-based solutions. Limited to small annotated data, the precision of the sentiment classification was relatively low, but thanks to considering emoji, the F1-score of each category outperformed previous methods without considering emojis by 6.93 (humorous), 7.41 (negative) and 7.19 (positive) percentage points. Our proposed emoji-aware attention-based GRU network approach has improved the performance showing that low-cost, smallscale data labeling is sufficient to outperform widely used state-of-the-art when emoji information is added to the deep learning process. 5
In our proposed approach, we paid attention to emojis in microblogs and investigated how adding pictogram features to a attention-based GRU network model for recognizing humorous posts which are problematic in sentiment analysis. Figure 4 presents an example of a microblog which was correctly classified by our proposed method as “humorous” while the baseline recognized it incorrectly as a positive one.
This and similar entries were usually posted as a comment a GIF or video showing a referee who displays her or his skills in basketball by performing a slam dunk. This post seems to express an implied humorous nuance of exaggerated surprise when the poster saw how good the referee was. Because this expression is accompanied by emoji, it improves the performance of classification and predicts the implicit humorous meaning.
Error analysis showed that some posts were wrongly predicted due to ambiguous usage of emojis which brought clearly negative impact on the results. In Figure 5 we show an example of such misclassification into “positive” category annotated as “humorous” by annotators. was considered as more positive than humorous by our annotators (67%/0%/33%, positive/negative/humorous). It seems that this particular user wrote a joke just for fun, however, our proposed method was misguided by this “smirking” emoji. Therefore, we plan to increase the number of evaluators for annotating Weibo emojis in fine-grained humorous emotion to enhance the reliability of the polarity of emojis. 6
In this paper, we applied information on sentiment of emojis to a attention-based GRU network model for sentiment analysis of undersized labelled data. Our experimental results show that the proposed method can significantly improve the F1-score for predicting sentiment polarity on Weibo.
For improving the performance of our proposed method, in the near future we are going to increase the amount of labelled data to acquire the hyperparameters automatically by machine learning approaches. Furthermore, we need to increase the number of evaluators for annotating Weibo emojis and Weibo data for more fine-grained categorization of humorous posts to enhance the reliability of our experiments. We also plan to add image processing for classifying stickers which also seem to convey rich emotional information.
Our ultimate goal is to investigate how much the newly introduced features are beneficial for sentiment analysis by feeding them to a deep learning model which should allow us to construct a high-quality sentiment recognizer for wider spectrum of sentiment in Chinese language. 7
This work was supported by JSPS KAKENHI Grant Number 17K00295.