English. In this paper, we present our approach that has won the ITAmoji task of the 2018 edition of the EVALITA evaluation campaign1. ITAmoji is a classification task for predicting the most probable emoji (a total of 25 classes) to go along with the target tweet written by a given person in Italian. We demonstrate that using only textual features is insufficient to achieve reasonable performance levels on this task and propose a system that is able to benefit from the multimodal information contained in the training set, enabling significant F1 gains and earning us the first place in the final ranking.
Italiano. In questo articolo presentiamo l’approccio con cui abbiamo vinto la competizione ITAmoji dell’edizione 2018 di EVALITA1. ITAmoji e` un task di classificazione per predire l’emoji piu` probabile (tra un totale di 25 classi) che possa essere associato ad un dato tweet scritto in italiano da uno specifico utente. Dimostriamo che utilizzare esclusivamente dati testuali non e` sufficiente per ottenere un ragionevole livello di performance su questo task, e proponiamo un sistema in grado di beneficiare dalle informazioni multimodali contenute nel training set, aumentando significativamente lo score F1 e guadagnando la prima posizione nella classifica finale.
Particularly over the last few years, with the increasing presence of social networks and instant 1EVALITA: http://evalita.it/2018 messaging services in our lives, we have been witnessing how common it has become for average users to enrich natural language by means of emojis. An emoji is essentially a symbol placed directly into the text, which is meant to convey a simple concept or more specifically, as the name says, an emotion.
The emoji phenomenon has attracted
considerable research interest. In particular, recent works
have studied the connection between the natural
language and the emojis used in a specific piece
of text. The 2018 edition of EVALITA ITAmoji
competition
In this paper, we describe our neural networkbased system that exhibited the best performance among the submitted approaches in this task. Our approach is able to successfully exploit user information, such as the prior emoji usage history of a user, in conjunction with the textual features that are customary for this task. In our experiments, we have found that the usage of just the textual information from the tweet provides limited results: none of our text-based models were able to outperform a simple rule-based baseline based on prior emoji history of a target user. However, by considering all the modalities of the input data that were made available to us, we were able to improve our results significantly. Specifically, we combine into a single efficient neural network the typical Bi-LSTM-based recurrent architecture, that has shown excellent performance previously in this task, with the multilayer perceptron applied to user-based features. ITAmoji task is a classification task of predicting one of the 25 emojis to go along with the tweet. The training set provided by the organizers of the competition consists of 250 000 Italian tweets, including for each tweet the text (without the target emoji), the user ID and the timestamp as features. Participants were explicitly forbidden to expand the training set. Figure 1 provides an overview of our approach. In this section, we provide detailed descriptions of the methods we employed to solve the proposed task. 2.1
In order to embed the textual content of the tweet,
we have decided to apply a vectorization based on
fastText
2http://commoncrawl.org/ 3https://github.com/facebookresearch/ fastText/blob/master/docs/crawl-vectors. md
1467220 2.2
Rather than relying solely on a text of the target tweet, we exploit additional user-based features to improve performance. The task features many variations of the smiling face and three different heart emojis, making it impossible even for a human to determine the most suitable one just based on a tweet. One of the features we considered was the prior emoji distribution for a target author. The hypothesis was that the choice of a particular emoji is driven mainly by the personal user preferences exemplified by the previous emoji choices.
To this end, we have collected two different types of emoji history for each user. Firstly, we use labels in the training set to compute emoji distributions for each user yielding vectors of size 25. Users from the test set that were not present in the training set were initialized with zeroes. Secondly, we have gathered the last 200 tweets for each user using Twitter API5, and then extracted and counted all emojis that were present in those tweets. This yielded a sparse vector of size 1284. At this step we took extra care to prevent data leaks: if a tweet from the test set ended up in the collected 200 tweets, it wasn’t considered in the user history. The runs that used the former, training set-based approach had a ” tr” suffix in its name. The ones that used the full user historybased approach had a ” ud” suffix.
In addition to prior emoji distribution, we did
preliminary experiments with user’s social graph.
Social graph, which is a graph of connections
between the users, is shown to be an important
feature for many tasks on social media, for example,
user profiling. We followed the recently proposed
approach
The Recurrent Neural Networks have turned out to be a powerful architecture when it comes to analyzing and performing prediction on sequential data. In particular, over the last few years, different variations of the RNN has shown to be the top performing approaches for a wide variety of tasks, including tasks in Natural Language Processing (NLP). RNN consumes the input sequence one element at the time, modifying the internal state along the way to capture relevant information from the sequence. When used for NLP tasks, RNN is able to consider the entirety of the target sentence, capturing even the longest dependencies within the text. In our system, we use the bi-directional long short-term memory (Bi-LSTM) variation of the RNN. This variation uses two separate RNNs to traverse the input sequence in both directions (hence bi-directional) and employs LSTM cells.
Input text provided by the organizers is split into tokens using a modified version of the Keras tokenizer (can be found in our repository). Then, the input tokens are turned into word vectors of fixed dimensionality using the embedding matrix of one of the approaches listed in Section 2.1. The resulting sequence is padded with zeroes to a constant length, in our case 48, and fed into the neural network. 2.4
In order to accommodate both textual and userbased features, we devise a joint architecture that takes both types of features as input and produces probability distribution for the target 25 classes. The general logic of our approach is shown in Figure 1. The core consists of two main components: Bi-LSTM. The recurrent unit consumes the input sequence one vector at a time, modifying the hidden state (i.e., memory). After the whole sequence is consumed in both directions, the internal states of the two RNNs are concatenated and used as a tweet embedding. Additionally, we perform l2-regularization of the input embedding matrix and the dropout to prevent overfitting and fine-tune the performance. Table 1a details the hyperparameters we used for a textual part of our approach. User-based features. The emoji distribution we collected (as described in Section 2.2) was fed as input to a multilayer perceptron: two densely-connected layers with tanh as activation and l2-regularization to prevent overfitting. Table 1b showcases the chosen hyperparameters for this component using the full user history as input.
The outputs of the two components are then concatenated and a final layer with the sof tmax activation is applied to acquire the probability distribution of the 25 emoji labels. The network is then optimized jointly with cross entropy as the objective function using Adam optimizer. Table 1c includes all relevant hyperparameters we used for this step.
Since the runs are evaluated based on macro-F1, in order to optimize our approach for this metric, we have introduced class weights into the objective function. Each class i is associated with the weight equal to: wi = maxi(N )
Ni (1) where Ni is the amount of samples in a particular class and = 1:1 is a hyperparameter we tuned for this task. This way the optimizer is assigning a greater penalty for mistakes in rare classes, thus optimising for the target metric.
During the training of our approach, we employ an early stopping criteria to halt the training once the performance on the validation set stops improving. In order to properly evaluate our system, we employ 10-fold cross-validation, additionally extracting a small validation set from the training set for that fold to perform the early stopping. For the final submission we use a simple ensemble mechanism, where predictions are acquired independently from each fold and then averaged out to produce the final submission. Additionally, one of the runs was submitted using predictions from the random fold. Runs exploiting the ensemble approach have the ” 10f” suffix, while runs using just one fold have the ” 1f” suffix.
The code used to preprocess data, train and evaluate our approach is available on GitHub6. 3
In this section, we provide details on some of the approaches we have tested during the development of our system, as well as the models we submitted for the official evaluation. In this paper, we report results for the following models:
MF HISTORY. A rule-based baseline that always outputs the most frequent emoji from the user history based on a training set.
BASE CNN. A basic Convolutional Neural Network (CNN) taking word embeddings as input without any user-based features.
BASE LSTM. A Bi-LSTM model described in Section 2.3 used with textual features only. BASE LSTM TR. The complete approach including both feature families with emoji distribution coming from the training set.
BASE LSTM UD. The complete approach with emoji distribution coming from the most recent 200 tweets for each user.
For the other models tested during our local evaluation and complete experimental results, please refer to our GitHub repository.
Additionally, for the BASE LSTM approach we report performance variations due to a choice
Remper/emojinet https://github.com/ of a particular word embedding approach. In particular, provided refers to the ones that were suggested by organisers, custom-100d indicates our fastText-based embeddings and common-300d refers to the ones available on fastText website. Table 2 details the performances of the mentioned models.
Finally, we submitted three of our best models for the official evaluation. All of the submitted runs use the Bi-LSTM approach with our custom-100d word embeddings along with some variation of user emoji distribution as detailed in Section 2.2. Two of the runs use the ensembling trick using all available cross-validation folds, while the remaining one we submitted (” 1f”) uses predictions from just one fold. 4
Here we report performances of the models benchmarked both during our local evaluation (Table 2) and the official results (Table 3). We started experiments with just the textual models testing different architectures and embedding combinations. Among those, the Bi-LSTM architecture was a clear choice, providing 1-2% F1 over CNN, which led to us abandoning the CNN-based models. Among the three word embedding models we evaluated, our custom-100d embedding exhibited the best performance on Bi-LSTM, while common-300d showed the best performance using the CNN architecture.
After we have acquired the user emoji distributions, we have devised a simple baseline (MF HISTORY), which, to our surprise, outperformed all the text-based models we’ve tested so far: 3% F1 improvement compared to the best BiLSTM model. When we introduced the user emoji histories in our approach, we have gained a significant performance gain: 4% when using the scarce training set data and 12% when using the complete user history of 1284 emojis from recent tweets. During the final days of the competition, we have tried to exploit other user-based features to further bolster our results, for example, the social graph of a user. Unfortunately, such experiments did not yield performance gains before the deadline.
During the official evaluation, complete user history-based runs exhibited top performance with ensembling trick actually decreasing the final F1. As we expected from our experiments, training set-based emoji distribution was much less perApproach MF HISTORY BASE CNN BASE LSTM BASE LSTM BASE LSTM BASE LSTM TR BASE LSTM UD
Embedding
– common-300d common-300d
provided custom-100d custom-100d custom-100d Run BASE UD 1F BASE UD 10F BASE TR 10F gw2017 p.list F1 macro formant but still offered significant improvement over the runner-up team (gw2017 p.list) as shown in Table 3. Additionally, we detail the performance of our best submission (BASE UD 1F) for each individual emoji in Table 4 and Figure 2. 5
Our findings suggest that emojis are currently used mostly based on user preferences: the more prior user history we added, the more significant performance boost we have observed. Therefore, the emojis in a text cannot be considered independently from the person that has used them and textual features alone can not yield a sufficiently performant approach for predicting emojis. Additionally, we have shown that the task was sensitive to the choice of a particular neural architecture as well as to the choice of the word embeddings used to represent text.
An analogous task was proposed to the
participants of the SemEval 2018 competition. The
winners of that edition applied an SVM-based
approach for the classification
In the future, we would like to investigate this
topic further by trying to study differences in
emoji usage between languages and communities.
Additionally, we aim to further improve our
approach by identifying more user-based features,
for example, by taking into account the feature
families suggested by Nechaev et al.