English. This document describes the results of our system in the evaluation campaign on the prediction of Emoji in Italian, organized in the context of EVALITA 20181 (Ronzano et al., 2018). Given the text of a tweet in Italian, the task is to predict the emoji most likely associated with that tweet among the 25 emojis selected by the organizers. In this report, we describe the three proposed systems for evaluation. The approach described starts from the possibility of creating two different models, one for the part of categorization, and the other for the part of polarity. And to combine the two models to get a better understanding of the dataset. Italiano. Questo documento descrive i nostri risultati del nostro sistema nella campagna di valutazione sulla predizione delle Emoji in italiano, organizzata nel contesto di EVALITA 2018.
Dato il testo di un tweet in italiano, il task consiste nel predire l’emoji più probabilmente associata a quel tweet tra le 25 emojis selezionate dagli organizzatori. In questo report descriviamo i tre sistemi proposti per la valutazione. L'approccio descritto parte dalla possibilità di creare due modelli diversi, uno per la parte di categorizzazione, e l'altro per la parte di polarità. E di unire i due modelli per ottenere una maggiore comprensione del dataset. 1
In the field of communication, the importance of addressing your audience with a common language in which the customer can recognize and identify with each other is fundamental. In social interactions, an increasing amount of communication occurs in a non-verbal way, as with emoji. Being able to predict the best emoji to use in a message can increase the perception of the same and give strength to the message itself.
In the context of the Italian Emoji Prediction task
called ITAmoji, we have tried to predict one of
25 possible emojis from different tweets.
Despite the knowledge of how a system of SVM
could be the best solution for the problem, as per
the previous context SemEval 2018
We first started by cleaning the given data from all the noise information. All the punctuation marks were removed from the text of tweets, and we focused on cleaning the text and removing ambiguities such as shortened words and abbreviations. We substituted all the hyperlinks with a more generic word “LINK” and we did the same with the usernames preceded by ‘@’ (users’ tags), after seeing that it was not relevant in the prediction of the most likely emoji for the tweet. We tried removing the stop words from the tweets’ text to leave only the words with relevant meaning in it, but the results were poor. Then we converted every word of the tweet’s text into its lemma, and while doing the lemmatization, we saw that sometimes the username was misleading in the text, so we chose to remove it and substitute it with a more generic word ‘USERNAME’.
We used two different fastText2 vectors created in the 2016 and the other created in 2017, all with Italian tweets containing at least one emojis. The idea is to analyze if different fastText vectors created with tweets published in different periods could discover the use of the emojis and its evolution over the time.
The system created is an Ensemble of two
different models to replicate the result obtained in the
emotion classification
The second is a LightGBM4, where the following properties are extracted from the tweet text: • • • • • • length of the tweet percentage of special characters the number of exclamation points the number of question marks the number of words the number of characters https://fasttext.cc https://keras.io https://github.com/Microsoft/LightGBM the ratio between words and stop words the ratio between words and spaces the ratio between words and hashtags and are joined to the vector created by the bigram and the trigram of the tweet itself at word and character level.
The number of leaves is 250, the learner set as ‘Feature’, and the learning rate at 0.04.
The ensemble is done in the weighted average when the BI_LSTM decide the 60% of the vote and the LightGBM the 40%.
It was also tried to add a linear classifier but the attempt did not provide any advantage. The cross-validation task to find a good weight was ineffectual and the provision was insignificant. 3
The results of the Bi-LSTM were:
BI-LSTM with 2016 fastText
LightGBM only text precision
recall 0.2399 0.3094 Table 3: precision, recall, and F1 score The LightGBM model was also tested by adding to the already mentioned properties additional information such as the user ID and information extracted from the tweet date such as day, month, the day of the week and time.
The results obtained also indicate here that there is a correspondence between the use of emojis, the user, the time and the day. For example the Christmas tree in December or the heart emoji in the evening hours.
LightGBM with user and date precision
recall 0.5044 0.2331 Table 4: precision, recall, and F1 score The level of Precision obtained in this way was very high even if the F1 score is still lower than the BI-LSTM model.
To avoid the unbalancing of the emojis present in the training dataset various undersampling and oversampling operations were performed without any appreciable results.
Turning to the result of the ensemble of the two models we had a marked increase in the F1 score thanks to the substantial growth of the Recall in both cases.
In the tables 5 and 6 there are the results from the minimum and the maximum F1 score obtained during the process of the ensemble. f1-score precision f1-score f1-score quantity BI-LSTM with 2017 fastText + LightGBM with user and date precision
recall
0.3650 0.2917
Table 6: precision, recall, and F1 score
The result of the validation was however very far
from that obtained during the training phase. It
will be necessary to evaluate if, as in the research
Exploring Emoji Usage and Prediction Through
a Temporal Variation Lens
If the tweets analyzed were too different from those of the training dataset, if the users in the test dataset have different behaviors, or if the system suffered from some kind of overfitting (visible in the third submission, gw2017_pe). Macro F1 Micro F1 Weighted F1 Coverage error Accuracy at 5 Accuracy at 10 Accuracy at 15 gw2017_e gw2017_p
gw2017_pe 0.222082 0.421920 0.368996 4.601440 0.713000 0.859040 0.943080 0.232940 0.400920 0.378105 5.661600 0.671840 0.814880 0.894160 0.037520 0.119480 0.109664 13.489400 0.279280 0.430360 0.560000 Accuracy at 20 0.982520 0.929920 0.662720 Table 7: macro F1, micro F1, weighted F1, coverage error, accuracy at 5, 10, 15 and 20 for the three runs submitted.
In table 8 we can observe the result of the three submissions split for each emoji.
In the study of the dataset, three critical issues emerged.
l
The first is that the use of similar emojis seems more dictated by a personal choice of the user.
There are not many pieces of evidence because the use of one emoji is preferred.
In particular for the following emoji: l
The second is that, especially in cases where a tweet begins by indicating a USERNAME, or in a mention or a direct response, the use of emoji takes on a sub-language value. That is, the use of a specific word or emoji has a meaning that only the tweet recipients know. Use of emoji and
could be irony or just references to previous pasted experiences in common. l
Thirdly, the strong imbalance of the training dataset is not the only reason for the unbalanced prediction of some emojis, as in the case of and . 5
The result of the ensemble was pretty good and demonstrate the validity of this kind of approach. The use of emoji is personal and also depends on the context and the people in the discussion. A system with the emojis with the same meaning merged could be more proficient and ready for the production.
In the near future, we will evaluate the speed and effectiveness of a CNN model in which the operation of the BI-LSTM and the features extrapolation used in the LightGBM model can be merged during the same training session.
We will also focus on the creation of fastText vectors of different size containing tweets for specific contexts and published in different periods to identify the periodicity and variation in the use of particular emoji. The intent is to discover other hidden patterns, more than the obvious that has emerged for the holiday periods.