We present our solution to the problem proposed by IROSTEREO's PAN Shared Task in 2022. It proposes the detection of irony and stereotype spreaders on Twitter. Throughout the memory, we will show how through: the technique based on neural networks for the pre-training of natural language processing such as BERT, the use of sentence embeddings, and two alternatives to put them together; it will be achieved from a set of tweets associated with a set of authors, predict whether an author is ironic or not. Lastly, it will be presented a model that achieves high accuracy and with which finally participated in the competition.
The data, for creating the first models, is located inside a folder, it contains 420 XML files, each of them referring to diferent authors; these authors are represented with an id. Each XML file is composed of 200 tweets from the author, these are in the English language. A truth.txt file is also provided which contains the tag associated with each author, this tag can be I if the author is ironic, or NI if the author is not ironic.
After performing the diferent experiments with the previous dataset, we were provided with a test dataset, which consists of 180 authors with 200 tweets each. Unlike the previous dataset, this one does not contain the truth.txt, since this is the one that will be taken into account to be evaluated by the contest.
It has been proposed to solve this problem by using Machine Learning based models and
without the use of a pre-processing, as the tweets are already pre-processed. Firstly, the
sentence embedding representation is going to be used through BERT [
After obtaining a sentence embedding for each of the authors (with either of the two methods), we will proceed to the training of a classical model. In our case, we have chosen to use the sklearn library, which ofers a set of classical models; we will pass these sentence embeddings and the associated author label to these models for training.
This section shows the results, using accuracy as a metric, of both methods presented in the previous section for diferent classical classifiers. The classifiers that have been used are the SVM1, MLP2, GaussianNB and RandomForest. It is important to note that all experiments have been performed using 10-fold cross-validation.
1Support Vector Machine 2Multi Layer Perceptron
The best result obtained from each of the classifiers and the value of their hyperparameters are shown below.
C=0.0001 hl1=128, hl2=64, hl3=32
smoothing=0.0001 max_depth=10, n_estimators=100, max_features=50
0.53809 0.64286 0.63095 0.64248
As shown in Table 1, the model with the highest accuracy is the MLP model when 3 hidden layers are incorporated between the input and output layers. Despite this, the accuracy value obtained is not entirely promising, since accuracy of 0.64286 in the classification of 2 diferent classes is not a very optimistic value. In the following method, the strategy implemented in this method will be improved.
The best result obtained from each of the pre-trained models used, the best classifier obtained with each one, and the value of its hyperparameters are shown below.
As we can see in Table 2 , we got a great improvement in the results achieved by averaging after (and not before) putting the data into the pre-training model. The best performer was given by the Multilingual Universal Sentence Encoder, although the diference is not significant.
hl1=32, hl2=32, hl3=16
C=21 hl1=4, hl2=64, hl3=32
Finally, we have to train a final model with all the data with the characteristics that we have seen that have performed best in the experiments. In order not to over-fit we are going to put only a few iterations, 250, since we have observed that if we do more we easily reach 100% accuracy in the training samples, which is not desirable. So we have trained a final MLP with 4, 64, and 32 hidden units for 250 iterations that receive as input the averaged embeddings of a user’s tweets that have been taken as input from the MUSE.
In the first submission to the TIRA platform [
As shown throughout the experiments and results, the use of MUSE together with an MLP has proven to be the most successful case for maximizing accuracy and detecting the presence of irony or non-irony of an author within a set of tweets associated with him/her.