This paper describes the participation of ELiRF-UPV team at the three subtasks proposed at IroSvA 2019 shared task. We have developed a model based on Transformer Encoders and Spanish Twitter embeddings learned from a large amount of tweets downloaded at our laboratory. Transformer Encoders are able to model long-range complex relationships among terms in a text without convolutional or recurrent layers. We addressed the three subtasks, related to three Spanish variants, using the same model. The results obtained on the validation corpus seems to con rm the adequacy of the proposed model for the irony detection task. In the nal ranking, our proposal is the only system that consistently outperforms the baselines of the organizers, being the rst ranked system by a considerable margin of Macro F1 averaged on the three subtasks.
Irony is a rhetorical device in which words are used in such a way that their
intended meaning is di erent from the actual meaning of the words. The
automatic detection of irony is an emerging topic in many natural language processing
tasks. It has important implications in the nal performance of some
applications that need automatic processing of texts, mainly if a semantic analysis is
required. For example, in tasks of sentiment analysis, polarity tends to change
when irony is used. In the Semeval workshop framework, tasks such as Task-11:
Sentiment Analysis of Figurative Language in Twitter at SemEval 2015 [
In this paper, we describe the main characteristics of the system designed
by the ELiRF-UPV team to address the tasks proposed at the IroSvA 2019
shared task [
In this section, we discuss the system architecture proposed to address all three IroSvA19 sub-tasks as well as the description of the resources used and the preprocessing applied to the tweets. 2.1
In order to learn a word embedding model for Twitter in Spanish, we downloaded
87 million tweets of several Spanish variants. To provide the embedding layer of
our system with a rich semantic representation on the Twitter domain, we use
300-dimensional word embeddings extracted from a skip-gram model [
We have applied the same preprocessing to all the given data, both the tweets used to learn the Word2Vec embeddings model and those provided by the organization to learn the irony detection model. Firstly, a case-folding process is applied to all the tweets; Secondly, we tokenized the tweets by using TokTokTokenizer from NLTK. Thirdly, user mentions, hashtags and URLS are replaced by three generic-class tokens (user, hashtag and url respectively); Finally, elongated tokens are diselongated allowing the same vowel to appear only twice consecutively in a token (e.g. jaaaa becomes jaa). 2.2
Our irony detection system is based on the Transformer [
Concretely, we use only the encoder part in order to extract vector representations that are useful to determine the presence of irony. We denote this encoding part of the Transformer model as Transformer Encoder. Figure 1 shows a representation of the proposed architecture for irony detection.
The input of the model is a tweet X = fx1; x2; :::; xT : xi 2 f0; :::; V gg where
T is the maximum length of the tweet and V is the vocabulary size. This tweet is
sent to a d-dimensional xed embedding layer, E, initialized with the weights of
our embedding model. Moreover, to take into account positional information we
also experimented with the sine and cosine functions proposed in [
Due to a vector representation is required to train classi ers on top of these encoders, a global average pooling mechanism was applied to the output of the last encoder, and it is used as input to a feed-forward neural network, with only one hidden layer, whose output layer computes a probability distribution over the the two classes of the task C = fIronic; N oIronicg.
We use Adam as update rule with 1 = 0:9 and 2 = 0:999 and Noam as
learning rate schedule [
3
The three subtasks proposed at IroSvA19 have the same goal: determine if a text sample is ironic or not according to a given context. The di erences among them are the Spanish variant in which the text is written and the kind of text to be classi ed. Subtask A aims to detect irony in Spanish tweets from Spain, Subtask B aims to detect irony in Mexican Spanish tweets, and Subtask C aims to detect irony in Spanish news comments from Cuba. In this work we used only the text sample, dispensing with the context.
In order to address the three subtasks, IroSvA19 organization provided three sample sets (one per subtask). Each set is composed by 2,400 labeled documents; 1,600 of which are labeled as N oIronic and the remaining 800 are labeled as Ironic. We divided each set provided by the organization into two subsets, a training set of 2,100 samples and a development set of 300 samples. To do this, we selected 200 N oIronic and 100 Ironic samples for development, maintaining the 2:1 imbalance towards the N oIronic class both in the training and the development sets.
During the training phase, we xed some hyper-parameters, concretely: dk = 64, dff = d, T = 50, h = 8 and batch size = 32. Another hyper-parameters such as p on warmup steps were set following some preliminary experiments to p = 0:7 and warmup steps = 15 epochs.
Moreover, we compare our proposal, which is based on Transformer Encoders
(TE), with another deep learning systems such as Deep Averaging Networks
(DAN) [
Also it is interesting to observe how some system mechanisms, like the positional encodings, or hyper-parameters like N x a ect to the results obtained in terms of macro-F1 (M F1), macro-recall (M R), macro-precision (M P ) and class level metrics ((F1i; Pi; Ri) : i 2 0 : N oIronic; 1 : Ironic). Concretely, we tried to remove the positional information and 1 N x 2 encoders. All these variants are applied only to the spanish subtask and the best two con gurations are used also in the remaining subtasks. All these results are shown in Table 1.
As shown in Table 1, for the 1-TE-Pos and 2-TE-Pos systems, the positional encoding information harms the performance of the system. Moreover, the results obtained with 1-TE-Pos are very similar to those obtained with 1-layer AttLSTM, that seems to indicate that the positional information, by using positional encodings or the internal memory of the LSTM, is not useful for the Spanish subtask.
It is interesting to see that when the positional information is not used, only one encoder behaves well, however, using N x = 2 in this case, hurts the performance of the system in comparison to N x = 1. This e ect does not happen when the positional information is considered, which seems to indicate that a large number of parameters are required to take into account the positional information.
The system 1-TE-NoPos outperforms the other systems, almost in all metrics, except on the precision over the class 1 and the recall over the class 0 with respect to DAN. Moreover, the F1 over the class 0 is very similar between both systems. However, the improvements provided by the 1-TE-NoPos system ( 4.5 points of F1 on the class 1, precision and recall on the class 0 as long as the improvement of 12 points in the recall of the class 1) make this system more competitive than DAN in terms of the macro metrics.
Then, due to these two systems are the most competitive on the development set of the ES subtask, we experimented with these architectures in the other subtasks to observe their behaviour.
On the MX subtask, the results between both systems are similar, obtaining again the system 1-TE-NoPos the best results in all the metrics. However, on the CU subtask, the di erences among the results of both systems are bigger, with improvements of 9 points of M F1, M P and M R.
Finally, our best system 1-TE-NoPos (ELiRF-UPV) is used to label the test
set. The results obtained are shown in Table 2. Our system outperforms the
proposed baselines [
We have proposed a system based on the encoder part of the Transformer architecture in order to extract useful word representations that are discriminative to decide the presence of irony on sort texts. The results obtained by our system are very promising especially considering they have been obtained without an extensive experimentation on the hyperparameters of the model. This opens the door to future improvements by exploring modi cations on the architecture and its hyperparameters.
This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R) and the GiSPRO project (PROMETEU/2018/176). Work of Jose-Angel Gonzalez is nanced by Universitat Politecnica de Valencia under grant PAID-01-17.