This paper describes the participation of the PiLN team in the IberLEF 2021 shared task on Irony Detection in Portuguese. The goal of this task is to create a system to detect irony in tweets and news texts. To deal with this topic, we develop a super cial featurebased strategy and an embeddings approach. Moreover, we adopt a backtranslation method as data-augmentation to make the tweets Corpus less unbalanced. We evaluate our approaches within several machine learningbased classi ers and take the rst and second places in the tweets Corpus category and seventh and thirteenth places in the news Corpus.
According to Oxford dictionary [
When dealing with textual content on the internet, it is possible to observe
the use of ironies as a mechanism to reinforce an utterance or e ciently state
a point of view. Therefore, automatic irony detection may be useful for several
tasks, such as detecting online harassment, identifying author pro ling, opinion
mining, and others [
In this paper, we describe our strategies to deal with automatic irony
detection in tweets and news texts in the IberLEF 2021 shared task on Irony Detection
in Portuguese [
The organizers of this shared task released two Corpora with ironic texts: tweets and news. In the training set, the rst Corpus has 15; 212 tweets, while the second one has 18; 494 news. Table 1 presents the number of ironic and nonironic texts for each Corpus. As we can see, the Corpora are unbalanced. The Corpus of news is unbalanced regarding the ironic label, whereas the Corpus of tweets is unbalanced concerning the non-ironic label.
From these Corpora, the goal of this task was de ned as a binary classi cation problem where the participants were asked to classify a text (tweets and news) as ironic or non-ironic.
In what follows, we detail our strategy to handle ironic texts. 3
In order to make the Corpus of tweets less unbalanced, we used the Corpus of
the SemEval 2018 Task 3 (Irony Detection in English Tweets) [
From this gure, we translated the original tweets from the SemEval
Corpus to Portuguese, using the machine translation model provided by the Google
Translate API3. We used that model because it achieved good results in the
semantic parsing [
Original tweets
English
Tweets Compare
Translate to Portuguese
Translated tweets
English Tweets
Portuguese
Tweets
Translate back
to English
there are no Portuguese reference tweets to evaluate the quality of the
translations. In this way, we may measure the quality of the translations, comparing
the original tweets with the back-translated tweets. To evaluate the quality of
the translations, we computed the harmonic mean between the ROUGE4 [
F1 = 2 rouge bleu rouge + bleu (1)
Based on that strategy, we obtained f-score values as shown in Table 2, when comparing the original tweets with the back-translated tweets. From this table, 25%, 50%, and 75% refer to the rst, second, and third quartile, respectively.
Taking into account the values of Table 2, we got the non-ironic tweets with f-scores greater than 0:79, which equals 680 tweets corresponding to the best translations, to make the Corpus of tweets less unbalanced. Our new Corpus of tweets has 12; 736 ironic tweets and 3; 156 non-ironic tweets. 4 We used the F-score of the ROUGE-L.
With the Corpus of news and the new Corpus of tweets, we adopted a method based on super cial features and an embedding approach to extract features of Corpora, train classi ers to predict tweets or news as ironic or non-ironic. In subsection 3.1, we detail our super cial feature method, and in subsection 3.2, we describe the embedding approach. 3.1
Super cial features In this approach, we tried several linguistic features to identify ironic texts, such as: number of named entities, presence/absence of some symbols, expressions, number of emojis, frequent words, among others. Table 3 presents the analyzed features in the Tweets and News datasets.
Number of tokens formed only by letters Number of Named Entities in text Percentage of wrong words Number of wrong words Percentage of words written in uppercase Use of quotation marks in text Use of expressions: `sqn', `SQN', `so que na~o', `SO QUE N A~O', `so que na~o', `SO QUE N A~O', `so que n~', `SO QUE N~', `So que na~o' Excessive use of question and exclamation marks Number of emojis in text Number of hashtags in text Number of smiley face symbols Use of laugh expressions in text: `kkk', `hahaha', `hehehe', `rsrsrs' Number of a ective adjectives in text 10 most frequent words in each class: ironic and non ironic, excluding repeated words
From this feature list, we experimented several machine learning algorithms
and observed that the most relevant feature is the f reqW ords. Thus, we adopted
a very simple strategy to detect an ironic text. We computed the Term
FrequencyInverse Document Frequency (TF-IDF) to extract features from the Tweets and
News Corpora, and we fed them into the Linear Support Vector Machine (SVM),
using the Stochastic Gradient Descent (SGD) technique to train the classi er.
Moreover, we applied a greedy search method to nd the best parameters for the
TF-IDF and the classi er. Table 4 presents the found out parameters for tweets
and news. From this table, number of features is the vector size, max. df refers
to the document frequency, and max. iterations is the number of iterations of
the training algorithm.
According to Le and Mikolov [
To extract embedding features from the Corpora, we trained the Distributed
Bag of Words Paragraph Vector model [
From this table, vector size is the dimensionality of vectors, i.e., each text has 300-dimensional embeddings, dm is the training algorithm, where the 0 value refers to the Distributed Bag of Words version of Paragraph Vector. We set 0:025 as initial learning rate on parameter alpha, 300 epochs, which is the number of iterations over the corpus, and choose to ignore words with lower frequency than 1 on parameter min count. The parameter window represents the distance between the current and predicted word, de ned by size 4 and for parameters sample and workers, we set empirical values based on initial experiments.
In the next section, we present our conducted experiments and obtained results. 4
To assess the systems for this shared task, the organizers made available test
sets for the news and tweets Corpora, where each test set has 300 texts. The
systems were ranked by the balanced accuracy metric (BACC) [
In general, one can see that the super cial features approaches outperformed the embeddings strategy. The features have discriminated better an ironic text from non-ironic text than embeddings. Moreover, we believe that the backtranslation method helped to produce the best results for the tweets Corpus.
The methods for the news Corpus achieved better scores than the tweets Corpus. We believe that it is due to two reasons: the news Corpus is larger than the tweets Corpus and the text style of the tweets since they are usually short, unstructured and seldom obey grammar and punctuation rules. Hence, generally, tweets texts are more challenging than news texts.
For the news Corpus, the embedding method seems to be unable to generalize
well, likely because of the news length. This result suggests the necessity to
investigate a more robust architecture. We believe that a transformer architecture,
as BERT [
It is important to highlight that simple approaches, as TF-IDF and embeddings, produce good results for the irony detection task. Our trained models are available at https://github.com/rafaelanchieta/PiLN/. 5
In this paper, we presented our strategies to IberLEF 2021 shared task on Irony Detection in Portuguese. We developed a method based on super cial features and an embedding-based approach. More than that, we used a back-translation strategy as data-augmentation to make the tweets Corpus less unbalanced. Our super cial feature approach ranked 1st and 7th and the embedding method ranked 2nd and 13th for the tweets and news Corpora, respectively. These results suggest that the data-augmentation strategy help to obtain the best results in the tweets Corpus and that it is necessary to investigate more robust approaches to the news Corpus.