English. This paper describes our UO IRO system developed for participating in the shared task IronITA, organized within EVALITA: 2018 Workshop. Our approach is based on a deep learning model informed with linguistic knowledge. Specifically, a Convolutional (CNN) and Long Short Term Memory (LSTM) neural network are ensembled, also, the model is informed with linguistics information incorporated through its second to last hidden layer. Results achieved by our system are encouraged, however a more finetuned hyper-parameters setting is required for improving the model's effectiveness.
Italiano. Questo articolo descrive il nostro sistema UO IRO, sviluppato per la partecipazione allo shared task IronITA, presso EVALITA 2018. Il nostro approccio si basa su un modello di deep learning con conoscenza linguistica. In particolare: una Convolutional Neural Network (CNN) e una Long Short Term Memory Neural Network (LSTM). Inoltre, il modello e` arricchito da conoscenza linguistica, incorporata nel penultimo hidden layer del modello. Sebbene sia necessario un miglioramento a grana fine dei parametri per migliorare le prestazioni del modello, i risultati ottenuti sono incoraggianti.
Computers interacting with humans through
language, in natural way, continues to be one of the
most salient challenge for Artificial Intelligent
researchers and practitioners. Nowadays, several
basic tasks related to natural language
comprehension have been effectively resolved.
Notwithstanding, slight advances have been archived by
the machines when figurative devices and
creativity are used in language with communicative
purposes. Irony is a peculiar case of figurative
devices frequently used in real life communication.
As human beings, we appeal to irony for
expressing in implicit way a meaning opposite to the
literal sense of the utterance
In this work we address the fascinating
problem of automatic irony detection in tweets
written in Italian language. Particularly, we describe
our irony detection system (UO IRO) developed
for participating in IronITA 2018: Irony Detection
in Italian Tweets
The motivation for this work comes from two
directions. In a first place, the recent and
promising results found by some authors
Our proposal learns a representation of the tweets in three ways. In this sense, we propose to learn a representation based on a recursive network with the purpose of capturing long dependencies among terms in the tweets. Moreover, a representation based on convolutional network is considered, it tries to encode local and partial relation between words which are near among themselves. The last representation is based on linguistic features which are calculated for the tweets. After that, all linguistic features previously computed are concatenated in a one-dimensional vector and it is passed through a dense hidden layer which encodes the linguistic knowledge and includes this information to the model.
Finally, the three neural network based outputs
are combined in a merge layer. The integrated
representations is passed to a dense hidden layer and
the final classification is performed by the output
layer, which use a softmax as activation function
for predicting ironic or not ironic labels. For
training the complete model we use categorical
crossentropy as loss function and the Adam method
We use a CNN model that consists in 3 pairs of convolutional layers and pooling layers in this architecture. Filters of size three, four and five were defined for the convolutional layers. In case of pooling layer, the maxpooling strategy was used. We also use the Rectified Linear Unit (ReLU), Normalization and Dropout methods to improve the accuracy and generalizability of the model. 2.4
In our work, we explored some linguistic features useful for irony detection in texts which can be grouped in three main categories: Stylistic, Structural and Content, and Polarity Contrast. We define a set of features distributed as follows:
Length: Three different features were considered: number of words, number of characters, and the means of the length of the words in the tweet.
Hashtags: The amount of hashtags.
Urls: The number of url.
Emoticons: The number of emoticons. Exclamations: Occurrences of exclamation marks.
Emphasized Words: Four different features were considered: word emphasized through repetition, capitalization, character flooding and exclamation marks.
Punctuation Marks: The frequency of dots, commas, semicolons, and question marks. Quotations: The number of expressions between quotation marks.
Antonyms: This feature considers the
number of pairs of antonyms existing in the
tweet. WordNet
Lexical Ambiguity: Three different features were considered using WordNet: the first one is the mean of the number of synsets of each word. The second one is the greatest number of synsets that has a single word. The last is the difference between the number of synsets of the word with major number of synsets and the average number of synsets.
Domain Ambiguity: Three different features were considered using WordNet: the first one is the mean of the number of domains of each word. The second one is the greatest number of domains that a single word has in the tweet. The last one is the difference between the number of domains of the word with major number of domains and the average number of domains. It is important to clarify that the resources WordNet Domains3 and SUMO4 were separately used.
Persons: This feature tries to capture verbs conjugated in the first, second, third person and nouns and adjectives which agree with such conjugations.
Tenses: This feature tries to capture the different verbal tenses used in the tweet. Questions-answers: Occurrences of questions and answers pattern in the tweet. Part of Speech: The number of nouns, verbs, adverbs and adjectives in the tweet are quantified.
Negation: The amount of negation words.
With the purpose of capturing some types of
explicit polarity contrast we consider the set of
features proposed in
WordPolarityContrast: It is the polarity difference between the most positive and the most negative word in the tweet. This feature, also consider the distance, in terms of tokens, between the words.
EmotiTextPolarityContrast: It is the polarity contrast between the emoticons and the words in the tweet.
AntecedentConsequentPolarityContrast: This considers the polarity contrast between two parts of the tweet, when it is split by a delimiter. In this case, adverbs and punctuation marks were used as delimiters. MeanPolarityPhrase: It is the mean of the polarities of the words that belong to quotes. PolarityStandardDeviation: It is the standard deviation of the polarities of the words that 3http://wndomains.fbk.eu/hierarchy.html 4http://www.adampease.org/OP/ belong to quotes.
PresentPastPolarityContrast: It computes the polarity contrast between the parts of the tweet written in present and past tense.
SkipGramPolarityRate:It computes the rate among skip-grams with polarity contrast and all valid skip-grams. The valid skip-grams are those composed by two words (nouns, adjectives, verbs, adverbs) with skip=1. The skip-grams with polarity opposition are those that match with the patterns positivenegative, positive-neutral, negative-neutral, and vise versa.
CapitalLetterTextPolarityContrast: It computes the polarity contrast between capitalized words and the rest of the words in the tweets. 3
In this section we show the results of the proposed model in the shared task of “Irony Detection” and discuss them. In a first experiment we analyze the performance of four variants of our model using 10 fold cross-validation strategy on the training set. Also, each variant was running in unconstrained and constrained setting, respectively. In Table 1, we summarize the obtained results in terms of F1 measure macro averaged (F1-AVG). Specifically, we rely on the macro for preventing systems biased towards the most populated classes.
For the run1-c and run1-u (CNN-LSTM) we
only combine the representation obtained by the
attention based LSTM model with the CNN
model, in these runs, no linguistic knowledge was
considered. Run2-c and run2-u
(CNN-LSTMSVM) are a modification of the CNN-LSTM
model, in this case we change the softmax layer
at the output of the model and use a Linear
Support Vector Machine (SVM) with default
parameters as final classifier. Run3-c and run3-u
(CNNLSTM-LING) represent the original introduced
model without any variations. Finally, for run4-c
and run4-u (CNN-LSTM-LING-SVM) we change
the softmax layer by a linear SVM as final
classifier. For unconstrained runs, we include the ironic
tweets provided by the corpus Twittiro`
Analyzing Table 1, several observations can be made. Firstly, unconstrained runs achieved better results than constrained ones. These results reveal that introducing more ironic examples improves the performance of the UO IRO. Secondly, the results achieved with the variants that consider the linguistic knowledge (run3-c, run4-c, run3-u and run4-u) obtain an increase in the effectiveness. With respect to the strategy used for the final classification of the tweets, generally, those variants that use SVM obtain a slight drop in the AVG-F1.
Regarding the official results, we submitted four runs, two for constrained setting (RUN1-c and RUN2-c) and two for unconstrained setting (RUN3-u and RUN4-u). For the unconstrained variants of the UO IRO, the tweets provided by the corpus Twittiro` were also used with the training set. Taking into account the results of the Table 1 we select to CNN-LSTM-LING (RUN1-c and RUN3-u) and CNN-LSTM-LING-SVM (RUN2-c and RUN4-u) as the most promising variants of the model for evaluating in the official test set.
As can be observed in Table 2, our four runs were ranked 12th, 13th, 14th and 15th from a total of 17 submissions. The unconstrained variants of the UO IRO achieved better results than constrained ones. Contrary to the results shown in the Table 1, the runs that use SVM as final classification strategy (RUN2-c and RUN4-u) were better ranked than the other ones. We think that this behavior may be caused by softmax classifiers (last layer of the UO IRO), those are more sensitive to the over-fitting problem than Support Vector Machines. Notice that, in all cases our model surpass the two baseline methods established by the organizers. In this paper we presented the UO IRO system for the task of Irony Detection in Italian Tweets (IronITA) at EVALITA 2018. We participated in the “Irony classification’ subtask and our best submission ranked 12nd out of 17. Our proposal combines attention-based Long Short-Term Memory Network, Convolutional Neural Network, and linguistics information which is incorporated through the second to last hidden layer of the model. The results shown that the consideration of linguistic features in combination with the deep representation learned by the neural network models obtain better effectiveness based on F1-measure. Results achieved by our system are interesting, however a more fine-tuned hyper-parameters setting is required for improving the model’s effectiveness. We think that including the linguistic features of irony into the firsts layers of the model could be a way to increase the effectiveness. We would like to explore this approach in the future work. Also, we plan to analyze how affective information flows through the tweets, and how it impacts on the irony realization.