English. This paper describes the UNIBA team participation in the IronITA 2018 task at EVALITA 2018. We propose a supervised approach based on LIBLINEAR that relies on keyword, polarity, microblogging features and representation of tweets in a distributional semantic model. Our system ranked 3rd and 4th in the irony detection subtask. We participated only in the constraint run exploiting the training data provided by the task organizers.
The irony is defined as “the use of words that say the opposite of what you really mean, often as a joke and with a tone of voice that shows this”1. This suggests us that when we are analyzing written text for detecting irony, we should focus our attention on those words that are used in an unconventional context. For example, given the tweet: “S&P ha declassato Mario Monti da Premier a Badante #declassaggi”2, we can observe 1Oxford Learner Dictionary 2In English: “S&P has downgraded Mario Monti from Premier to Caregiver” that the word “badante” (caregiver) is used in an unconventional context, since “caregiver” usually does not co-occur with words “Premier” or “Mario Monti”.
Following this idea in our work we introduce a
feature able to detect words used out of their usual
context. Moreover, we integrate further features
based on keywords, bigrams, trigrams, polarity
and micro-blogging features as reported in
We evaluate our approach in the context of the
IronITA task at EVALITA 2018
The paper is structured as follows: Section 2 describes our system, while evaluation and results are reported in Section 3. Final remarks are provided in Section 4. 2
Our approach adopts a supervised classifier based
on LIBLINEAR
4i.e., sequences of “ah”.
5Since MultiWordNet is based on WordNet 1.6.
many switches from POS to NEG, or vice
versa, occur in the tweet.
distributional semantics features : we compute
two kinds of distributional semantics
features:
1. given a set of unlabelled downloaded
tweets, we build a geometric space in
which each word is represented as a
mathematical point. The similarity
between words is computed as their
closeness in the space. To represent a tweet
in the geometric space, we adopt the
superposition operator
Distributional Semantics Features
The distributional semantics model is built on a
collection of tweets. We randomly extract 40M
tweets from TWITA and build a semantic space
based on the Random Indexing (RI)
The mathematical insight behind the RI is the
projection of a high-dimensional space on a lower
dimensional one using a random matrix; this kind
of projection does not compromise distance
metrics
Formally, given a n m matrix A and an m k matrix R, which contains random vectors, we define a new n k matrix B as:
An;m Rm;k = Bn;k k << m (1) 6We call this set of words: the vocabulary.
The new matrix B has the property to preserve
the distance between points, that is if the distance
between two any points in A is d; then the distance
dr between the corresponding points in B will
satisfy the property that dr c d. A proof of that
is reported in the Johnson-Lindenstrauss lemma
Specifically, RI creates the WordSpace in two steps: 1. A context vector is assigned to each word.
This vector is sparse, high-dimensional and ternary, which means that its elements can take values in f-1, 0, 1g. A context vector contains a small number of randomly distributed non-zero elements, and the structure of this vector follows the hypothesis behind the concept of Random Projection; 2. Context vectors are accumulated by analyzing co-occurring words. In particular, the semantic vector for any word is computed as the sum of the context vectors for words that co-occur with the analyzed word.
Formally, given a corpus C of n documents, and a vocabulary V of m words extracted from C, we perform two steps: 1) assign a context vector ci to each word in V ; 2) compute a semantic vector svi for each word wi as the sum of all context vectors assigned to words co-occurring with wi. The context is the set of m words that precede and follow wi.
For example, considering the following tweet: “siete il buono della scuola fatelo capire”. In the first step we assign a random vector to each term as follows: csiete = ( 1; 0; 0; 1; 0; 0; 0; 0; 0; 0) cbuono = (0; 0; 0; 1; 0; 0; 0; 1; 0; 0) cscuola = (0; 0; 0; 0; 1; 0; 0; 0; 1; 0) cfatelo = (0; 1; 0; 0; 0; 1; 0; 0; 0; 0) ccapire = ( 1; 0; 0; 0; 0; 0; 0; 0; 0; 1)
In the second step, we build a semantic vector for each term by accumulating random vectors of its co-occurring words. For example fixing m = 2, the semantic vector for the word scuola is the sum of the random vectors siete, buono, fatelo, capire. Summing these vectors, the semantic vector for scuola results in ( 1; 1; 0; 2; 0; 1; 0; 1; 0; 1). This operation is repeated for all the sentences in the corpus and for all the words in V . In this example, we used very small vectors, but in a real scenario, the vector dimension ranges from hundreds to thousands of dimensions. In particular, in our experiment we use a vector dimension equals to 200 with 10 no-zero elements.
In order to compute the aci score for a word wi in a tweet, we build a context vector cwi as the sum of random vectors assigned to words that co-occur with wi in the tweet. Then we compare the cosine similarity between cwi and the semantic vector svi assigned to wi. The idea is to measure how the semantic vector is dissimilar to the context vector. If the word wi has never appeared in the context under analysis, its semantic vector does not contain the random vectors of the words in the context, this results in low cosine similarity. Finally, the divergence from the context is computed as 1 cosSim(cwi ; svi). 3
We perform the evaluation using the data provided by the task organizers. The number of tweets in the training set is 3,977, while the testing set consists of 872 tweets. The only parameter to set in LIBLINEAR is C (the cost), after a 5-fold cross validation on training we set C=1.
We submit two runs: UNIBA1 includes the semantic vector representing the tweet as a feature, while UNIBA2 does not include this vector. Nevertheless, features about the divergence are included in both the runs.
Official results are reported in Table 1. Our runs rank third and fourth in the final rank. Our team is classified as second since the first two runs in the rank belong to the team1. We can notice that runs are very close in the rank. The last run is ranked below the baseline random, while any system is ranked below the baseline baseline-mfc that assigns the most frequent class (non-ironic).
Results show that our system is not able to improve performance exploiting the distributional representation of tweets, since the two runs report the same average F1-score. We performed further experiments in order to understand the contribution of each feature. Some relevant outcomes are reported in Table 2, in particular: keyword-based features are able to achieve the best performance, in particular bigrams team team1 team1 UNIBA1 UNIBA2 team3 team6 random team7 baseline-mfc and trigrams contribute to improve the performance (run 1 and 2); different kernels for distributional and keywordbased features.
DSM features introduce some kind of noise when are combined with other features, in fact run 4, 5 and 6 achieve good performance without DSM; DSM alone without any other kind of features is able to achieve remarkable results, it is important to notice that in this run only the tweet vector is used as a feature; blog, polarity, and context features are not able to give a contribution to the overall system performance, however we can observe that using only context features (only three features for each tweet) we are able to overcome both the baselines.
Analyzing results we can conclude that a more effective way to combine distributional with nodistributional features is needed. We plan to investigate as a future work the combination of two 4
We propose a supervised system for detecting irony in Italian tweets. The proposed system exploits different kinds of features: keyword-based, microblogging features, polarity, distributional semantics features and a score that measure how a word is used in an unconventional context. The word divergence from its conventional context is computed exploiting the distributional semantics model build by the Random Indexing.
Results prove that our system is able to achieve good performance and rank third in the official ranking. However, a deep study on different combinations of features shows that keyword-based features alone are able to achieve the best result, while distributional features introduce noise during the training. This outcome suggests the need for a different strategy for combining distributional a no-distributional features.