English. In this paper we describe and show the results of the two systems that we have specifically developed to participate at Ironita 2018 for the irony detection task. We scored as the third team in the official ranking of the competition, thanks to the X2C-B system, at a distance of just 0.027 of F1 score from the best system.
In social media, the use of irony in tweets and
Facebook posts is widely spread and makes very
difficult for sentiment analysis tools to properly
automatically classify people opinion
In this paper we describe and show the results
of the two systems that we have specifically
developed to participate at Ironita 2018
This paper is structured as follow: after the introduction we present the descriptions of our two systems submitted for the irony detection task; then we show and discuss the results on the official test set of the competition, finally we provide our conclusions. 2
The dataset provided by Ironita organizers has
been split into training set (80% of the documents)
and development set (the remaining 20%). We
randomly sampled the examples for each
category, thus obtaining different sets for training/test
set, by keeping the distribution of ironic and
nonironic samples through the two sets. We submitted
two runs, as the results of the two different
systems we developed for each category, called
X2CA and X2C-B. The former has been developed on
top of the Scikit-learn library in Python language
Both the X2C-A and X2C-B unconstrained run
were trained using the SENTIPOLC 2016 Irony
training set and test set
LinearSVM NB SGD Ensemble In the model selection process, the two best algorithms have been Naive Bayes Multinomial and SMO, both using unigram features. We took into account the F1 score on the positive labels and the Macro-F1 in order to select the best algorithm. As shown in Table 2, Naive Bayes Multinomial reached a Macro F1 score 2.38% higher on the constrained run and a 14.2% on the unconstrained run, thus both the constrained and the unconstrained submitted runs were produced using this algorithm.
Comparing the results in Table 2 with the ones in Table 1, we can notice that X2C-B unconstrained reached the highest performance on the development set, while X2C-B constrained obtained the lowest score. In Table 3 we show the results of our runs on the official test set of the competition. In accordance with what we noticed before, comparing Table 1 and Table 2, our best run is X2C-B unconstrained, which reached the best F1 overall on non-ironic documents; it also ranks fifth in the overall F1score, at a distance of 0.027 from the best system. The performance of the X2C-A run is very similar to the unconstrained run, obtaining a F1-score that is only 0.002 higher than the constrained run. The difference between the two X2C-B runs is larger in relative terms, but is only of 0.021. We can also see that our X2C-B-u shows the best F1 score on the non-ironic tweets compared to all of the systems.
We added to this ranking also the model that
reached the first position on the Irony task at
SENTIPOLC 2016
In this paper we described the two systems that we built and submitted for the Ironita 2018 competition for the irony detection task. The results show that our system X2C-B scored as the third team at a distance of just 0.027 of F1 score from the best system. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 team team 1 team 1 team 2 team 2 X2C-B-u team 4 team 4 X2C-A-u team 5 X2C-A-c X2C-B-c team 6 team 6 team 6 team 6 baseline-1 team 7 X2C2016 team 7 baseline-2 (IronITA) in Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA’18).