In this article we describe the first steps of the annotation process of specific irony activators in TWITTIR O`-UD, a treebank of Italian tweets annotated with fine-grained labels for irony on one hand, and according to the Universal Dependencies scheme on the other. We discuss in particular the annotation scheme adopted to identify irony activators and some of the issues emerged during the first annotation phase. This helped us in the design of the guidelines and allowed us to draw future research directions.
In the last decade, several efforts have been devoted to address the challenges of sentiment analysis and related tasks, working mainly in English and other languages such as Italian, Spanish or French. Provided that most of the existing approaches in NLP are based on supervised semantic shallow analysis and machine learning techniques, there has been a strong push towards the development of resources from where related knowledge can be learned.
In particular the detection of irony is among the tasks currently considered as especially challenging since its presence in a text can reverse the polarity of the opinion expressed, that is using positive words for intending a negative meaning or – less often – the other way around. This can significantly undermine systems’ accuracy and makes it crucial to develop irony-aware systems (Bosco et al., 2013; Reyes et al., 2013; Riloff et al., 2013; Wang, 2013; Barbieri et al., 2014; Joshi et al., 2015; Herna´ndez Far´ıas et al.,
Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
2015; Hernan´dez Far´ıas et al., 2016).
Additionally, the challenge is further complicated when
there is a co-occurrence with sarcasm or satire
The growing interest in irony detection is also
attested by the proposal of shared tasks focusing
on this topic within NLP evaluation campaigns.
For instance, the pilot task on irony detection
proposed for Italian in SENTIPOLC at EVALITA1,
in 2014 and 2016
While the majority of the participating
systems in the above-mentioned shared-tasks are
based on classical machine learning techniques
These settings are a clear indication of the
growing interest for a deeper analysis of the linguistic
phenomena underlying ironic expressions. Such
kind of analysis naturally calls for the exploitation
of finer-grained features and resources in order to
improve the performance of automatic systems.
For instance, an especially fine-grained annotation
format for irony is the one proposed
The present work is, indeed, part of a wider joint
project with other research groups working on
English and French
This new corpus collocates in the panorama of treebanks with data extracted from social media, such as those recently developed for Italian and released in the UD repository3, and to the best of our knowledge it is one of the few linguistic resources where sentiment analysis and syntactic annotation are applied within the same framework. The main research question that we want to address is: RQ 1. Is there any syntactic pattern that can help us to automatically detect irony? The intuition that we follow in this work is that if such “syntactic patterns” which activate irony do actually exist, therefore, they should be particularly evident in the syntactic context of certain lexical elements that create a semantic clash in a text.
For this reason, in the present article, we describe the first steps of the annotation process
UniversalDependencies/UD_ Italian-PoSTWITA. of specific irony activators in the TWITTIR O`-UD corpus, taking advantage of the fact that the annotation format we adopted for the syntactic annotation allows us also to label specific activators at token level and retrieve dependency relations connected to them. In doing so, we are led to the following research questions, anticipated by the title of the paper: RQ2. Is there an effective way to annotate irony activators? RQ3. If so, is the one we propose valid? The paper is organized as follows. In Section 2 the novel dataset TWITTIR O`-UD and its annotation layers are presented. In Section 3 we describe the annotation process concerning irony activators, and we comment the inter-annotator agreement showing some examples. Finally, in Section 4 and Section 5 we discuss some difficult cases and we conclude the paper. 2
The current version of TWITTIR O`-UD comprises 1,424 tweets, annotated at multiple levels: a pragmatic level that attempts to model irony (see Section 2.1) and a syntactic level based on the UD scheme that represents the underlying syntactic structure of the tweets in the corpus (Section 2.2). In addition, we have recently introuced a further level that tries to act as an interface between the previous two (Section 3). 2.1
As far as the annotation for irony is concerned, the
data of this corpus were manually annotated
according to a multi-layered annotation scheme
described
Irony is often activated by the presence of a clash or a contradiction between two elements (also called P1 and P2). This motivates the annotation of the two different activation types at level 2: explicit when both these elements are lexicalized in the message, implicit otherwise.
4See annotation guidelines at https://github. com/IronyAndTweets/Scheme. The main linguistic devices reported in literature as irony triggers are described instead at level 3 by the categories of the scheme (i.e. analogy, euphemism, false assertion, oxymoron/paradox, context shift, hyperbole, rhetorical question and other). Table 1 shows the distribution of ironic categories throughout the corpus.
ANALOGY EUPHEMISM EX:CONTEXT SHIFT EX:OXYMORON PARADOX HYPERBOLE IM:FALSE ASSERTION OTHER RHETORICAL QUESTION TOTAL Finally the clues of level 4 are lexical or morphosyntactic signals of the activation types and categories that can be found in a given ironic tweet, such as the preposition “like” or the presence of comparative structures in the analogy type, or the adverb “very” for hyperbole. For more details about this annotation scheme, see Karoui et al. (2017). 2.2
The availability of social media data annotated
also at syntactic level is a prerequisite for our study
and for the kind of annotation we intend to
perform; as a dependency-based representation was
deemed to be more suitable for our purposes,
Universal Dependencies became our natural choice.
To obtain the data thus annotated, we ran
UDPipe
Figure 1 shows an example from the TWITTIR O`-UD corpus6 in CoNLL-U format: along with the typical fields indicating the sentence id and the raw text, two resource-specific fields have been introduced, to encode the information on irony categories (described in Section 2.1) and irony activators (see Section 3).
As also described in Cignarella et al. (2019), and as expected, the main critical issues in applying the UD scheme to our corpus namely consisted in finding the proper tags and coding conventions for those linguistic phenomena typically occurring in Italian tweets. The guidelines provided in Sanguinetti et al. (2018) represented a helpful ground5More details in Cignarella et al. (2019).
6The id of the tweet and the user mention are encrypted due to privacy regulations. – Translation: The Democratic Party is split in two. It has never been so united. [@user].
root det
nsubj Il
Pd diviso
T1 punct case obl in parataxis advmod cop advmod aux
vocative:mention advmod punct punct punct due .
Non ` e mai stato cos`ı .
] unito T2 work in this respect.
The fully-annotated treebank, including the annotation of irony categories, is going to be made available with the release of UD version 2.5. Due to its preliminary nature, however, the annotation of irony activators will be included in the resource at a later stage. 3
As previously mentioned, irony is activated by the presence of a clash or a contradiction between two elements or two propositions (P1 and P2), which are indeed the triggers of the activation of irony. According to the scheme proposed by Karoui et al. (2017) there are two kinds of activation types: EXPLICIT when both these elements are lexicalized in the message, IMPLICIT otherwise.
In this step of our work, we focused our
attention on the manual annotation of irony activators
and on providing annotation guidelines that could
be useful also for other datasets in different
languages, within the same multilingual project.
Indeed, the starting point of the present work is
connected to the work of Karoui (2017), on a French
dataset, in which the author tried to annotate at
tweet level some elements that are responsible for
the activation of irony. In that approach, each
tweet had to be annotated using the Glozz tool
With respect to this work we opted for a finergrained annotation also taking advantage from the availability of tokenized data and a full syntactic analysis in UD format. 3.1
Our aim is to annotate irony activators in the whole
TWITTIR O`-UD corpus. Differently from what
proposed
Given an ironical utterance (in our case a tweet) and its dependency-based syntactic representation, where each node in the tree structure represents a word, T1 and T2 is thus a pair of words – regardless of their grammatical category – such that: either they are both lexicalized (in explicit irony) or one of them is left unspecified (implicit irony); they act as triggers by signaling the presence of an ironic device.
The intuition behind this choice is inspired by the
work of Saif et al. (2016), in which the authors
underline the importance of contextual and
conceptual semantics of words when calculating their
sentiment, which in turn comes from the popular
dictum “You shall know a word by the company it
keeps!”
Therefore, as we have already highlighted in Section 1.1, if any kind of “syntactic pattern” that can help us to automatically detect irony does exist, we assume this will be particularly evident in the “syntactic circle” around the lexical elements that create a contradiction and are the lexical activators of the ironic realization, namely T1 and T2.
In the present research, being a preliminary study, and in order to validate the strengths and weaknesses of annotation guidelines for irony activators, two skilled annotators (A1 and A2) annotated a first sample of 277 tweets, focusing on the most frequent category: EX:OXYMORON PARADOX, which covers almost 20% of the whole corpus, as it is shown in Table 1 in Section 2.1. In the following sections we will describe the guidelines that emerged throughout the discussion between A1 and A2, we will discuss the most relevant comments reported by the annotators and we will comment on some examples, thus providing an evaluation and the measures of inter-annotator agreement. 3.2
A sample of 277 tweets, from the ironic category EX:OXYMORON PARADOX, was annotated in parallel by two skilled annotators (A1 and A2), experts both in sentiment analysis annotations and also familiar with the CoNLL-U format.
Both of them were asked, given a tweet, to annotate two words T1 and T2 that are responsible for the activation of irony, bearing in mind these basic guiding principles:
T1 and T2 can be nodes of any type: no specific constraints are given on the morphosyntactic category; the identification of the proper T1 and T2 is guided by the irony category: for example, if the ironic tweet fits the category oxymoron/paradox, select the activators so that the type of relation triggered will be a contrast or a contradiction:
la cosa bella del governo Monti e` che ha accesoT 1 le speranze di tutti ... ... e le spegnera´ T 2 pure ... ! the good thing about the Monti government is that it has kindled everyone’s hopes ... ... and it will stifle them as well Figure 2 provides an example of annotated tweet, where the words diviso (divided) and unito (united) have been annotated as T1 and T2, respectively. From a procedural perspective, since the tokens “diviso” and “unito” are respectively at position 3 and 12 in the CoNLL-U format (cfr. Figure 1), annotators were asked to add a line in the header of the annotation file, such as this one: # activators = 3 12 Furthermore, the annotators were asked to annotate any kind of doubt it might occur to them in order to provide material to a discussion about the efficacy of the guidelines. 3.3
In a first phase, the annotators sketched a draft of the guidelines for the annotation of ironic activators T1 and T2, and, as a pilot experiment, they tested their efficacy on a sample of 50 tweets. Discussing the uncertain cases and the instances in disagreement helped to significantly improve the quality of the annotation choices between A1 and A2. In fact, after the first “training phase”, the guidelines were cleared up, and the annotators could proceed to annotate all the 277 OXYMORON PARADOX tweets. The inter-annotator agreement (IAA) on the 277 tweets was later calculated by means of simple observed agreement (expressed in percentage).
As we can see from Figure 3 a complete agreement was immediately reached on 113 tweets (40.9%), other 94 tweets (34.1%) were in partial agreement (meaning that the annotators agreed only on T1 or T2), while 69 (25%) presented a complete disagreement.
After the first annotation step was completed and the agreement was calculated, the annotators tried to solve the partial disagreement. As a result, the percentage of T1-T2 pairs where agreement has been reached went up to approximately 69.2% (191 tweets), while the proportion of complete disagreement rose to approximately 30.8% (85 tweets).
Overall, the outcome of the experimental annotation of irony activators is rather encouraging. Not only from a quantitative perspective (see Section 3.3), but also from a qualitative point of view. In fact, annotators pointed out several difficult cases, but in general they were able to find an agreement discussing the possibilities within the few restrictions posed by the guidelines.
Among the unresolved cases of disagreement (difficult cases) we were able to find recurring patterns, that need to be addressed adding new specific rules before continuing with the annotation on the rest of the dataset. Below we provide a short description.
in the following tweet a list of names is presented. The contrast is created with migliori (best) and all three entities, but it is difficult to only choose one.
Fantagoverno. Fabio VoloT 1, Giovanni SartoriT 1, Roberto SavianoT 1: ecco il governo dei MiglioriT 2 Mario Monti ... URL ! Fantagovernment. Fabio Volo, Giovanni Sartori, Roberto Saviano: here is the government of the best Mario Monti... URL Multiple categories There is more than one ironic category (e.g. overlap between an ANALOGY and a PARADOX). Such as in the tweet below, in which there is a clear analogy between Superman and Mario Monti; but also the paradoxical sentence “if you didn’t exist you should be invented!” referred to a country (Italy), which, of course already exists.
E vai adesso con Mario MontiT 1/SupermanT 2, crisi finita, stipendi in aumento, e riforme. Grazie StatoT 1! Se non ci fossi bisognerebbe inventarti!T 2 ! And now let’s go with Mario Monti/Superman, the crisis is over, the salaries are raising, and there are reforms. Thank you country! If you didn’t exist you should be invented! Paraprosdokian There is a peculiar kind of ironic production, known in literature as “paraprosdokian”, in which the latter part of a sentence is surprising or unexpected in a way that causes the reader or listener to reinterpret the first part. This kind of ironic production is not specifically taken into account in the annotation scheme.
I Soliti Idioti in scena a SanremoT 1. Ieri erano alla CameraT 2. [@user] #dopofestival ! The Usual Idiots on Sanremo’s stage. Yesterday there were at the Chamber of Deputies. [@user] #afterfestival Different activation type The tweet has been annotated as EXPLICIT, but the elements that create the ironic clash are to be found in the outer world (world knowledge is needed).
#labuonascuola e` avere una scuola.
! #thegoodschool is to have a school. 5
In this article we described the preliminary steps
of the annotation process of irony activators in
the TWITTIR O`-UD corpus, a novel Italian treebank
of ironic tweets. In particular, we described the
problems that emerged during the first annotation
phase, the strengths and weaknesses of the scheme
itself, in order to highlight future research
directions. Being a preliminary study, and having no
benchmark to compare with, the results obtained
in the observed agreement are rather promising;
moreover, the tweets included in TWITTIR O` were
retrieved from different pre-existing Italian
corpora (as described
Furthermore, the availability of English and French datasets annotated with the same scheme described in Section 2.1 (see Karoui et al. (2017) allows the direct applicability of the annotation of irony activators in other languages than Italian. While this can be considered a further validation step to test the overall validity and portability of the scheme, it may also provide useful insights into the linguistic mechanisms underlying verbal irony in different languages.
The actual usability of this kind of resources will be finally tested when training NLP tools for irony detection, in both mono- and multi-lingual settings.
The work of C. Bosco and M. Sanguinetti was partially funded by Progetto di Ateneo/CSP 2016 (Immigrants, Hate and Prejudice in Social Media, S1618L2BOSC01). The work of P. Rosso was partially funded by the Spanish MICINN under the research project MISMIS-FAKEnHATE on MISinformation and MIScommunication in social media: FAKE news and HATE speech (PGC2018096212-B-C31).