=Paper=
{{Paper
|id=Vol-2481/paper18
|storemode=property
|title=Is This an Effective Way to Annotate Irony Activators?
|pdfUrl=https://ceur-ws.org/Vol-2481/paper18.pdf
|volume=Vol-2481
|authors=Alessandra Teresa Cignarella,Manuela Sanguinetti,Cristina Bosco,Paolo Rosso
|dblpUrl=https://dblp.org/rec/conf/clic-it/CignarellaSBR19
}}
==Is This an Effective Way to Annotate Irony Activators?==
https://ceur-ws.org/Vol-2481/paper18.pdf
Is This an Effective Way to Annotate Irony Activators?
Alessandra Teresa Cignarella1,2 , Manuela Sanguinetti1 , Cristina Bosco1 , Paolo Rosso2
1. Dipartimento di Informatica, Università degli Studi di Torino, Italy
2. PRHLT Research Center, Universitat Politècnica de València, Spain
{cigna|msanguin|bosco}@di.unito.it, prosso@dsic.upv.es
Abstract 2015; Hernańdez Farı́as et al., 2016). Addition-
ally, the challenge is further complicated when
In this article we describe the first steps there is a co-occurrence with sarcasm or satire
of the annotation process of specific irony (Hernández Farı́as and Rosso, 2016; Joshi et al.,
activators in TWITTIR Ò - UD, a treebank of 2017; Ravi and Ravi, 2017).
Italian tweets annotated with fine-grained The growing interest in irony detection is also
labels for irony on one hand, and accord- attested by the proposal of shared tasks focusing
ing to the Universal Dependencies scheme on this topic within NLP evaluation campaigns.
on the other. We discuss in particular For instance, the pilot task on irony detection pro-
the annotation scheme adopted to iden- posed for Italian in SENTIPOLC at EVALITA1 ,
tify irony activators and some of the is- in 2014 and 2016 (Barbieri et al., 2016; Basile
sues emerged during the first annotation et al., 2014), and the related task proposed for
phase. This helped us in the design of the French at DEFT at TALN 2017 (Benamara et al.,
guidelines and allowed us to draw future 2017). For what concerns English, after a first
research directions. task at SemEval-2015 focusing on figurative lan-
guage in Twitter (Ghosh et al., 2015), a shared task
1 Introduction on irony detection in tweets has been proposed in
2018 (Van Hee et al., 2018). Concerning Spanish,
In the last decade, several efforts have been de- the most recent shared task about irony in social
voted to address the challenges of sentiment anal- media has been organized at IberLEF 2019 Irony
ysis and related tasks, working mainly in English Detection in Spanish Variants (IroSvA 2019), ex-
and other languages such as Italian, Spanish or ploring the differences among varieties of Spanish
French. Provided that most of the existing ap- from Spain, Cuba and Mexico (Ortega et al., 2019)
proaches in NLP are based on supervised semantic in which the organizers also proposed a focus on
shallow analysis and machine learning techniques, context, stressing the importance of contextual se-
there has been a strong push towards the develop- mantics in ironic productions.
ment of resources from where related knowledge While the majority of the participating sys-
can be learned. tems in the above-mentioned shared-tasks are
In particular the detection of irony is among based on classical machine learning techniques
the tasks currently considered as especially chal- (Cignarella and Bosco, 2019; Frenda and Patti,
lenging since its presence in a text can reverse 2019), researchers have recently started to exploit
the polarity of the opinion expressed, that is us- approaches based on neural networks. Among
ing positive words for intending a negative mean- these, Huang et al. (2017) applied attentive re-
ing or – less often – the other way around. current neural networks (RNNs) that capture spe-
This can significantly undermine systems’ accu- cific words which are helpful in detecting the pres-
racy and makes it crucial to develop irony-aware ence of irony in a tweet, while Wu et al. (2018)
systems (Bosco et al., 2013; Reyes et al., 2013; exploited densely connected LSTMs in a multi-
Riloff et al., 2013; Wang, 2013; Barbieri et al., task learning strategy, adding PoS tag features, and
2014; Joshi et al., 2015; Hernández Farı́as et al., Zhang et al. (2019) took advantage of recent ad-
Copyright c 2019 for this paper by its authors. Use vancements in transfer learning techniques.
permitted under Creative Commons License Attribution 4.0
1
International (CC BY 4.0). http://www.evalita.it/
These settings are a clear indication of the grow- of specific irony activators in the TWITTIR Ò - UD
ing interest for a deeper analysis of the linguistic corpus, taking advantage of the fact that the
phenomena underlying ironic expressions. Such annotation format we adopted for the syntactic an-
kind of analysis naturally calls for the exploitation notation allows us also to label specific activators
of finer-grained features and resources in order to at token level and retrieve dependency relations
improve the performance of automatic systems. connected to them. In doing so, we are led to the
For instance, an especially fine-grained annotation following research questions, anticipated by the
format for irony is the one proposed in Karoui title of the paper:
et al. (2017), concerning French, Italian and En-
glish. The same scheme has later been applied on RQ2. Is there an effective way to annotate irony
a new Italian corpus: TWITTIR Ò (Cignarella et al., activators?
2018a). The resulting annotated corpus was used RQ3. If so, is the one we propose valid?
as reference dataset in the IronITA 2018 shared
task2 on Irony and Sarcasm Detection in Italian The paper is organized as follows. In Section 2
Tweets (Cignarella et al., 2018b). the novel dataset TWITTIR Ò - UD and its annota-
tion layers are presented. In Section 3 we describe
1.1 Motivation and Research Questions the annotation process concerning irony activa-
The present work is, indeed, part of a wider joint tors, and we comment the inter-annotator agree-
project with other research groups working on En- ment showing some examples. Finally, in Section
glish and French (Karoui et al., 2015). As men- 4 and Section 5 we discuss some difficult cases
tioned above, in Cignarella et al. (2018a), we cre- and we conclude the paper.
ated an Italian corpus of tweets, i.e. TWITTIR Ò,
annotated with a fine-grained tagset for irony, 2 Corpus Description
and later on, we extended the same resource ap-
The current version of TWITTIR Ò - UD comprises
plying the Universal Dependencies (UD) scheme
1,424 tweets, annotated at multiple levels: a prag-
(Nivre et al., 2016), thus creating TWITTIR Ò - UD
matic level that attempts to model irony (see Sec-
(Cignarella et al., 2019).
tion 2.1) and a syntactic level based on the UD
This new corpus collocates in the panorama
scheme that represents the underlying syntactic
of treebanks with data extracted from social
structure of the tweets in the corpus (Section 2.2).
media, such as those recently developed for
In addition, we have recently introuced a further
Italian and released in the UD repository3 , and
level that tries to act as an interface between the
to the best of our knowledge it is one of the few
previous two (Section 3).
linguistic resources where sentiment analysis and
syntactic annotation are applied within the same 2.1 Annotating Irony
framework. The main research question that we
As far as the annotation for irony is concerned, the
want to address is:
data of this corpus were manually annotated ac-
cording to a multi-layered annotation scheme de-
RQ 1. Is there any syntactic pattern that can help
scribed in Karoui et al. (2017), which in turn in-
us to automatically detect irony?
cludes 4 different levels.4 Beyond the annotation
of irony vs non-irony (henceforth level 1), the mul-
The intuition that we follow in this work is that
tifaceted annotation scheme is organized in three
if such “syntactic patterns” which activate irony
further layers, namely the activation type (level 2),
do actually exist, therefore, they should be partic-
the categories (level 3) and the clues (level 4).
ularly evident in the syntactic context of certain
Irony is often activated by the presence of a
lexical elements that create a semantic clash in a
clash or a contradiction between two elements
text.
(also called P1 and P2). This motivates the annota-
For this reason, in the present article, we
tion of the two different activation types at level 2:
describe the first steps of the annotation process
explicit when both these elements are lexicalized
2
http://di.unito.it/ironita18. in the message, implicit otherwise.
3
https://github.com/
4
UniversalDependencies/UD_ See annotation guidelines at https://github.
Italian-PoSTWITA. com/IronyAndTweets/Scheme.
Figure 1: Example of tweet in CoNLL-U format.
The main linguistic devices reported in literature To obtain the data thus annotated, we ran UD-
as irony triggers are described instead at level 3 Pipe (Straka and Straková, 2017) for tokenization,
by the categories of the scheme (i.e. analogy, PoS tagging, lemmatization and dependency pars-
euphemism, false assertion, oxymoron/paradox, ing, using a model trained on two Italian resources
context shift, hyperbole, rhetorical question and available in the UD repository, the ISDT (Simi et
other). Table 1 shows the distribution of ironic cat- al., 2014) and PoSTWITA-UD (Sanguinetti et al.,
egories throughout the corpus. 2018) treebanks5 . The former includes multiple
text genres (legal texts, news, Wikipedia articles,
n# %
among others), but it mostly deals with well-edited
ANALOGY 261 18%
EUPHEMISM 84 6% texts and a standard language. The latter is made
EX : CONTEXT SHIFT 185 13% up of so-called user-generated contents, an in par-
EX : OXYMORON PARADOX 277 19%
HYPERBOLE 81 6%
ticular of Twitter posts in Italian. As using both
IM : FALSE ASSERTION 117 8% resources for training proved to give better results
OTHER 198 14% when analyzing Italian tweets (Sanguinetti et al.,
RHETORICAL QUESTION 221 16%
TOTAL 1,424
2018), we used the same approach in this work.
Figure 1 shows an example from the TWIT-
Table 1: Ironic categories in TWITTIR Ò - UD. TIR Ò - UD corpus6 in CoNLL-U format: along with
the typical fields indicating the sentence id and the
Finally the clues of level 4 are lexical or morpho- raw text, two resource-specific fields have been in-
syntactic signals of the activation types and cate- troduced, to encode the information on irony cate-
gories that can be found in a given ironic tweet, gories (described in Section 2.1) and irony activa-
such as the preposition “like” or the presence of tors (see Section 3).
comparative structures in the analogy type, or the As also described in Cignarella et al. (2019),
adverb “very” for hyperbole. For more details and as expected, the main critical issues in apply-
about this annotation scheme, see Karoui et al. ing the UD scheme to our corpus namely consisted
(2017). in finding the proper tags and coding conventions
for those linguistic phenomena typically occurring
2.2 Annotating Universal Dependencies
in Italian tweets. The guidelines provided in San-
The availability of social media data annotated guinetti et al. (2018) represented a helpful ground-
also at syntactic level is a prerequisite for our study
5
and for the kind of annotation we intend to per- More details in Cignarella et al. (2019).
6
form; as a dependency-based representation was The id of the tweet and the user mention are encrypted
due to privacy regulations. – Translation: The Democratic
deemed to be more suitable for our purposes, Uni- Party is split in two. It has never been so united.
versal Dependencies became our natural choice. [@user].
parataxis
advmod
cop
root
punct advmod vocative:mention
obl aux
det nsubj case advmod punct punct punct
Il Pd diviso in due . Non è mai stato cosı̀ unito . [ @user ]
T1 T2
Figure 2: Dependency graph of the tweet in Figure 1 with irony activators T1 and T2 highlighted in red
and blue, respectively.
work in this respect. 3.1 Our approach
The fully-annotated treebank, including the an- Our aim is to annotate irony activators in the whole
notation of irony categories, is going to be made TWITTIR Ò - UD corpus. Differently from what pro-
available with the release of UD version 2.5. Due posed in Karoui (2017), in which the elements
to its preliminary nature, however, the annotation creating an ironic contrast (P1 and P2) could be
of irony activators will be included in the resource words, phrases or even full sentences; in this work,
at a later stage. since we want to highlight the interaction between
the pragmatic phenomenon of irony and its syn-
3 Annotating Irony Activators tactic representation, we define as irony activators
a pair of words T1 and T2 that must correspond to
As previously mentioned, irony is activated by the nodes of the syntactic dependency tree.
presence of a clash or a contradiction between two Given an ironical utterance (in our case a tweet)
elements or two propositions (P1 and P2), which and its dependency-based syntactic representation,
are indeed the triggers of the activation of irony. where each node in the tree structure represents a
According to the scheme proposed by Karoui et word, T1 and T2 is thus a pair of words – regard-
al. (2017) there are two kinds of activation types: less of their grammatical category – such that:
EXPLICIT when both these elements are lexical-
ized in the message, IMPLICIT otherwise. • either they are both lexicalized (in explicit
In this step of our work, we focused our atten- irony) or one of them is left unspecified (im-
tion on the manual annotation of irony activators plicit irony);
and on providing annotation guidelines that could
• they act as triggers by signaling the presence
be useful also for other datasets in different lan-
of an ironic device.
guages, within the same multilingual project. In-
deed, the starting point of the present work is con- The intuition behind this choice is inspired by the
nected to the work of Karoui (2017), on a French work of Saif et al. (2016), in which the authors
dataset, in which the author tried to annotate at underline the importance of contextual and con-
tweet level some elements that are responsible for ceptual semantics of words when calculating their
the activation of irony. In that approach, each sentiment, which in turn comes from the popular
tweet had to be annotated using the Glozz tool dictum “You shall know a word by the company it
(Widlöcher and Mathet, 2009), in terms of units keeps!” (Firth, 1957). Our idea is, in fact, to pro-
and relationships between units (if the relationship ceed in two steps: firstly, to annotate irony trig-
existed). Three types of relationship were taken gers at token level, and subsequently to retrieve
into account: 1) relation of comparison, 2) rela- the other tokens that “keep company” to them by
tion of explicit contradiction, and 3) relation of means of the dependency relations available from
cause/consequence. the UD annotation.
With respect to this work we opted for a finer- Therefore, as we have already highlighted in
grained annotation also taking advantage from the Section 1.1, if any kind of “syntactic pattern” that
availability of tokenized data and a full syntactic can help us to automatically detect irony does ex-
analysis in UD format. ist, we assume this will be particularly evident in
the “syntactic circle” around the lexical elements tokens “diviso” and “unito” are respectively at po-
that create a contradiction and are the lexical acti- sition 3 and 12 in the CoNLL-U format (cfr. Fig-
vators of the ironic realization, namely T1 and T2. ure 1), annotators were asked to add a line in the
In the present research, being a preliminary header of the annotation file, such as this one:
study, and in order to validate the strengths and
weaknesses of annotation guidelines for irony ac- # activators = 3 12
tivators, two skilled annotators (A1 and A2) anno-
Furthermore, the annotators were asked to anno-
tated a first sample of 277 tweets, focusing on the
tate any kind of doubt it might occur to them in
most frequent category: EX : OXYMORON PARA -
order to provide material to a discussion about the
DOX , which covers almost 20% of the whole cor-
efficacy of the guidelines.
pus, as it is shown in Table 1 in Section 2.1. In
the following sections we will describe the guide- 3.3 Evaluation and Agreement
lines that emerged throughout the discussion be- In a first phase, the annotators sketched a draft
tween A1 and A2, we will discuss the most rele- of the guidelines for the annotation of ironic ac-
vant comments reported by the annotators and we tivators T1 and T2, and, as a pilot experiment,
will comment on some examples, thus providing they tested their efficacy on a sample of 50 tweets.
an evaluation and the measures of inter-annotator Discussing the uncertain cases and the instances
agreement. in disagreement helped to significantly improve
the quality of the annotation choices between A1
3.2 Annotation process
and A2. In fact, after the first “training phase”,
A sample of 277 tweets, from the ironic category the guidelines were cleared up, and the annotators
EX : OXYMORON PARADOX , was annotated in par- could proceed to annotate all the 277 OXYMORON
allel by two skilled annotators (A1 and A2), ex- PARADOX tweets. The inter-annotator agreement
perts both in sentiment analysis annotations and (IAA) on the 277 tweets was later calculated by
also familiar with the CoNLL-U format. means of simple observed agreement (expressed
Both of them were asked, given a tweet, to an- in percentage).
notate two words T1 and T2 that are responsible
for the activation of irony, bearing in mind these
basic guiding principles:
• T1 and T2 can be nodes of any type: no
specific constraints are given on the morpho-
syntactic category;
• the identification of the proper T1 and T2
is guided by the irony category: for exam-
ple, if the ironic tweet fits the category oxy- Figure 3: Observed IAA on 277 tweets.
moron/paradox, select the activators so that
the type of relation triggered will be a con- As we can see from Figure 3 a complete agreement
trast or a contradiction: was immediately reached on 113 tweets (40.9%),
other 94 tweets (34.1%) were in partial agreement
la cosa bella del governo Monti è che ha
(meaning that the annotators agreed only on T1
accesoT 1 le speranze di tutti ... ... e le speg-
or T2), while 69 (25%) presented a complete dis-
neráT 2 pure ...
agreement.
→ the good thing about the Monti government
After the first annotation step was completed
is that it has kindled everyone’s hopes ... ...
and the agreement was calculated, the annotators
and it will stifle them as well
tried to solve the partial disagreement. As a re-
sult, the percentage of T1-T2 pairs where agree-
Figure 2 provides an example of annotated tweet, ment has been reached went up to approximately
where the words diviso (divided) and unito 69.2% (191 tweets), while the proportion of com-
(united) have been annotated as T1 and T2, respec- plete disagreement rose to approximately 30.8%
tively. From a procedural perspective, since the (85 tweets).
4 Discussion ically taken into account in the annotation scheme.
Overall, the outcome of the experimental annota-
tion of irony activators is rather encouraging. Not I Soliti Idioti in scena a SanremoT 1 . Ieri erano
alla CameraT 2 . [@user] #dopofestival
only from a quantitative perspective (see Section → The Usual Idiots on Sanremo’s stage. Yesterday
3.3), but also from a qualitative point of view. In there were at the Chamber of Deputies. [@user]
fact, annotators pointed out several difficult cases, #afterfestival
but in general they were able to find an agreement
discussing the possibilities within the few restric-
Different activation type The tweet has been
tions posed by the guidelines.
annotated as EXPLICIT, but the elements that cre-
Among the unresolved cases of disagreement ate the ironic clash are to be found in the outer
(difficult cases) we were able to find recurring pat- world (world knowledge is needed).
terns, that need to be addressed adding new spe-
cific rules before continuing with the annotation #labuonascuola è avere una scuola.
on the rest of the dataset. Below we provide a short → #thegoodschool is to have a school.
description.
More than two irony activators For instance, 5 Conclusion
in the following tweet a list of names is presented. In this article we described the preliminary steps
The contrast is created with migliori (best) and all of the annotation process of irony activators in
three entities, but it is difficult to only choose one. the TWITTIR Ò - UD corpus, a novel Italian treebank
Fantagoverno. Fabio VoloT 1 , Giovanni
of ironic tweets. In particular, we described the
SartoriT1
, Roberto SavianoT 1 : ecco il governo dei
problems that emerged during the first annotation
MiglioriT 2 Mario Monti ... URL
phase, the strengths and weaknesses of the scheme
→ Fantagovernment. Fabio Volo, Giovanni Sartori,
itself, in order to highlight future research direc-
Roberto Saviano: here is the government of the
tions. Being a preliminary study, and having no
best Mario Monti... URL
benchmark to compare with, the results obtained
in the observed agreement are rather promising;
moreover, the tweets included in TWITTIR Ò were
Multiple categories There is more than one retrieved from different pre-existing Italian cor-
ironic category (e.g. overlap between an ANAL - pora (as described in Cignarella et al. (2017)): the
OGY and a PARADOX ). Such as in the tweet be- heterogenous sources the data were gathered from
low, in which there is a clear analogy between Su- thus represents a signal of the potential portability
perman and Mario Monti; but also the paradoxi- of the scheme and paves the way for a more sys-
cal sentence “if you didn’t exist you should be in- tematic annotation process of the whole dataset.
vented!” referred to a country (Italy), which, of The next steps will then consist in the guidelines
course already exists. improvement and the annotation of the remaining
part of TWITTIR Ò - UD accordingly.
E vai adesso con Mario MontiT 1 /SupermanT 2 ,
Furthermore, the availability of English and
crisi finita, stipendi in aumento, e riforme. Grazie
French datasets annotated with the same scheme
StatoT 1 ! Se non ci fossi bisognerebbe inventarti!T 2
described in Section 2.1 (see Karoui et al. (2017)
→ And now let’s go with Mario Monti/Superman,
allows the direct applicability of the annotation
the crisis is over, the salaries are raising, and there
of irony activators in other languages than Italian.
are reforms. Thank you country! If you didn’t exist
While this can be considered a further validation
you should be invented!
step to test the overall validity and portability of
the scheme, it may also provide useful insights
Paraprosdokian There is a peculiar kind into the linguistic mechanisms underlying verbal
of ironic production, known in literature as irony in different languages.
“paraprosdokian”, in which the latter part of a The actual usability of this kind of resources
sentence is surprising or unexpected in a way that will be finally tested when training NLP tools for
causes the reader or listener to reinterpret the first irony detection, in both mono- and multi-lingual
part. This kind of ironic production is not specif- settings.
Acknowledgments Corpus with a Multi-layered Annotation for Irony.
In Proceedings of the Fourth Italian Conference on
The work of C. Bosco and M. Sanguinetti was Computational Linguistics (CLiC-it 2017), volume
partially funded by Progetto di Ateneo/CSP 2016 2006, pages 101–106. CEUR-WS.org.
(Immigrants, Hate and Prejudice in Social Media, Alessandra Teresa Cignarella, Cristina Bosco, Viviana
S1618L2BOSC01). The work of P. Rosso was par- Patti, and Mirko Lai. 2018a. Application and Anal-
tially funded by the Spanish MICINN under the ysis of a Multi-layered Scheme for Irony on the
research project MISMIS-FAKEnHATE on MIS- Italian Twitter Corpus TWITTIRÒ. In Proceed-
information and MIScommunication in social me- ings of the Eleventh International Conference on
Language Resources and Evaluation (LREC-2018),
dia: FAKE news and HATE speech (PGC2018- pages 4204–4211. ELRA.
096212-B-C31).
Alessandra Teresa Cignarella, Simona Frenda, Valerio
Basile, Cristina Bosco, Viviana Patti, Paolo Rosso,
References et al. 2018b. Overview of the EVALITA 2018
task on Irony Detection in Italian Tweets (IronITA).
Francesco Barbieri, Horacio Saggion, and Francesco In Sixth Evaluation Campaign of Natural Language
Ronzano. 2014. Modelling sarcasm in Twitter, a Processing and Speech Tools for Italian (EVALITA
novel approach. In Proceedings of the 5th Work- 2018), volume 2263, pages 1–6. CEUR-WS.org.
shop on Computational Approaches to Subjectivity,
Sentiment and Social Media Analysis, pages 50–58. Alessandra Teresa Cignarella, Cristina Bosco, and
ACL. Paolo Rosso. 2019. Presenting TWITTIRÒ-UD:
An Italian Twitter Treebank in Universal Dependen-
Francesco Barbieri, Valerio Basile, Danilo Croce, cies. In Proceedings of SyntaxFest 2019.
Malvina Nissim, Nicole Novielli, and Viviana Patti.
2016. Overview of the EVALITA 2016 SENTIment John R Firth. 1957. A synopsis of linguistic theory,
POLarity Classification Task. In Proceedings of 1930-1955. Studies in linguistic analysis.
Third Italian Conference on Computational Linguis-
tics (CLiC-it 2016) & Fifth Evaluation Campaign Simona Frenda and Viviana Patti. 2019. Computa-
of Natural Language Processing and Speech Tools tional models for irony detection in three spanish
for Italian (EVALITA 2016), volume 1749. CEUR- variants. In Proceedings of the Iberian Languages
WS.org. Evaluation Forum (IberLEF 2019), co-located with
34th Conference of the Spanish Society for Natu-
Valerio Basile, Andrea Bolioli, Malvina Nissim, Vi- ral Language Processing (SEPLN 2019). CEUR-
viana Patti, and Paolo Rosso. 2014. Overview WS.org.
of the EVALITA 2014 SENTIment POLarity Clas-
sification task. In Proceedings of the 4th Evalua- Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso,
tion Campaign of Natural Language Processing and Ekaterina Shutova, John Barnden, and Antonio
Speech tools for Italian (EVALITA 2014). ELRA. Reyes. 2015. Semeval-2015 Task 11: Sentiment
Analysis of Figurative Language in Twitter. In Pro-
Farah Benamara, Cyril Grouin, Jihen Karoui, ceedings of the 9th International Workshop on Se-
Véronique Moriceau, and Isabelle Robba. 2017. mantic Evaluation (SemEval-2015), pages 470–478.
Analyse d’opinion et langage figuratif dans des ACL.
tweets: présentation et résultats du Défi Fouille de
Textes DEFT2017. In Actes de l’atelier DEFT2017 Delia Irazú Hernández Farı́as, José-Miguel Benedı́, and
associé à la conférence TALN. Association pour le Paolo Rosso. 2015. Applying basic features from
Traitement Automatique des Langues (ATALA). sentiment analysis for automatic irony detection. In
Pattern Recognition and Image Analysis, volume
Cristina Bosco, Viviana Patti, and Andrea Bolioli. 9117, pages 337–344. Springer.
2013. Developing Corpora for Sentiment Analysis:
The Case of Irony and Senti-TUT. IEEE Intelligent Delia Irazú Hernańdez Farı́as, Viviana Patti, and Paolo
Systems, 28(2):55–63. Rosso. 2016. Irony detection in Twitter: The role
of affective content. ACM Transactions on Internet
Alessandra Teresa Cignarella and Cristina Bosco. Technology (TOIT), 16(3):19.
2019. ATC at IroSvA 2019: Shallow Syntactic
Dependency-based Features for Irony Detection in Delia Irazú Hernández Farı́as and Paolo Rosso. 2016.
Spanish Variants. In Proceedings of the Iberian Irony, Sarcasm, and Sentiment Analysis. In Senti-
Languages Evaluation Forum (IberLEF 2019), co- ment Analysis in Social Networks, pages 113–128.
located with 34th Conference of the Spanish Soci- Elsevier Science and Technology.
ety for Natural Language Processing (SEPLN 2019).
CEUR-WS.org. Yu-Hsiang Huang, Hen-Hsen Huang, and Hsin-Hsi
Chen. 2017. Irony detection with attentive Recur-
Alessandra Teresa Cignarella, Cristina Bosco, and Vi- rent Neural Networks. In European Conference on
viana Patti. 2017. TWITTIRÒ: a Social Media Information Retrieval, pages 534–540. Springer.
Aditya Joshi, Vinita Sharma, and Pushpak Bhat- Language Processing, (EMNLP 2013), pages 704–
tacharyya. 2015. Harnessing context incongruity 714. ACL.
for sarcasm detection. In Proceedings of the 53rd
Annual Meeting of the Association for Computa- Hassan Saif, Yulan He, Miriam Fernandez, and Harith
tional Linguistics and the 7th International Joint Alani. 2016. Contextual Semantics for Senti-
Conference on Natural Language Processing (Vol- ment Analysis of Twitter. Information Processing
ume 2: Short Papers), pages 757–762. ACL, July. & Management, 52(1):5–19.
Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Car- Manuela Sanguinetti, Cristina Bosco, Alberto Lavelli,
man. 2017. Automatic sarcasm detection: A survey. Alessandro Mazzei, Oronzo Antonelli, and Fabio
ACM Computing Surveys (CSUR), 50(5):73. Tamburini. 2018. PoSTWITA-UD: an Italian Twit-
ter Treebank in Universal Dependencies. In Pro-
Jihen Karoui, Farah Benamara, Véronique Moriceau, ceedings of the 11th International Conference on
Nathalie Aussenac-Gilles, and Lamia Hadrich Bel- Language Resources and Evaluation (LREC-2018).
guith. 2015. Towards a contextual pragmatic model
to detect irony in tweets. In 53rd Annual Meeting of Maria Simi, Cristina Bosco, and Simonetta Monte-
the Association for Computational Linguistics (ACL magni. 2014. Less is more? Towards a reduced
2015). inventory of categories for training a parser for the
Italian Stanford Dependencies. In Language Re-
Jihen Karoui, Farah Benamara, Véronique Moriceau, sources and Evaluation 2014, pages 83–90. Euro-
Viviana Patti, and Cristina Bosco. 2017. Exploring pean Language Resources Association (ELRA).
the Impact of Pragmatic Phenomena on Irony De-
tection in Tweets: A Multilingual Corpus Study. In Milan Straka and Jana Straková. 2017. Tokenizing,
Proceedings of the European Chapter of the Associ- POS Tagging, Lemmatizing and Parsing UD 2.0
ation for Computational Linguistics. with UDPipe. In Proceedings of the CoNLL 2017
Shared Task: Multilingual Parsing from Raw Text to
Jihen Karoui. 2017. Détection automatique de l’ironie Universal Dependencies, pages 88–99. ACL.
dans les contenus générés par les utilisateurs. Ph.D.
thesis, Université Toulouse 3 Paul Sabatier; Faculté Cynthia Van Hee, Els Lefever, and Véronique Hoste.
des Sciences Economiques et de Gestion, Université 2018. SemEval-2018 Task 3: Irony detection in En-
de Sfax (Tunisie). glish Tweets. In In Proceedings of the International
Workshop on Semantic Evaluation (SemEval-2018).
Joakim Nivre, Marie-Catherine De Marneffe, Filip ACL.
Ginter, Yoav Goldberg, Jan Hajic, Christopher D.
Angela P. Wang. 2013. #Irony or #Sarcasm — A
Manning, Ryan T. McDonald, Slav Petrov, Sampo
Quantitative and Qualitative Study Based on Twitter.
Pyysalo, Natalia Silveira, et al. 2016. Universal
In Proceedings of the PACLIC: the 27th Pacific Asia
Dependencies v1: A Multilingual Treebank Col-
Conference on Language, Information, and Compu-
lection. In Proceedings of the 10th International
tation, pages 349–356. ACL.
Conference on Language Resources and Evaluation
(LREC-2016). Antoine Widlöcher and Yann Mathet. 2009. La
plate-forme glozz: environnement d’annotation et
Reynier Ortega, Francisco Rangel, Irazú Hernández,
d’exploration de corpus. In Actes de TALN 2009.
Paolo Rosso, Manuel Montes, and José E. Medina.
2019. Overview of the Task on Irony Detection Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu,
in Spanish Variants. In Proceedings of the Iberian Zhigang Yuan, and Yongfeng Huang. 2018.
Languages Evaluation Forum (IberLEF 2019), co- Thu ngn at semeval-2018 task 3: Tweet irony detec-
located with 34th Conference of the Spanish Soci- tion with densely connected LSTM and multi-task
ety for Natural Language Processing (SEPLN 2019). learning. In Proceedings of The 12th International
CEUR-WS.org. Workshop on Semantic Evaluation, pages 51–56.
Kumar Ravi and Vadlamani Ravi. 2017. A novel Shiwei Zhang, Xiuzhen Zhang, Jeffrey Chan, and
automatic satire and irony detection using ensem- Paolo Rosso. 2019. Irony detection via sentiment-
bled feature selection and data mining. Knowledge- based transfer learning. Information Processing &
Based Systems, 120:15–33. Management, 56(5):1633–1644.
Antonio Reyes, Paolo Rosso, and Tony Veale. 2013.
A Multidimensional Approach for Detecting Irony
in Twitter. Language Resources and Evaluation,
47(1):239–268.
Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-
dra De Silva, Nathan Gilbert, and Ruihong Huang.
2013. Sarcasm as Contrast between a Positive Sen-
timent and Negative Situation. In Proceedings of the
2013 Conference on Empirical Methods in Natural