This research aims at investigating the role that syntax can play in Sentiment Analysis tasks, in particular focusing on irony and stance detection. Benefiting from the application on our data of the standard de facto morpho-syntactic annotation format Universal Dependencies, we observe the syntactic structures occurring in these parsed social media texts in order to extract and model novel syntactic features to be used in the implementation of Sentiment Analysis tools. For achieving this goal we will focus on how irony is exploited in texts about politics, which contain an open stance towards a predetermined target entity and are affected by a strong polarization. Furthermore, thanks to the multilingual versatility of the UD annotation format, the approach will be tested on two different case-studies, i.e. English and Spanish.
In the last decade, the interest towards social networking sites has grown considerably, and the NLP community is relying more and more on data extracted from social media and micro-blogs. In particular, thanks to the the various APIs it provides and the great availability in the platform of expressions of people’s sentiments and opinions, Twitter has become one of the most exploited sources for data retrieval, especially for Sentiment Analysis and Opinion Mining. Contents generated by users in Twitter, as some other forms of computer-mediated communication (cmc), are not revised by the authors for grammatical correctness before publication and therefore theymight contain nonstandard forms such as: misspelled words, erroneous punctuation, dialectical forms, emojis, emoticons, and elongated words. Although humans understand each other when communicate with this kind of language, analyzing it automatically proves to be a very difficult challenge, especially for what concerns parsing and Part of Speech tagging.
In particular here we aim at studying the dependency syntax relations (deprels), represented through the standard format Universal Dependencies1, and applied in two different case studies: irony detection and stance detection, within a multilingual framework, taking into account the English and Spanish language.
On the one hand, for what concerns irony
detection, the aim of the task consists in
investigating whether a short message (a tweet
or a blog post) is ironic or not, with respect to
a given context. The issue is therefore
tackled as a binary classification task. On the
other hand, stance detection consists in
automatically determining whether the author
of a text is in favour, against or neutral
towards a given target (i.e. a statement, an
event, a person or an organization)
The motivation of our research arises from
the exponential growth of the interest
towards analyses about a specific product or
political actor, and the importance of
creating automatic systems able to determine the
attitude of a user within political elections or
towards the product of a certain company. It
has been studied how people (and therefore
social media users) resort to irony in order
to express also their negative opinions in a
less aggressive way
The recognition of irony and the
identification of pragmatic and linguistic devices
triggering it are known as very
challenging tasks to be performed by both humans
or automatic tools
Some of the first attempts on the
automatic classification of irony in social media
data have been made by
In the last few years, some shared task
centered on irony detection have been
proposed. In 2018 a shared task on irony
detection in tweets has been organized
(SemEval2018 Task 3: Irony detection in English
tweets)2
On the other hand, also detecting the stance
expressed by people towards specific
targets is a field of NLP research that is
currently collecting a growing interest. Only
recently stance detection has been
acknowledged as an independent task having its
own characteristics, peculiarities and
importance. The first shared task on stance
detection was indeed held for English at
SemEval2016 - Task6
The main research hypothesis we want to investigate is that the inclusion of ad-hoc tailored syntactic features among those used by a sentiment analysis system may enhance its performance in processing texts. The first goal to be accomplished is therefore to observe the syntactic structures occurring in parsed texts from social media (namely Twitter and Reddit), to design some features encoding them and finally to investigate the impact of these features on two case studies: irony and stance detection.
Furthermore we would like to investigate
these issues also in different languages, such
as English, Spanish (and later Italian and
French). Consequently, as main reference for
syntax a format has been selected that has
been drawn having the wider perspective on
the variety of human languages in mind: the
Universal Dependencies
Our approach has been inspired by some
previous work which provided support to our
hypothesis. We can cite in particular the study
of
We extended these ideas by referring to a particular syntactic framework, namely UD, and testing in some specific application scenarios the novel syntactic features that can be extracted from a UD parsed text. In both cases described below (see Sec. 4.1 and Sec. 4.2) during the pre-processing we applied indeed to the texts the morphological and syntactic analysis according to the UD framework. In Figure 1 an example of a Spanish tweet in UD format is provided5.
In particular, we trained UDPipe6, which includes tokenization, Part of Speech tagging and parsing, respectively on the UD SpanishGSD corpus7 for Spanish, and on the en core web sm8 for English. This allowed us the extraction of the novel syntactic features on which our approach is centered.
We started by testing our hypothesis in the case of stance and participating in the SemEval2019 - Task 7 on Rumor Detection (RumorEval 20199), which is focused on the identification of stance towards a set of given rumours in English texts from Twitter (see Sec. 4.1). As far as irony is concerned, we observed the relationships between UD syntactic structures and figurative language phenomena and we encoded our observations in the features we used in the system we proposed in the Irony Detection in Spanish Variants task: IroSvA 201910 (see Sec. 4.2).
In the following, we describe in more detail our participation in these shared tasks. 4.1
The RumorEval task was articulated in the following sub-tasks: Task A (open stance classification – SDQC) which was a multiclass classification for determining whether a message is a “support”, a “deny”, a “query” or a “comment” with regard to the original post; Task B (verification) which was a binary classification for predicting the veracity of a given rumour into “true” or “false”.
We will focus here only on commenting the approach and results regarding Task A, that more related to stance detection.
In our system
5Translation: I have launched a reporter into the air and they do not fly...
6https://pypi.org/project/ufal.udpipe/ 7https://github.com/UniversalDependencies/ UD_Spanish-GSD 8https://spacy.io/models/en 9https://competitions.codalab.org/ competitions/19938 10http://www.autoritas.net/IroSvA2019// aux root obj det nmod case conj det punct cc advmod punct AUX VERB DET NOUN ADP DET NOUN CCONJ ADV VERB PUNCT SYM He lanzado un relator a el aire y no vuelan ... tactic “dependency relations” (deprels) allows us to extract from data: (a) the ratio of negation dependencies compared to all the other relations; (b) the Bag of Relations (BoR all) considering all the deprels attached to all the tokens; (c) the Bag of Relations (BoR list) considering all the deprels attached to the tokens belonging to a selected list of words11; and finally (d) Bag of Relations (BoR verbs) considering all the deprels attached to all the verbs, thus fully exploiting morpho-syntactic knowledge.
We tested different machine learning classifiers performing 10-fold cross-validation.
The results showed that Logistic Regression (LR) produces the highest scores. In general, the syntactic features we proposed here are really straightforward and simple, nonetheless they helped us to improve the overall performance of the system.
Task A macro-f1
48.95
In Table 1 we present the final scores achieved accordingly to the task organizers. 4.2
The IroSvA 2019 shared task at IberLEF
2019 was focused on Irony Detection in
Spanish Variants, to be addressed as a classical
binary classification task. It allowed us to test
the importance of morpho-syntactic
information in the task of irony detection too. For
this purpose, we exploited a straightforward
methodology: we combined a Support Vector
Classifier with a linear kernel (SVM), with
11We manually created a list of words
semantically significant and useful for discriminating the four
sub-classes. For more details refer to
In addition to the features already
proposed in
LDSE W2V Word nGrams MAJORITY Our system f-avg 0.6579 0.6376 0.6192 0.4000 0.6302 quite promising, and that the dependencybased features deserves a finer-grained study, in future work we will further investigate them observing their portability in other languages different than Spanish. 5
Our research aims at investigating the role that syntax can play in sentiment analysis tasks, in particular focusing on irony and stance detection. Up to now we have tailored some shallow morpho-syntactic features based on dependency syntax as they are represented according to the UD format.
In particular, we have explored until now the role of dependency relations within two contexts and across two languages: stance detection for English and irony detection for Spanish. We are currently working on the development of four different datasets extracted from Twitter for four different langauges (English, Spanish, French and Italian) and annotated for both stance and irony. The datasets thus built will help us in testing our hypothesis in a real multilingual setting. Considering the relevance of the syntactic annotation for the suitability of our approach, we are also working in the development and improvement of resources annotated in UD format.