=Paper=
{{Paper
|id=Vol-2765/150
|storemode=property
|title=SSN NLP @ SardiStance : Stance Detection from Italian Tweets using RNN and Transformers (short paper)
|pdfUrl=https://ceur-ws.org/Vol-2765/paper150.pdf
|volume=Vol-2765
|authors=Kayalvizhi S,Thenmozhi D,Aravindan Chandrabose
|dblpUrl=https://dblp.org/rec/conf/evalita/SDC20
}}
==SSN NLP @ SardiStance : Stance Detection from Italian Tweets using RNN and Transformers (short paper)==
https://ceur-ws.org/Vol-2765/paper150.pdf
SSN NLP@SardiStance : Stance Detection from Italian Tweets using
RNN and Transformers
Kayalvizhi S Thenmozhi D Aravindan Chandrabose
SSN College Of Engineering SSN College Of Engineering SSN College Of Engineering
kayalvizhis@ssn.edu.in theni d@ssn.edu.in aravindanc@ssn.edu.in
Abstract and then detected using Multi-layer Perceptron
(MLP) (Riedel et al., 2017). Different method-
Stance detection refers to the detection of
ologies like Support Vector Machine, Long Short
one’s opinion about the target from their
Term Memory (LSTM) and Bi-directional LSTM
statements. The aim of sardistance task is
(Augenstein et al., 2016) have also been used to
to classify the Italian tweets into classes of
detect stance. Recurrent Neural Network (RNN)
favor, against or no feeling towards the tar-
(Yoon et al., 2019) and altering recurrent net-
get. The task has two sub-tasks : in Task
works with different short connections pooling
A, the classification has to be done by con-
and attention layers have also been experimented
sidering only the textual meaning whereas
in (Borges et al., 2019) to detect stance. Bi-
in Task B the tweets must be classified
directional Encoder Representation of Transform-
by considering the contextual information
ers (BERT) (Devlin et al., 2018) and Named En-
along with the textual meaning. We have
tity Recognition (NER) model (Küçük and Can,
presented our solution to detect the stance
2019) have also been used to detect stance. A large
utilizing only the textual meaning (Task A)
dataset has been collected from twitter and all the
using encoder-decoder model and trans-
existing approaches have been discussed in (Con-
formers. Among these two approaches,
forti et al., 2020).
simple transformers have performed bet-
For other languages, a multilingual data set
ter than the encoder-decoder model with
(Vamvas and Sennrich, 2020) have been taken,
an average F1-score of 0.4707.
language is identified and then multi-lingual
1 Introduction BERT model have been used to detect stance.
Stance have been detected in Russian Language
Stance is the opinion of a person against or in fa-
(Lozhnikov et al., 2018) by vectorizing using Tf-
vor of the target. In the sardistance task, the stance
IDF and then classifying using different classifiers
detection refers to the detection of stance from
like Bagging, AdaBoost Boosting, Stochastic Gra-
the Italian tweets collected from Sardines move-
dient Descent classifier and Logistic Regression.
ment. The tweets imply the authors’ standpoint
Stance from different languages (Lai et al., 2020)
towards the target. The aim of this task is to detect
like English, Italian, French, Spanish have been
the stance of the author with the help of textual
detected using different features extraction.
and contextual information about the tweets. The
task has two sub-tasks in which the stance is de- 3 Task Description
tected using only textual information in one sub-
task while the other sub-task makes use of contex- The sardistance task (Cignarella et al., 2020) of
tual meaning along with the textual meaning. Evalita (Basile et al., 2020) has two sub-tasks
namely Task A - textual stance detection and Task
B - contextual stance detection.
2 Related Work Both tasks are classification tasks that have three
Many approaches have been done to detect stance classes namely favor, against and none. In the first
from the English text. Stance text are vectorized task, the system has to predict the class by us-
ing only the textual information from the tweets
Copyright c 2020 for this paper by its authors. Use per-
mitted under Creative Commons License Attribution 4.0 In- whereas in the second task it has to predict the la-
ternational (CC BY 4.0). bel with the help of some additional information
like 4.2.1 Encoder-Decoder Model
Details of post : the number of re-tweets, replies, The encoder-decoder model is a Neural Machine
quotes Translation (NMT) model with sequential data
Details of user : the number of tweets, user bio’s, model with Recurrent Neural Network (RNN).
user’s number of friends and followers The Seq-to-Seq model differs in terms of type
Details of their social network : friends, replies, of recurrent unit, residual layers, depth, direc-
re-tweets, quotes’ relation. tionality and attention mechanism. The types of
In both the tasks, there can be two submissions the recurrent unit are Long Short Term Mem-
like constrained where we have to use only the ory(LSTM), Gated Recurrent Unit (GRU) and
dataset provided and unconstrained where we can Google Neural Machine Translations. The depth
use some additional data if required. Each team is altered by changing the number of layers and
can submit two runs for both constrained and un- the directionality is either uni-directionality or bi-
constrained runs. directionality.The two types of attention mecha-
nism are scaled luong (sl) and normed bahdanau
3.1 Data set description
(nb). The given training set is divided into devel-
For Task A, the train.csv file was provided with opment set and training set and the performance
three columns namely tweet id,user id and text la- is measured using the development set which is
bel. For Task B, files namely tweet.csv, user.csv, shown in Table 2. The model was trained for about
friend.csv, quote.csv, reply.csv and re-tweet.csv “10,000 steps”, 6 epoch step with “128 units”,
are given to explain the contextual details about batch size of “128”, dropout of “0.2” and learning
the post, user and social network. For both the rate of “0.1”.
tasks, the training set had about 2,132 instances
and the test set had about 1,110 instances. In 4.3 Transformers
the training set, there are 1,028 instances in the In this approach, the stances were detected using
against class, 587 favor instances and 515 neutral simple transformers. Simple transformers are the
instances which is explained in Table 1. In the test- wrapper of transformers. Transformers are mech-
ing set, there are 742 against instances, 196 favor anism that utilizes the attention mechanisms with-
instances and 687 none instances. out using recurrent units. Bi-directional Encoder
Representation of Transformers (BERT) is used to
detect stance with the multilingual model and base
4 Methodology
model for the development set whose performance
The stances were detected using an encoder- is given in Table 3. Multilingual Bert model (De-
decoder model which is a recurrent neural network vlin et al., 2018) of hugging face Pytorch trans-
with different recurrent units and using transform- formers (Wolf et al., 2019) has been used to de-
ers. tect stance in our approach which was submitted
as Run-1.
4.1 Data pre-processing
The data is pre-processed by removing the hash 5 Results
tags, ’@’ symbols, Unicode characters and punc-
Table 2 shows the different models evaluated
tuation.
based on the development set. From the table, the
4.2 Recurrent Neural Network model with two layers of gated recurrent unit and
scaled luong attention mechanism seems to per-
In this approach, the stance were detected using a
form better.
encoder-decoder model (Luong et al., 2017) using
Table 4 shows the performance of various teams
Gated Recurrent unit(GRU) as its recurrent unit
in this task of detecting stance. Twelve teams
and Scaled Luong (Luong et al., 2015) as its at-
have participated in which one team have submit-
tention mechanism. The model has two encoder-
ted both constrained and unconstrained runs which
decoder layers along with the embedding layer
is denoted by the suffix “ u” in the table. Remain-
that vectorizes the input and a loss layer that calcu-
ing all runs are constrained runs which are done
lates the loss function. Recurrent Neural Network
only using the data set provided.
has been made use to detect the stance since it cap-
tures the contextual long-short term dependencies.
Data Distribution against favor none Total
Training set 1028 587 515 2132
Testing set 742 196 172 1110
Total instances 1770 783 687 3242
Table 1: Data distribution
Model name Accuracy Acknowledgments
2l nb gru 37.0
We would like to express our gratefulness towards
2l sl gru 38.0
DST-SERB funding agent and HPC laboratory of
3l nb gnmt 33.7
SSN College Of Engineering for providing space
3l sl gnmt 33.7
and resources required for this experiment.
4l nb gru 36.4
4l sl gru 35.7
3l sl gnmt residual 37.5 References
3l nb gnmt residual 37.5
Isabelle Augenstein, Tim Rocktäschel, Andreas Vla-
chos, and Kalina Bontcheva. 2016. Stance detec-
Table 2: Performance of various models tion with bidirectional conditional encoding. arXiv
preprint arXiv:1606.05464.
Model mcc loss function Valerio Basile, Danilo Croce, Maria Di Maro, and Lu-
Bert- Multilingual 0.167 1.098 cia C. Passaro. 2020. Evalita 2020: Overview
of the 7th evaluation campaign of natural language
Bert - Base 0.141 1.150 processing and speech tools for italian. In Valerio
Basile, Danilo Croce, Maria Di Maro, and Lucia C.
Table 3: Performance of BERT models Passaro, editors, Proceedings of Seventh Evalua-
tion Campaign of Natural Language Processing and
Speech Tools for Italian. Final Workshop (EVALITA
The performance metrics used are class-wise 2020). CEUR-WS.org.
prediction of precision, recall, F1-score and aver- Luı́s Borges, Bruno Martins, and Pável Calado. 2019.
age F1-score. The ranking is done using an av- Combining similarity features and deep representa-
erage F1-score which is shown in 4. The best per- tion learning for stance detection in the context of
formance in constrained run is 0.6801 whereas our checking fake news. Journal of Data and Informa-
tion Quality (JDIQ), 11(3):1–26.
approach of transformers (SSN NLP run 1) has an
average F1 score of 0.4707 and encoder-decoder Alessandra Teresa Cignarella, Mirko Lai, Cristina
model (SSN NLP run 2) has an average score of Bosco, Viviana Patti, and Paolo Rosso. 2020.
0.4473. SardiStance@EVALITA2020: Overview of the Task
on Stance Detection in Italian Tweets. In Valerio
Basile, Danilo Croce, Maria Di Maro, and Lucia C.
Passaro, editors, Proceedings of the 7th Evalua-
6 Conclusion tion Campaign of Natural Language Processing and
Speech Tools for Italian (EVALITA 2020). CEUR-
WS.org.
Italian tweets about the Sardines movement have
been utilized to detect the opinion of the author Costanza Conforti, Jakob Berndt, Mohammad Taher
Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd,
towards the target. Different approaches have and Nigel Collier. 2020. Will-they-won’t-they: A
been made to detect the stance in the tweets by very large dataset for stance detection on twitter.
many other teams. We detected the stance using arXiv preprint arXiv:2005.00388.
encoder-decoder model and simple transformers
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
of multilingual Bert model in which transformers Kristina Toutanova. 2018. Bert: Pre-training of
performed better than the encoder-decoder model deep bidirectional transformers for language under-
with a F1-average score of 0.4707. The perfor- standing.
mance can further be improved by utilizing the ad- Dilek Küçük and Fazli Can. 2019. A tweet dataset
ditional dataset to train the model better to detect annotated for named entity recognition and stance
the stance in the tweets. detection. arXiv preprint arXiv:1901.04787.
Team F-average
SSN NLP run 1 (transformers) 0.4707
SSN NLP run 2 (encoder-decoder model) 0.4473
Team A - 1 u 0.6853
Team A - 1 c 0.6801
Team A - 2 c 0.6793
Team B - 1 0.6621
Team A - 2 u 0.6606
Team C - 1 0.6473
Team D - 1 0.6257
Team C - 2 0.6171
Team E 0.6067
Team B - 1 0.6004
Team D - 2 0.5886
Team F 0.5784
Team G - 1 0.5773
Team H 0.5749
Team I - 1 0.5595
Team I - 1 0.5329
Team J 0.4989
Team G - 2 0.4705
Team K 0.3637
Table 4: Results
Mirko Lai, Alessandra Teresa Cignarella, Delia ric Cistac, Tim Rault, Rmi Louf, Morgan Funtow-
Irazú Hernández Farı́as, Cristina Bosco, Viviana icz, Joe Davison, Sam Shleifer, Patrick von Platen,
Patti, and Paolo Rosso. 2020. Multilingual stance Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
detection in social media political debates. Com- Teven Le Scao, Sylvain Gugger, Mariama Drame,
puter Speech & Language, page 101075. Quentin Lhoest, and Alexander M. Rush. 2019.
Huggingface’s transformers: State-of-the-art natural
Nikita Lozhnikov, Leon Derczynski, and Manuel Maz- language processing. ArXiv, abs/1910.03771.
zara. 2018. Stance prediction for russian: data and
analysis. In International Conference in Software Seunghyun Yoon, Kunwoo Park, Joongbo Shin,
Engineering for Defence Applications, pages 176– Hongjun Lim, Seungpil Won, Meeyoung Cha, and
186. Springer. Kyomin Jung. 2019. Detecting incongruity between
news headline and body text via a deep hierarchical
Minh-Thang Luong, Hieu Pham, and Christopher D encoder. In Proceedings of the AAAI Conference on
Manning. 2015. Effective approaches to attention- Artificial Intelligence, volume 33, pages 791–800.
based neural machine translation. arXiv preprint
arXiv:1508.04025.
Minh-Thang Luong, Eugene Brevdo, and Rui Zhao.
2017. Neural machine translation (seq2seq) tutorial.
https://github.com/tensorflow/nmt.
Benjamin Riedel, Isabelle Augenstein, Georgios P Sp-
ithourakis, and Sebastian Riedel. 2017. A sim-
ple but tough-to-beat baseline for the fake news
challenge stance detection task. arXiv preprint
arXiv:1707.03264.
Jannis Vamvas and Rico Sennrich. 2020. X-stance:
A multilingual multi-target dataset for stance detec-
tion. arXiv preprint arXiv:2003.08385.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pier-