Stance detection refers to the detection of one's opinion about the target from their statements. The aim of sardistance task is to classify the Italian tweets into classes of favor, against or no feeling towards the target. The task has two sub-tasks : in Task A, the classification has to be done by considering only the textual meaning whereas in Task B the tweets must be classified by considering the contextual information along with the textual meaning. We have presented our solution to detect the stance utilizing only the textual meaning (Task A) using encoder-decoder model and transformers. Among these two approaches, simple transformers have performed better than the encoder-decoder model with an average F1-score of 0.4707.
Stance is the opinion of a person against or in
favor of the target. In the sardistance task, the stance
detection refers to the detection of stance from
the Italian tweets collected from Sardines
movement. The tweets imply the authors’ standpoint
towards the target. The aim of this task is to detect
the stance of the author with the help of textual
and contextual information about the tweets. The
task has two sub-tasks in which the stance is
detected using only textual information in one
subtask while the other sub-task makes use of
contextual meaning along with the textual meaning.
Many approaches have been done to detect stance
from the English text. Stance text are vectorized
and then detected using Multi-layer Perceptron
(MLP)
For other languages, a multilingual data set
The sardistance task
Both tasks are classification tasks that have three classes namely favor, against and none. In the first task, the system has to predict the class by using only the textual information from the tweets whereas in the second task it has to predict the label with the help of some additional information like Details of post : the number of re-tweets, replies, quotes Details of user : the number of tweets, user bio’s, user’s number of friends and followers
re-tweets, quotes’ relation.
In both the tasks, there can be two submissions like constrained where we have to use only the dataset provided and unconstrained where we can use some additional data if required. Each team can submit two runs for both constrained and unconstrained runs. 3.1
For Task A, the train.csv file was provided with three columns namely tweet id,user id and text label. For Task B, files namely tweet.csv, user.csv, friend.csv, quote.csv, reply.csv and re-tweet.csv are given to explain the contextual details about the post, user and social network. For both the tasks, the training set had about 2,132 instances and the test set had about 1,110 instances. In the training set, there are 1,028 instances in the against class, 587 favor instances and 515 neutral instances which is explained in Table 1. In the testing set, there are 742 against instances, 196 favor instances and 687 none instances. 4
The stances were detected using an encoderdecoder model which is a recurrent neural network with different recurrent units and using transformers. 4.1
The data is pre-processed by removing the hash tags, ’@’ symbols, Unicode characters and punctuation. 4.2
In this approach, the stance were detected using a
encoder-decoder model
The encoder-decoder model is a Neural Machine Translation (NMT) model with sequential data model with Recurrent Neural Network (RNN). The Seq-to-Seq model differs in terms of type of recurrent unit, residual layers, depth, directionality and attention mechanism. The types of the recurrent unit are Long Short Term Memory(LSTM), Gated Recurrent Unit (GRU) and Google Neural Machine Translations. The depth is altered by changing the number of layers and the directionality is either uni-directionality or bidirectionality.The two types of attention mechanism are scaled luong (sl) and normed bahdanau (nb). The given training set is divided into development set and training set and the performance is measured using the development set which is shown in Table 2. The model was trained for about “10,000 steps”, 6 epoch step with “128 units”, batch size of “128”, dropout of “0.2” and learning rate of “0.1”. 4.3
In this approach, the stances were detected using
simple transformers. Simple transformers are the
wrapper of transformers. Transformers are
mechanism that utilizes the attention mechanisms
without using recurrent units. Bi-directional Encoder
Representation of Transformers (BERT) is used to
detect stance with the multilingual model and base
model for the development set whose performance
is given in Table 3. Multilingual Bert model
Model name 2l nb gru 2l sl gru 3l nb gnmt 3l sl gnmt 4l nb gru 4l sl gru 3l sl gnmt residual 3l nb gnmt residual 37.0 38.0 33.7 33.7 36.4 35.7 37.5 37.5
The performance metrics used are class-wise prediction of precision, recall, F1-score and average F1-score. The ranking is done using an average F1-score which is shown in 4. The best performance in constrained run is 0.6801 whereas our approach of transformers (SSN NLP run 1) has an average F1 score of 0.4707 and encoder-decoder model (SSN NLP run 2) has an average score of 0.4473. 6
Italian tweets about the Sardines movement have been utilized to detect the opinion of the author towards the target. Different approaches have been made to detect the stance in the tweets by many other teams. We detected the stance using encoder-decoder model and simple transformers of multilingual Bert model in which transformers performed better than the encoder-decoder model with a F1-average score of 0.4707. The performance can further be improved by utilizing the additional dataset to train the model better to detect the stance in the tweets.
We would like to express our gratefulness towards DST-SERB funding agent and HPC laboratory of SSN College Of Engineering for providing space and resources required for this experiment.
SSN NLP run 1 (transformers) SSN NLP run 2 (encoder-decoder model) Team A - 1 u Team A - 1 c Team A - 2 c Team B - 1 Team A - 2 u Team C - 1 Team D - 1 Team C - 2
Team E Team B - 1 Team D - 2
Team F Team G - 1
Team H Team I - 1 Team I - 1
Team J Team G - 2
Team K F-average 0.4707 0.4473 0.6853 0.6801 0.6793 0.6621 0.6606 0.6473 0.6257 0.6171 0.6067 0.6004 0.5886 0.5784 0.5773 0.5749 0.5595 0.5329 0.4989 0.4705 0.3637