This paper presents our submission to the SardiStance 2020 shared task, describing the architecture used for Task A and Task B. While our submission for Task A did not exceed the baseline, retraining our model using all the training tweets, showed promising results leading to (favg 0.601) using bidirectional LSTM with BERT multilingual embedding for Task A. For our submission for Task B, we ranked 6th (f-avg 0.709). With further investigation, our best experimented settings increased performance from (f-avg 0.573) to (f-avg 0.733) with same architecture and parameter settings and after only incorporating social interaction features- highlighting the impact of social interaction on the model's performance.
Framed as a classification task, the stance
detection consists in determining if a textual utterance
expresses a supportive, opposing or neutral
viewpoint with respect to a target or topic
In social media, classical features can be
extracted by using stylistic signals from text such as
bag of n-grams, char-grams, part-of-speech labels,
and lemmas
The stance detection task has been defined in
previous work as consisting in determining the
viewpoint of an utterance with respect to a
target topic
quency vectors including unigram, punctuation
and hashtags provided by
Italian Wikipedia Embedding
2https://fasttext.cc/docs/en/
pretrained-vectors.html
3https://github.com/mirkolai/
one trained by us without any preprocessing and
each vector has 300 dimensions, referred to as
TWITA300. We also experimented with
multilingual BERT in Task A 4
To represent user interactions as nodes and edges, we used a counting scalar value and added one if each of the following relationships exists: friendships, retweets, quotes and replies, e.g. if all of them exist then the edge weight between two accounts is four. We calculated all the accounts provided and generate a directed complex graph conditioned by the existence of friendship, resulting in 669,745 nodes, 2,871,791 edges with an average in-degree of 4.2879 and average out-degree of 4.2879.
Generating GNN Embeddings. Taking as input
the encoded network relationships, GNN
embeddings use different sampling techniques to
represent every node as a vector. To extract these
vectors, we experiment with different graph
neural network models, namely struct2vec
Figure 1, we have different deep learning models to extract features separately for both word embedding and similarity vectors matrices. In our work, we experiment with Convectional Neural Network (CNN) models and Long short-term memory (LSTM) models. Variations of CNN models where applied to NLP downstream tasks as feature extraction methods for text classification. In our work, we used two variations of CNN. In one model, we used a CNN as a one-head 1DCNN with kernel size of 5 allowing the model to extract features with 5-grams vectors using 32 filters. Followed by a max pooling layer with pool size of 2 then flattened layer. In another model, we used a CNN as a multi-headed 2D-CNN with 1, 2, 3, 5 grams filter sizes, initialising the kernel weights with a Rectified Linear Unit (ReLU) activation function and normal distribution weights. Followed by a max pooling layer with different evalita-sardistance/
4https://tfhub.dev/tensorflow/bert_
multi_cased_L-12_H-768_A-12/2
pooling sizes taken as one columns pooling
filter with the maximum text length excluding few
grams sizes. For the LSTM, we used two variants.
One is a simple bidirectional LSTM of 64 units
followed by concatenations of max pooling and
average pooling layers, and attention bidirectional
LSTM proposed by
Sentence Cleaning. We set the cleaning function
to match the preprocessing function by
We used four final layers to receive the features and concatenate them (see Figure 1). In all of the experiments, our dropout layer set to 0:2, followed by a dense layer with rule activation function and another dropout layer of 0:2. Finally, a probability vector of the three classes is generated. To determine the correct class, we choose the one class with the highest probability. 5
In this section, we discuss the results of our systems submitted to the two tasks.
For Task A, we used attention Bidirectional LSTM model performance compared to using different word embedding models, also we analysed impact of the preprocessing of the runs. Since there are too many parameters to compare with, we compared the performance of the embedding models. Our submitted models, BERT and TWITA300 illustrated in Table 1 with showed most promising results using different settings. With only %80 training data, similarity vectors generalised better than all other embedding models. While, when all data are trained, the best model is the multilingual BERT embedding with no pre-processing (f-avg 0.601), followed by similarity vectors using cleaned text (f-avg 589).
For Task B, we used different feature extraction, frequency vectors, word embedding and social interaction embedding models, and monitor their performance while activating the pre-processing step in all experiments. With a diverse range of parameters, we experimented with a total of 3845 random runs. Then, we selected the best mod5https://www.kaggle.com/mlwhiz/ attention-pytorch-and-keras Tst. f-avg els considering macro f-score for the two classes under consideration (AGAINST and FAVOR) (favg). Results are shown in Table 2. By comparing our runs by adding social interaction features, our models with different settings showed a clear improvement on our models. In 1#M, we utilise Conv2D (see NeuralNetwork-based features) for embedding vectors with TfiDF unigram and tweet length, where the model achieved an increase on performance of (f-avg 0.16) when social interaction vectors incorporated into the model. All other models showed the same improvement with an increase of (f-avg 0.115, 0.118, 0.081, 0.021) for 3#M, 5#M, 7#M and 9#M, respectively. 6
The pipeline depicted in Figure 1 was designed to investigate the impact of multiple features on stance detection using variations of feature extraction methods, which have been experimented in previous work but we adapted them to the Italian language in our settings. The training set contains 2132 instances with no evaluation set. In our work, we create a stratified split of 80-20 to evaluate the model, which leads to a training data with 1705 samples. Further, our investigation attempted to randomise different settings, with the aim of submitting the top two with highest f-avg score on the remaining set (Eval. 426) for both tasks. Consequently, we found that this methodology did not generalise well with the testing results. However, our main findings remain consistent across different settings when compared with our results using the stratified split (T%80) and when the model was retrained using all the data (T%100). While our submission evaluated both tasks separately, we discuss all conclusions jointly in this section.
Having different random settings over all frequency-based features (14, in our case) would be a bad strategy to evaluate the methods and come up with the best approach. To verify if we need to include all of these, we run an experiment by including only one feature from (unigram, Tfidf_unigram, chargrams, network_reply_community, userinfobio). The selection of these features where based on selecting the best runs using only one feature from our randomised parameters. Using all the training set and CONV2D with (fasttext;TWEC300) and reduced SVs with deepwalk user’s social interaction vector, (userinfobio;chargrams) achieved (favg 0.703 and 0.704), respectively. This is also higher than using AttLSTM for the same settings which achieved (f-avg 0.638 and 0.610).
In general, we achieve better performance with CONV2D than AttnLSTM for the same settings on the test data. In another experiment, we reduced all the 14 frequency-based parameters achieving (f-avg 0.714) which performs worse than our best 3#M (see 2). Our main conclusion is that the number of features available is not necessarily correlated with the model’s performance boost.
In another experiment, we attempted to compare the performance of TWEC100 with TWEC300 (see Section 4). From Table 1, we observed that lower dimensionality and preprocessing may cause the model to under perform by around (f-avg 0.050), at least. Though, this impact was not significant with T%100. However, matching the processing between the embedding vocabulary and the annotated set yields better performance. For example, TWITA100 was more persistent on performance between T%80 and T%100. This highlights the importance of pre-processing and reducing the differences between the embedding vocabularies and labelled sentences. In general, our embedding experiment for Task A show high sensitivity on model performance with pre-processing settings.
Inspired by previous work on encoding word meanings, we experimented with SVs embedding.
Interestingly, these vectors showed high f-avg, Eval.
f-avg
Tst. f-avg
T%80 T%100 % are the ones represented with . Bold fonts show highest/above baseline results better than BERT and TWITA300 with T%80 and social interactions. Using different random although it showed a significant drop when the runs, our best model achieved (f-avg 0.733) levermodel was trained with T%100. This finding aging deepwalk-based knowledge graphs embedopens an investigation towards the ability of SVs dings, FastText and similarity feature vectors exto perform better under different settings. For that, tracted by two multi-headed convolutional neural we removed PCA(SVs) and run same settings of networks from auther’s utterance. This motivates #M1, and our model achieved (f-avg 0.678), showour future, aiming to reduce the model complexity ing a significant impact of SVs on model’s perforand automate the feature selection process. mance. Further, we investigate the robustness of deepwalk modelling over node2vec and struct2vec for the same best settings of #M1, resulting on (favg 0.641 and 0.604) for node2vec and struct2vec, respectively. Also, in terms of accuracy, the deepwalk model produces an improved accuracy of (% 0.725) compared to node2vec (% 0.665) and struct2vec (% 0.658). This indicates that deepwalk is more reliable on this testing set than other models. 7
In this work, we described a state-of-the-art stance detection system leveraging different features including author profiling, word meaning context 8
This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT.