In this paper we describe our participation to the SardiStance shared task held at EVALITA 2020. We developed a set of classifiers that combined text features, such as the best performing systems based on large pre-trained language models, together with user profile features, such as psychological traits and social media user interactions. The classification algorithms chosen for our models were various monolingual and multilingual Transformer models for text only classification, and XGBoost for the non-textual features. The combination of the textual and contextual models was performed by a weighted voting ensemble learning system. Our approach obtained the best score for Task B, on Contextual Stance Detection.
One of the most important research topics in the field of Natural Language Processing (NLP) is automatic information extraction from textual data. The recent rise of social media has completely changed the way in which people communicate their ideas and has thus led to the emergence of new research problems regarding the automatic analysis of online contents, such as sentiment analysis, emotion recognition, or fake news detection. Stance detection (usually considered as a subproblem of sentiment analysis) is part of the aforementioned family of research problems
Copyright © 2020 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
Thus, we address the problem of
automatic stance detection in tweets written in
Italian language for the SardiStance 2020 shared
task
In this sense, the main idea behind our model is to exploit textual information, based on finetuning large pre-trained language models for text classification, together with contextual information using several feature categories, such as psychological traits of the user, social media data, and network based features. As a result of our joint effort, we submitted 4 and 5 runs, respectively, to tasks A and B. The official results show that our systems obtained the 3rd position among the constrained runs submitted to Task A, which considered only textual information for prediction, and 1st position from 13 participants for Task B, which considered textual and contextual information.
In this section we first describe the text classification systems developed for Task A and then the contextual features used to train XGBoost classifiers for Task B. We also include a description of the strategies used to combine the classifiers from both tasks, which resulted in the winner system for Task B. 2.1
The main objective of our participation in Task A
was to benchmark the performance, on the stance
detection task for Italian, of large pre-trained
language models based on the transformer
architecture
As for many other Natural Language Processing
(NLP) tasks, current best performing systems for
text classification are based on large pre-trained
language models which allow to build rich
representations of text based on contextual word
embeddings. Deep learning methods in NLP
represent words as continuous vectors on a low
dimensional space, called word embeddings. The
first approaches generated static word embeddings
In order to address this problem, contextual word embeddings were proposed. The idea is to be able to generate word representations according to the context in which the word occurs. Currently there are many approaches to generate such contextual word representations, but we will focus on publicly available multilingual and monolingual pre-trained models for Italian.
There are several multilingual versions of these
models. Thus, the multilingual version of BERT
These multilingual models perform very well in
tasks involving high-resourced languages such as
English or Spanish, but their performance drops
when applied to languages not so well represented
in the language model
We have benchmarked four monolingual pretrained language models for Italian: AlBERTo, GilBERTo, UmBERTo and Italian BERT XXL with the aim of comparing them with respect to the multilingual pre-trained models previosly mentioned, namely, mBERT and XLM-RoBERTa.
AlBERTo is a BERT base pre-trained
lowercased model containing a vocabulary of 128k
terms from 200M of Italian tweets
The Italian BERT XXL models 2 are also based
on the BERT base architecture. The training data
contains the Italian Wikipedia, various parts of the
OPUS corpus and the OSCAR corpus for Italian
GilBERTo3 is based on the RoBERTa base
UmBERTo5 also leverages the RoBERTa base architecture, the OSCAR corpus for Italian and the SentencePiece tokenizer, but it adds Whole Word Masking to the training process. The idea is to mask an entire word, instead of subwords, if at least one of all (sub-)tokens generated by SentencePiece was originally selected as mask.
In this task, we use several sets of features with the purpose of trying to model user’s behaviour when writing a tweet. We obtain such features from both the text and the social network. Our hypothesis is that the stance of a user regarding a particular tweet is highly correlated with the way of writing of the own user extracted in terms of psychological and emotional features. On the other hand, we focus on exploring how the concept of “homopohily”, namely, the tendency of individuals to associate and bond with similar individuals, previously studied in DellaPosta et al. (2015). In order to test this hypothesis, we have tested different models that are explained below.
In this task, we use several sets of features with the purpose of trying to model user’s behaviour when writing a tweet. We obtain such features from text and the network.
The complete set of features extracted from the data is depicted in Table 1. The set of features used in the model can be divided into five main types: psychological, emotional, Twitter-based, networkbased, and language model features.
Psychological features. These features were extracted using a third-party API developed by Symanto6. Each tweet was sent to the API in order to retrieve the personality traits and communication styles obtained from the analysis of the tweet contents.
The personality traits value would be either “emotional” or “rational” depending on the analysis of the user’s text. The value returned by the API when the communication styles are re6https://symanto-research.github.io/symanto-docs/ quested is a collection of traits, such as selfrevealing, which means sharing one’s own experience and opinion; fact-oriented, which implies focusing on factual information, objective observations or statements; information-seeking, that is, posing questions; and action-seeking or aiming to trigger someone’s action by giving recommendation, requests or advice.
Emotional features. In order to retrieve the
emotion values from the tweets, we used Russell’s
circumplex model of affect
The values for the degree of arousal and valence
of the tweets were obtained using an adaptation to
Italian language of the Affective Norms for
English Words (ANEW)
Twitter features. Exploring how the users behave in the social network could offer some insights on the stance tendency of the users. The collection of Twitter data of each user contained four features: the number of statuses published by the user, the number of users followed by the user, the number of users following the user, and the creation date of the Twitter account of the user.
Network features. Using the FRIEND.csv data provided, we built a network consisting of 669817 nodes (or users) and 2847197 edges (or relationships) in order to represent the following network of the users. From that network, we extracted a sub-graph containing the users of known stance from the training data and the users involved in testing in order to calculate the mean distances of each user to the rest of known stance users using the following formula: dT (n) =
PjiT=j1 d2n!i
1 jT j where jT j is the total number of users of a determined stance (AGAINST, FAVOR, NONE) and
2 dn!i corresponds to the square distance in users from node n to node i. From this calculation we obtained 3 values per user: mean distance to users against (dagainst), mean distance to users in favor (dfavor), and mean distance to neutral users (dnone)
Language model features. In order to incorporate the language model results into the rest of the features of the system we choose the best performing, at the development phase, of the models described in Section 2.1, which was UmBERTo. Since this kind of language models use a great amount of features for learning and training, the strategy used in order to incorporate the language model without having a great imbalance in the number of features representing each category, consisted in extracting the probabilities assigned by the model to each class for each tweet. In this way, the language model would be present in 3 of the 18 features of the model, and it would therefore have a balanced size with regards to the rest of features of the model. 3 3.1
As we use the base version of every transformer model we can fine-tune them in a basic GPU of 12GB RAM. Hyperparameter tuning (batch size, maximum sequence length, learning rate and number of epochs) was performed on the development set. For mBERT, AlBERTo, Italian BERT XXL and UmBERTo the best configuration was: maximum sequence length 256, batch 32, learning rate 5e-5, and 5 epochs. For GilBERTo we used the same values except the number of epochs, which was increased to 10. Finally, the best performing hyperparameters for XLM-RoBERTa was the following: maximum sequence length 256, batch 16, learning rate 2e-5, and 10 epochs.
While the monolingual models clearly outperformed both mBERT and XLM-RoBERTa on the development data, we decided to submit the three best monolingual runs and the best multilingual one. Table 2 reports the official results obtained by each of the models and their position with respect to the ranking of constrained runs for Task A released by the task organizers. Our submission based on Italian BERT XXL was clearly the best of our four runs, although its performance was around 1.5 scores in F1 lower than the winner system for Task A. Furthermore, the ranking obtained in the test does not correspond with the results obtained during the development phase, where UmBERTo outperformed the other monolingual models by more than 3 points in F1 score. 3.2
We presented a total of five models to Task B, which consisted of different combinations of the features listed in Table 1.
Models 1, 2, and 3. During the training and
development phases of the models, several
configurations were tested on models 1, 2, and 3,
including training with different classifiers, such as
Random Forest Classifier, Decision Tree Classifier and
XGBoost Classifier. The best performing
classifier was XGBoost configured for multi-class
classification and taking into account class weights in
order to deal with the imbalance present in the
data. XGBoost is an efficient and scalable
implementation of gradient boosting framework by
Models 4 and 5. These two models were constructed using voting based ensemble learning. The voting system for model 4 considered predictions of models 1, 2, and 3 as well as predictions by the best performing language models on
Ixa DeepReading DeepReading
UNED
UNED the development data: UmBERTo, GilBERTo, and Italian BERT XXL, described in Section 2.1. The most common predicted value among the 6 systems was chosen as the final prediction of model 4. In case of having two or more values with the same counts, the final value is randomly selected. On the other hand, model 5 used a weighted voting ensemble learning in which each of the systems considered had as weight the F1 value obtained on the development data. Therefore, the model considered the weighted predictions of each system in order to choose the final prediction.
Table 3 shows the official results obtained by each model and their position with respect to the ranking for Task B on Contextual Stance Detection. As it can be noted, model 5 ranked first in this task, obtaining an average F1 of 0.7445. Models 3 and 4 also had promising results in the official test set, ranking third and fourth, respectively, and just 0.0079 below the system which obtained the second best result. Model 2 had a slightly worse performance, ranking seventh from a total of 13, but still 0.0604 above the baseline. Finally, model 1 had the lowest performance, ranking last for the task. 4
Furthermore, we can see that predictions from model 3 also experimented a great increase in true positives of each of the classes. This increase is related to the inclusion of the language model into the features of model 2, which demonstrates the importance of textual data in stance detection tasks.
Finally, models 4 and 5 shows the adequacy of combining several complementary systems in order to improve results. Since each single model can detect the stance for different instances, a proper combination of them could outperform single models. 5
In this paper we have shown the benefits of exploiting information from different and heterogeneous sources. For our participation to the SardiStance 2020 shared task we have experimented with classifiers trained with the textual content of the tweets as well as with features based on social networks. This combination of features has allowed us to obtain the best overall results in the task.
As future work, we plan to further explore the contribution of network information. Besides, we want to develop new divergent models and study how to combine them. This work has been partially funded by the Spanish Ministry of Science, Innovation and Universities (DeepReading RTI2018-096846-B-C21, MCIU/AEI/FEDER, UE), and DeepText (KK2020/00088), funded by the Basque Government. Rodrigo Agerri is additionally funded by the RYC2017-23647 fellowship and acknowledges the donation of a Titan V GPU by the NVIDIA Corporation. Maria S. Espinosa is also funded by the European Social Fund through the Youth Employment Initiative (YEI 2019).