Detecting stances is a Natural Language Processing task that has focused mainly on analysing debates and controversial topics. In this case, the VaxxStance@IberLEF 2021 shared task has focused on the Antivaxxers movement in Basque and Spanish tweets. In this paper, we present the participation of the MultiAzterTest team and test two approaches: a language model based approach and a linguistic and stylistic feature based approach. We also introduce the \one stance per tuiter@lari" heuristic to integrate contextual information. The best results are obtained with language models, but the linguistic and stylistic feature based approach o ers more interpretability.
Identifying stances in social media has gained a lot of interest in the Natural
Language Processing (NLP) community and debates [21] and political debates
have been the main topics [
In this paper we present the participation of the MultiAzterTest team in the
close track of VaxxStance@IberLEF 2021, a language-speci c evaluation where
only the provided data for each language is allowed to use. There are, moreover,
two settings in this track: i) textual, where only the tweets can be used and ii) the
contextual, where in addition to the texts, features related to user-based Twitter
information can be used. To tackle this task, we present two approaches: the rst
one is based on language models and the second one is based on linguistic and
stylistic features plus a classical machine learning classi er. In order to include
the contextual information, we apply a heuristic inspired by the \one sense per
discourse" [
Language models have been proved to be very e ective in many NLP tasks. However, they lack of interpretability. That is why, we think that the use of linguistic and stylistic features may help to understand the underlying linguistic characteristics that are used when expressing a con or pro opinion. With that in mind, our aim is to explore if these features help in the task.
This paper is structured as follows: in Section 2 we present the corpus analysis carried out with MultiAzterTest, in Section 3 we describe our approaches and the experimental set-up, in Section 4 we present the results and we conclude and outline the future work in 5. 2
In this section we present the exploratory analysis of the linguistic features of
VaxxStance corpus [
Based on the linguistic and stylistic pro ling of MultiAzterTest, we present in Table 1 the mean of some descriptive linguistic features of the VaxxStance dataset.
As we can see, the words and the lemmas are longer in Basque than in Spanish, but sentences are longer and deeper in Spanish. The propositions per sentence is similar in both languages. Regarding lexico-semantic measures, there are more di erent words in Basque, but there are less rare words. This may indicate that although there are many di erent words, these words are common. The incidence of content words is also bigger in Basque due to its typology. In the Basque words, there are more words that express the negation (negative particles) than in Spanish. However, the incidence is low and this leads us to think that contrary opinions may be subtle and not so direct. The use of connective is much bigger in Basque. It would be interesting to see if Basque tweets are written in a more formal, elaborated register than the Spanish ones.
Due to the high number of features, it is possible that some of them highly
correlate in this dataset. We have also analysed the correlation with the python's
package Feature Selector [
Finally, we have also analysed the most predictive features with Weka's [
In this section we present the approaches we have followed to perform the stance classi cation task. 3.1
Language Model Approach
The Language Model (LM) approach uses BERTeus [
For the experiments, we have truncated the texts with more than 200 tokens and padded, the shorter with zeroes. We have added two tokens to mark the beginning and the end of the sequence to each input text, [CLS] and [SEP] respectively. We have applied a pre-processing step besides the standard bytepair encoding. This pre-processing step consists on segmenting the hashtags, replacing `&' with `&', removing trailing white-spaces and nally converting the text to lowercase only in Spanish. We have used the PyTorch framework to create our model. We have probed with two sequential models on top of BERT: { A dropout layer to ght over tting. The dropout probability was set equal to 0.1. On top of the dropout Layer, we have added a linear layer and sigmoid activation function. The input dimension of the linear layer was 768 and the output 3 (equal to the number of classes). { A linear layer, ReLU activation function and linear layer model. The input dimension of the rst linear was 768 and the output 50, and the input dimension of the second linear was 50 and the output 3 (equal to the number of classes).
For each of the outputs, we have used the cross-entropy loss function.
To train the model, we have split the training data into 80 % for train and
20 % for validation. The training batch size was made equal to 32 and the model
was trained for 10 epochs using early stopping technique. The best result in the
validation data was obtained after running 9 epochs in Basque and 6 epochs in
Spanish, setting the tweet length to 200, and the learning rate to 5e-06 with
Linear-ReLU-Linear sequential model and the Adam optimizer [
Regarding the evaluation, the metric we use is F 1 Macro, the one used in the
VaxxStance shared task and proposed by Mohammad et al. [
In Table 3 we present the results in the training and validation data. As we can see, the results seem to be competitive and they are higher in Spanish than in Basque. 3.2
Approach based on Linguistic Features and Machine Learning The second approach consists on the use of linguistic and stylistic plus a classical machine learning classi er.
Obtaining linguistic features First, in order to get the linguistic and
stylistic features, we have used MultiAzterTest [
In Table 4 we present the results (F 1 macro) of these experiments. Contrary
to what happens in readability assessment [
So, for each user, we take the most predicted label by the system and apply it to the rest of its tweets. In the case of tie, we apply the favor label. In Algorithm 1, we present the OSPT algorithm. 4
In this section we present the results obtained in the test data for the textual and the contextual evaluation settings as provided by the organisers.
Looking at the results of the textual setting (Table 5), we see that the LM approach gets better results in the F 1 macro than the linguistic features plus SMO (MATS-Vaxxstance) in both Basque and Spanish. Moreover there is a big di erence between them: almost 16 points in Basque and 10 in Spanish. It is also remarkable that the LM approach works much better in Spanish than in else
end end end end Algorithm 1: \One stance per tuiter@lari" (OSPT) algorithm for each tweet do userid=Get the owner id of the tweet ; countusertweets=Count the number of tweets from that user ; if countusertweets 2 then countuserfavortweets=counts predicted user tweets as favor; countuseragainsttweets=counts predicted user tweets as against; if countuserfavortweets == 0 and countuseragainsttweets == 0 then pass; if countuserfavortweets countuseragainsttweets then
changeallusertweets(userid,'FAVOR'); else
changeallusertweets(userid,'AGAINST'); Basque (around 24 points of di erence). There are also important di erences in the retrieval of Against and Favor instances, except for the case of the MATSVaxxStance in Basque.
Concerning the contextual setting (Table 6), we obtain mixed results. In Spanish, the LM+OSPT approach obtains the highest F 1 macro while in Basque MATS-VaxxStance+OSPT performs better, although the between approaches di erence is insigni cant. In Spanish, however, the di erences are bigger (9 points). Regarding the retrieval of the Against and Favor instances, there are also big di erences, except for the case of the Spanish LM.
As we have only applied the \one stance per tuiter@lari" in the contextual evaluation, we can see which is the consequences of applying this heuristic. In Basque, applying the heuristic in the LM approach worsens the results, while they are improved in the Spanish LM. Regarding the MATS-VaxxStance approach, it helps in both languages. This lead us to think that OSPT gives consistency to the MATS-VaxxStance results.
All results considered, we see that the language models, although they do not o er any interpretability and are far from perfect in this task, are more competitive than the linguistic information. We also want to point out that a corpus and resource analysis will be necessary to know why the results obtained in Spanish are better that the ones in Basque. 5
In this paper we have presented the participation of the MultiAzterTest team in the VaxxStance@IberLef 2021 shared task. We have participated in the two evaluation settings (textual and contextual) of the close track where we have presented two approaches: the rst is based on well-known languages models, exactly BERTeus for Basque and BETO for Spanish; and the second approach is based on linguistic and stylistic features provided by the open source tool MultiAzterTest plus SMO as classi er. To integrate the contextual features, inspired by the \one sense per discourse" idea and we have created the \one stance per tuiter@lari" heuristic, where if a user has mainly an opinion, we apply that label to the rest of its tweets. Regarding the results, the approach based on language models obtains in general better results and the results got for Spanish are better than those for Basque.
As this is a mainly exploratory work, there is a lot of work to do. Regarding
MultiAzterTest-Social more features need to be integrated in Basque. Besides,
more classi ers can be tested e.g. Random Forest [
We acknowledge the following projects: DeepText (KK-2020/00088), DeepReading RTI2018-096846-B-C21 (MCIU/AEI/FEDER, UE), BigKnowledge for Text Mining, BBVA and IXA taldea, A motako ikertalde nkatua (IT1343-19). 18. Montes, M., Rosso, P., Gonzalo, J., Aragon, E., Agerri, R., Angel Alvarez Carmona, M., Alvarez Mellado, E., de Albornoz, J.C., Chiruzzo, L., Freitas, L., Adorno, H.G., Gutierrez, Y., Zafra, S.M.J., Lima, S., de Arco, F.M.P., (eds.), M.T.: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021. CEUR Workshop Proceedings (2021) 19. Platt, J.: Fast training of support vector machines using sequential minimal optimization. In: Schoelkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods - Support Vector Learning. MIT Press (1998), http://research.microsoft.com/~jplatt/smo.html 20. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., Manning, C.D.: Stanza: A python natural language processing toolkit for many human languages. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 101{108 (2020) 21. Somasundaran, S., Wiebe, J.: Recognizing stances in online debates. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. pp. 226{234 (2009) 22. Taule, M., Pardo, F.M.R., Mart , M.A., Rosso, P.: Overview of the task on multimodal stance detection in tweets on catalan# 1oct referendum. In: IberEval@ SEPLN. pp. 149{166 (2018) 23. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Huggingface's transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)