In this paper we describe the participation of the WordUp! team in the VaxxStance shared task at IberLEF 2021. The goal of the competition is to determine the author's stance from tweets written both in Spanish and Basque on the topic of the Antivaxxers movement. Our approach, in the four dierent tracks proposed, combines the Logistic Regression classier with diverse groups of features: stylistic, tweet-based, user-based, lexicon-based, dependency-based, and network-based. The outcomes of our experiments are in line with state-of-the-art results on other languages, proving the ecacy of combining methods derived from NLP and Network Science for detecting stance in Spanish and Basque.
In the last ve years there has been a noticeable growth of interest in determining whether the author of a social media text is in favor, against, or neutral towards a statement or targeted event, person, organization, government policy or movement. The research area investigating such matter has been dened in literature as Stance Detection (SD) [31].
Investigating on this topic could have a huge impact on dierent aspects of everyday life such as policy-making, security choices and public administration strategies. A practical application of SD techniques, in fact, could support the automatic identication of people’s extremist tendencies on the one hand (e.g., ? The contribution of each author is detailed in the Acknowledgments section.
IberLEF 2021, September 2021, MÆlaga, Spain.
Copyright c 2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
religious extremism [
Along with the above mentioned shared tasks, which are representatives of
the so-called target-specific stance classification , there have been others
dedicated to a dierent fashion of categorization, i.e., open stance
classification. This second type of SD task is often mentioned with the acronym SDQC,
by referring to the four categories 5 exploited for indicating the attitude of a
message with respect to a rumour [
Although stance detection is an NLP problem still in its emerging stage,
within these competitions but not uniquely there has a considerable body of
conducted research on the topic, exploring a great variety of methodologies. In
the most part, it is a common practice to exploit various classiers and compare
their results. According to recent related work, the most employed techniques
for detecting stance are: rule-based algorithms; supervised algorithms like SVM,
nave Bayes, boosting, decision tree and random forest, Hidden Markov
Models (HMM) and Conditional Random Fields (CRF); graph algorithms such as
MaxCut, and other approaches such as Integer Linear Programming (ILP) and
Probabilistic Soft Logic (PSL) [
based on deep learning, mostly exploit word embeddings, such as word2vec and
fastText as additional features [
In this paper we combine insights gained from some previous work of two
dierent research groups 6 [
The aim of VaxxStance @ IberLEF 2021 [
(a) Textual
(b) Contextual
2. Open Track
3. Zero-Shot Track
For more information regarding the details of each track and setting, please refer
to Agerri et al. [
Our team WordUp! is composed by four researchers that have already dealt
with SD in their previous work. In particular, this work is a joint research with
some of the organizers of the SardiStance @ EVALITA shared task [
SardiStance and VaxxStance present some similarities. Indeed, SardiStance
explored for the rst time the setting of contextual stance detection in
Italian tweets with the addition of information on the tweet itself (e.g., the
number of retweets, the number of favors and the date of posting) and contextual
information about the author (e.g., as follower count, location, user’s
biography, and their social media network). Furthermore, some of the authors have
acquired experience in predicting stance in Spanish tweets participating in the
StanceCat shared task at IberEval 2017 [
In the sections below we list, in the most accurate way possible, all the diverse features that have been implemented for this work. For each feature, we also propose an acronym to be used later in the description of the 16 submitted systems. In fact, not all the features listed below have been ultimately employed in the submissions, but the best features for each track (and setting) were rather selected on the basis of the results obtained performing a 5-fold cross-validation on the training set (see Table 3 for details on the submitted systems). 2.1
The rst type of features we propose are commonly used in sentiment analysis [33]. They are based on the bag-of-words model, and they are namely: Bag of Words (BoW): a binary feature selecting 1-3 word n-grams of the textual content of the tweet.
Bag of Chars (BoC): a binary feature selecting 3-5 character n-grams of the textual content of the tweet.
We also propose some features that try to capture the author’s style of writing.
They include the use of punctuation marks, the recourse to uppercase words
(commonly used in social media for shouting) [
Punctuation Marks (PM): a 6-dimensional feature that includes the frequency of exclamation marks, question marks, periods, commas, semicolons, and nally the sum of all the punctuation marks mentioned before. Uppercase Words ( UpW): this 4-dimensional feature refers to the amount of upper-cased words of at least two chars, the number of words starting with a capital letter, the number of lowercase words containing at least two uppercase characters, and the ratio between uppercase and lowercase words. Laughter (Lau): a binary 1-dimensional feature that checks the presence of laughter (e.g., ahahah or jajaja).
Percentage Numbers ( PN): a 5-dimensional feature that counts the number of percent sign (%), real percentage numbers, of real percentage numbers greater than 50%, of real percentage numbers lower than 50%, and of real percentage numbers greater than 90%.
The last stylistic feature we propose estimates the values of lexical complexity,
which might be related to the level of education of a user and its prociency in
writing [
Lexical Complexity ( LC): a 9-dimensional feature measuring lexical diversity
(3 features) and readability (6 features) as proxies for textual complexity.
We selected among several metrics [
The second type of features that we implemented is based on lexical resources.
Relying on positive results obtained in previous research in SD on other
languages such as English [
Open Cue Words (OpenCW): a binary representation counting the
presence/absence of words related to the following categories: belief, denial,
doubt, fake, knowledge, negation, question, report [
The organizers released some metadata about the tweets and their authors in addition to the textual content of the tweet itself. A Tweet_object , derived form Twitter’s APIs, has a long list of attributes, including fundamental ones such as id, created_at, and text. It also includes the User_object of the author that contains, in turn, other attributes such as created_at, follower_count, and statuses_count . We thus propose the following features for representing a tweet as a numerical vector:
Closed Tweet’s info ( CloseTinfo): a 4-dimensional feature that takes into account the number of retweets and favorites that the tweet received, and the year, the month, and the hour of publication.
Closed Tweet source tag ( CloseTSource): a one-hot encoding representation of the source used for posting the tweet (e.g., Android, iOS). Closed User’s info ( CloseUinfo): a 7-dimensional feature that represents a user from the number of statuses posted, the number of followers and friends, the number of lists which the user is a member of, the year and month in which the account was created, and nally, the ratio of tweets posted per day by the user.
With the aim of exploring data augmentation, we also recovered the description eld from the User_objects employing the Twitter’s API GET users/show 8. This attribute contains a string that is used by the account for describing himself. We propose the following feature:
Augmented Bag of Description ( ABoD): a binary feature selecting the word 1-grams of the textual content of the user’s description (bio). 2.4
Word embeddings are a type of word representation, based on distributional semantic theories, that allows words with similar meaning to have a similar vector representation. Despite we do not propose features exclusively based on word embeddings, we use them for rening some of the best features that we describe in the next paragraphs. To the best of our knowledge, a word embedding trained specically on tweets for Spanish or Basque is not publicly available. Therefore, we decided to create ourselves two word embeddings models for both languages.
First, we collected about 907,000 tweets in Spanish and 853,000 tweets in
Basque from January 2018 until April 2021 using Twitter’s Academic Full Search
API9. In order to obtain a random sample, we split one year in 1,460
timestamps (4365) t spaced from each other by 6 hours. Then, we request 10 tweets
(max_results ) for each timestamp t shifting the query parameter end_time of
a random value between -6 and +6 hours from t. We employ the lang and *
operators for retrieving tweets in each language. Using the same method, but
ltering with the words vacun* and txert* ( vaccine, respectively in Spanish
and Basque), we then collected about 894,000 random tweets in Spanish and
12,000 random tweets in Basque. We also collected about 2,000,000 Facebook’s
messages in Spanish and 2,700 in Basque [
The whole two corpora (composed by Twitter’s posts, Facebook’s messages, and Wikipedia’s pages) have been used for training two word embeddings models of size 100 (es_embeddings and eu_embeddings) employing fastText (the module we used is included in the python’s library Gensim ). We chose fastText because it allows to query for words that do not appear in the training data. This characteristic is very useful in social media domains, in order to represent unknown hashtags that are composed of known substrings. 8 https://developer.twitter.com/en/docs/twitter-api/v1/accounts-and-user s/follow-search-get-users/api-reference/get-users-show 9 https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-re ference/get-tweets-search-all. 10 https://help.crowdtangle.com/en/articles/4302208-crowdtangle-for-acade mics-and-researchers. 2.5
The availability of morphological and syntactic knowledge is crucial for
engineering the last group of features, which relies mainly on dependency syntax,
encoded through the format of Universal Dependencies 11 (UD). Therefore, to
obtain a UD representation of the texts of both training and test set, we apply
the UDPipe pipeline (for tokenization, PoS-tagging and parsing) to them. 12 For
doing so, we train two dierent models (one for Basque and one for Spanish) on
all the available treebanks for those two languages: AnCora [38], GSD [
mark cop det amod advcl advmod aux root
punct obj
advmod punct punct
The features we design on the availability of morphology and dependency syntax are then following: upos (upos): a binary feature selecting PoS tags and creating a bag of 13 PoS n-grams. For instance: [ adp, aux, det, adj, noun, adv, verb, punct...] deprelneg (drN): we consider the presence of negation in the text, relying on the morphosyntactic cues present in the UD format. When a negation was present, we append the correspondent dependency relation in the feature vector. For instance in Figure 1, we spot a negation in [... primera vacunacin no ha llorado ...], the dependency relation of no is advmod, therefore, we append it in the feature vector; 11 https://universaldependencies.org/. 12 See: http://ufal.mff.cuni.cz/udpipe and the Pyhton library: https://pypi.org /project/spacy-udpipe/. id token lemma
upos xpos feats deprel (dr): we build a bag of words of 5-grams, 6-grams and 7-grams of dependency relations as occurring in the linear order of the sentence from left to right; e.g. [mark cop det amod advcl advmod aux root punct advmod punct obj punct] ; relationformVERB (rV): we create a feature vector with all the tuples of tokens that are connected with a dependency distance = 1, by starting from a verb and at the same time we blank the verb itself. For instance, in the example a verb is llorado and some of the tuples of tokens connected through this verb are, e.g. [noVERBnada, noVERBha, noVERB., haVERBno, haVERBnada...] ; relationformNOUN (rN): we apply the same procedure of the feature above but considering nouns as starting points for collecting tuples; relationformADJ (rA): in the same fashion of the two features above, we repeat the same procedure for adjectives too; Sidorovbigramsform (sF): we create a bag of word-forms (tokens), considering the 2-grams that can be collected following the syntactic tree structure (rather than the 2-grams that can be collected reading the sentence from left to right).13 Such that: e.g. [llorado ha, llorado no, llorado vacunacin, llorado nada, ... vacunacin para, vacunacin ser, ..., nada casi, ...] ; Sidorovbigramsupostag ( sT): as the feature above, we create a bag of partof-speech tags following the syntactic tree structure, starting from the root; 13 Please refer to [36] and [35] for more details on this regard.
Sidorovbigramsdeprel ( sDR): as the two features above, we creat a bag of words based on dependency relations ( deprels ) following the syntactic tree structure, starting from the root.
We then propose two additional features that represent a tweet by using the dependency relations between the target of interest (for instance, the noun vaccine or verb vaccination), in both Basque and Spanish, and the connected tokens in the dependency tree.
Target Context Level 1 ( TC1): this 200-dimensional feature consists in concatenating the word embeddings representations of the previous and next with respect to the target in the dependency tree.
Target Context Level 2 ( TC2): this 400-dimensional feature integrates the feature TC1 including the word embeddings representations of the second level of previous and next words connected to the target in the dependency tree.
‘Vaccines do not cause autism’ Figure 2 shows the dependency tree of the sentence Las vacunas no provocan autismo (Vaccines do not cause autism). In this sentence, the target is the noun vacunas (vaccines). We can observe that the target has a dependency relation of type det (determiner) with the denite article las (the) and a relation of type nsubj (nominal subject) with the verb provocan ([they] cause). Therefore, the feature TC1 consists in the concatenation of the word embeddings representation of the words las and provocan.
concatenate([ model(‘las’), model(‘provocan’) ]) Where the function model(X) (es_embeddings or eu_embeddings, depending on the language) returns the word embeddings representation of size 100 of the word X.
Then, the feature TC2 includes the word embeddings representations of the second level of previous and next words connected to the target in the dependency tree. In this example there are no words related with the denite article las, but there are two words no (no) and autismo (autism) are connected with the verb provocan. We ll the feature with zeros when a related word is missing, and we average the word embeddings when two or more words exist at the same level of the dependency tree. The feature TC2 represents the example as follows: concatenate([ 100*[0], model(‘las’), model(‘provocan’), average([model(‘no’), model(‘autismo’)]) ]) The two features are padded with zeros when the target is not mentioned in the text and we averaged the word embeddings representations when the target is mentioned two or more times. 2.6
A peculiarity of this NLP task is its willingness to explore the interplay of online social networks and users’ stance. Indeed, the organizers released the user_id of every user the author of the tweet is following 14 (otherwise known as their friend) and the user_id of every original tweet the author retweeted in their User_timelines .15 We then create two friends network directed graphs - one for each language - where the nodes are users and an edge between two users exists if one follows the other. Similarly, we create two retweets network directed graphs where the nodes are users and an edges between two users exists if one retweeted the other.
With the aim of exploring data augmentation - taking advantage of Twitter’s Academic Full Search API 9 - we also retrieve the list of user_id retweeted by the author and the list of user_id that retweeted the author. 16 We obtain two augmented retweets network directed graphs - one for each language where the nodes are users and an edge between two users exists if one retweeted the other. The order (number of nodes) and the size (number of edges) of the two networks are shown in Table 2.
14 https://developer.twitter.com/en/docs/twitter-api/v1/accounts-and-user s/follow-search-get-users/api-reference/get-friends-ids . 15 https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/ api-reference/get-statuses-user_timeline . 16 We retrieved additional data from January 1st, 2019 until May 1st, 2021. Network data allow us to propose three types of features that respectively measure the centrality of each node, the distance between all pairs of them, and the mixing of the networks on stance towards vaccination (i.e., assortative mixing).
The rst type of network feature is based on node-level indices focusing on
the relevance of nodes. Each user represents a node in the network and the ties
between them are of two types: friendship (or following/followed) and
retweeting. Among several measures of network centrality available, we consider:
indegree, outdegree, Kleinberg’s hub and authority scores, closeness, betweenness,
eigenvector, Bonacich’s power, and Google PageRank [
Network Retweet Centralities ( NetRC): a 4-dimensional feature measuring user centrality for the retweets network (Spanish indices: indegree, outdegree, hub, and authority; Basque indices: outdegree, authority, closeness, and eigenvector).
Network Augmented Retweet Centralities ( ANetRC): a 4-dimensional feature measuring user centrality for the augmented retweets network , exploiting the same indices selected for the Spanish and Basque Retweets networks from the train and test set ( NetRC).
NetFC and NetRC features are computed by combining the graphs of the train and test sets. In case some users are disconnected in the networks’ graphs of friends and/or retweets, they are assigned a value of 0 for each of the computed centrality indices.
The second type of network feature is based on the distances among users in
the networks. For each of them, a distance matrix among subjects is computed.
The distance is dened as the shortest path, forcing the graph to be undirected.
The Distance Matrix is then projected into a euclidean space trough a
Multidimensional Scaling (MDS) [
The third type of network feature is based on the homophily principle which
states that social networks contacts among similar users occur at a higher rate
than among dissimilar ones [
Friends’ Stance (NetFS): this feature counts the number of friendship relations among the author of the tweet and the other users, grouping them by stance, in the friends network.
Retweets’ Stance ( NetRS): this feature counts the number of retweet relations among the author of the tweet and the other users, grouping them by stance, in the retweets network.
Augmented Retweets’ Stance ( ANetRS): this feature exploits the augmented retweets’ network for counting the number of retweet relations, in the augmented retweets network. 3
The organizers provided a training dataset of 2,400 tweets written in Spanish and 1,073 in Basque labeled with stance ( against, favor, and neutral) toward the topic of the Antivaxxers movement . The dataset also include additional information on the tweet itself, and contextual information about the authors (1,261 for Spanish, and 149 for Basque). The last contextual information available consists in the relations of friends and retweets among the authors and the other Twitter’s users. The organizers have also made available the Home_timeline of the Basque authors, but we did not employ this information in any feature. As mentioned earlier, the task is divided in four dierent tracks that establish dierent constraints on the use of data and external resources.
We nally submitted 16 runs (2 for each track and for each language) choosing among the best combination of features that we retrieve by performing a feature selection. We employed a 5-fold cross validation over the train set via Logistic Regression.
Table 3 shows the combination of features submitted for each run. We include the
F1 macro-average score of two classes favor and against achieved with 5-fold
cross validation over the training set. 17 The rst thing that catches the eye is that
three runs for Basque reach about 98 F1 macro-average (in the Close-Contextual,
Open, and Zero-Shot tracks). A possible explanation is to be encountered in the
fact that the social network based on friendship relations exhibits homophily by
stance. In fact, users tend to follow people who have similar opinions to theirs
[
Another feature that seems to be very promising is ABoD, which proles users by their descriptions. Indeed, the description often includes users’ professions and interests that are very useful for predicting stance towards vaccinations (e.g., some of them are doctors and nurses). Example of a user description:
(Nurse.
) In the example above the user identies herself as a nurse, using also the emoji identied by the shortcode :woman_health_worker: that combines the woman emoji ( ) and the medical symbol emoji ( ). In future work it would be interesting to exploit the emojis contained in user descriptions as feature, as well as word-embeddings representations in order to capture all the professions that are in any kind of relation to medical/health eld.
Although using contextual features allows us to achieve high level of averaged F1 macro, also using Stylistic and Dependency-based features helps to improve results in both Basque and Spanish. Our ndings, in fact, provide a meaningful support to the hypothesis that morphosyntactic knowledge extracted from treebanks can be usefully exploited for addressing the stance detection task. In particular, they pave the way for a further investigation where the combination of a dependency-based syntactic approach and state-of-the-art neural models can be explored. Thanks to dependency syntax it is possible to grasp connections among words that are not captured by n-grams or word embeddings standing alone. If we had only considered approaches such as those that take into account only the words that are in the immediate proximity of each other, the deeper pragmatic meaning of a sentence, might have been lost. Indeed, with dependency-based features is possible to capture the information in which words are syntactically related to each other also if they have a long-distance relation. 17 All values in the result tables have been multiplied by 100 to enhance readability and to be consistent with ocial rankings provided by the organizers (see Table 4). In order to assess the performance of the participating systems, a test set of 694 and 312 unlabeled tweets were provided respectively for Spanish and Basque. The four tracks have been evaluated separately for each language. Three teams participated to Close-Textual and Close-Contextual tracks and we ranked as the rst position in both sub-tasks for Basque and Spanish. The dierence of our results from those of other teams is particularly evident. 18 Furthermore, our team is the only one that participated to the Open and Zero-shot tracks. Table 4 shows the ocial results, as provided on the task website, sorted by track and language setting.
If we compare the scores of Table 3 with the scores of Table 4, there is a general drop in performance, with the exception of Close (textual and contextual) and Open Track for Spanish. Indeed, ocial results are higher than experimental ones in these cases. We can see right away that the run Close-Contextual eu_01 obtains a very low result (0.04 F1 Macro).
The features used in this particular track and setting are: Lau, PN, NetFC, NetFS, NetFM and they seem not to be relevant at all. The features are mostly based on the network of users, but the authors of the tweets belonging to the test set are not very connected to the authors of the train set. For this reason the system that uses only Network -based features is not able to detect a stance and tends to associate none labels. In any case, Network -based features, employed in addition to Stylistic and Dependency -based ones, reach 77.71 and 89.13 F1 macro-average respectively for Basque and Spanish. These results overcome those achieved in the Close-Textual track (57.34 and 80.92 F1 macro-average respectively for Basque and Spanish). The same situation can be observed in the Spanish setting: the use of contextual features (in particular the Network -based 18 For the complete rankings refer to: https://vaxxstance.github.io/#results . ones) improves the performance of our systems. The are no clear improvements in performance exploring data augmentation in the Open Track for both Spanish and Basque. 4
In order to gain more insights on the performance of our models we carry out two types of error analysis. On the one hand we look the tweets that have been misclassied and we observe them linguistically. On the other hand we carry out a test on the process of feature selection with an automatic tool. In this subsection we compare the two outputs predicted with the two best performing models in Spanish (Close-Textual es_01 and Close-Contextual es_02). We focus only on the Spanish language, because, regrettably none of the authors is uent in Basque.
We investigate the predictions especially of those two models because the they dier one from the other only by the employment of one feature. The CloseTextual es_01 model, in fact, employs the following features: Bag of Words, Bag of Characters, Punctuation Marks , Uppercase Words, Laughter, Percentage Numbers, deprelNeg, deprel, relationVERBS, Sidorovbigramsform and Target Context Level 2. The Close-Contextual es_02 model employs the same features as the other system, plus the addition of Network Friends’ Stance (NetFS). Due to this reason, we believe their comparison might lead to interesting discoveries. In fact, by the single addition of one last feature, results are boosted up +10 points in terms of F1 Macro in Spanish (from 79.06 to 89.13, see Table 4). The same feature (together with NetFM ), tested on the Basque dataset, induces a boost of +20 points of Macro F1 (from 57.34 to 77.71, see Table 4).
In Table 5 we report the confusion matrices of the labels predicted by the two models compared against the gold test set.
d l o g against favor none
predicted against favor none 105 22 13 21 294 44 12 37 146 d l o g against favor none
predicted against favor none 129 5 6 7 307 45 7 34 154 (a) Close-Textual (es_01).
(b) Close-Contextual (es_02).
From Table 5, if we compare the left table (a) with the right table (b), it can be
seen that both precision and recall increase for the class against. The results
suggest that the Close-Contextual model, beneting from the NetFS feature,
shows an improvement in the detection of highly polarized labels ( against vs.
favor), while on the other hand, it continues to fail to successfully discern
between less polarized choices (e.g., favor vs. none or against vs. none).
Furthermore, the Network Friends’ Stance feature is highly procient in detecting
correctly especially the tweets that are labeled as against. This highlights the
fact that authors that follow (and are followed) by other users who are skeptical
about vaccinations, tend to be against the vaccines. Similar ndings within
polarized debates have been found also by Lai et al. [
Additionally, we observe that 12 tweets that have been correctly predicted by the Close-Textual es_01 model were misclassied by the Close-Contextual es_02. On the opposite end, there are other 57 tweets that were misclassied from the Close-Textual es_01 model, but that were classied correctly by the Close-Contextual es_02 model. This procedure might lead to better understanding and plausible explanations of the usefulness of the NetFS feature.
Furthermore, we observed the impact of the Network Friends’ Stance feature and its contribution in stance detection. We recall that such feature counts for each user the number of directed friendship relations ( in or out) grouped by stance. For instance, the tweet below has been wrongly classied in the Textual Track (against), but was labeled correctly in the Contextual Track ( favor): Cuando tienes la sensacin de que la sociedad va hacia atrÆs en vez de avanzar, y todo por las malas decisiones del ser humano...Ni a sus hijos ni a sus perros: el nuevo peligro del movimiento antivacunas son las mascotas https://t.co/AvLXpf8G91 en @elpais_espana (When you have the feeling that society is going backwards instead of moving forward, and all because of the bad decisions of human beings...Not their children or their dogs: the new danger of the anti-vaccine movement is pets https://t.co/AvLXpf8G91 in @elpais_espana) gold: favor close-textual es_01: against close-contextual es_02: favor Interestingly, we observed 7 friendship relationships with users labeled as against (3 in, 4 out) versus 12 friendship relationships with users labeled as favor (4 in, 8 out). Here, the use of the NetFS feature contributed to the correct classication of the tweet.
Although, using this approach might lead also to the opposite deduction. In fact, some tweets have been correctly classied by the Textual model, but were wrongly labeled by the Contextual model. For instance, in the following tweet, the NetFS feature pointed the Contextual Model towards the label against, since it observed 21 friendship relationships with users labeled as against (10 in, 11 out) versus 12 friendship relationships with users labeled as favor (5 in, 7 out):
Un alcalde y lder de las Juventudes del PSOE de 29 aaeos en Valencia se vacun del coronavirus el primer da Rojos haciendo cosas de rojos...#ADisfrutarDelPucherazo https://t.co/Dllq3pRVAl ( A 29-year-old mayor and leader of the PSOE Youth in Valencia was vaccinated against the coronavirus on the rst day Reds doing red things...#LetsEnjoyThePout https://t.co/Dllq3pRVAl ) gold: none close-textual es_01: none close-contextual es_02: against For the author of the tweet above we did not detect any relationships with users labeled as none, hence we can speculate that the NetFS feature led to a misclassication. 4.2
In this subsection, we aim at including some analysis on both languages, since unfortunately Basque was overlooked in the previous manual analysis. We exploit a univariate feature selection by sampling the best 100 features based on the ANOVA F-value for the test set.
According to the ANOVA test, the best features exploited in the run
CloseTextual es_01 mainly include n-grams and char-grams belonging to the BoW
and BoC groups of features. If on the one hand we encounter n-grams such
as somoslaresistencia (we are the resistance) and vacuna contra (against
vaccines), on the other hand, we nd several features attributable to the presence
of URLs in the text. This conrms similar ndings from earlier studies [
We also nd similar results inspecting the highest scoring features in the runs submitted for Basque. The best 100 features of the run Close-Textual eu_01 includes n-grams such us zientzia (science) and char-grams attributable to the word txertoa (vaccine) belonging to the BoW and BoC groups of features. We also nd several n-grams and char-grams features attributable to the presence of URLs in the text, similarly to what was observed in the analysis of the run for Spanish. The Close-Contextual eu_02 run diers from Close-Textual eu_01 only for the presence of the Network Friends’ Stance (NetFS) and Network Friend MDS (NetFM) features. The features based on social media networks prove to be also relevant in Basque, in particular the best performing features include 5 out of 6 NetFS features (noneout, againstin, againstout, favorin, and favorout) and 2 out of 3 dimensions of the NetFM feature. 5
In this paper we presented an overview of the WordUp! submission for the VaxxStance task at IberLEF 2021. We participated in all the proposed tracks by submitting 16 dierent runs in the detection of author’s stance towards the target ‘vaccines ’ for tweets in Basque and Spanish. Our approach, mainly employing Stylistic, Dependency -Based and Contextual features, proved to be highly successful concerning the task of stance in both languages. We ranked as the rst position among three participating teams in all tracks in both languages. The results show that the addition of contextual features such as Network -Based ones, produced a signicant contribution to the stance detection task. For instance, the exploitation of the Network Friends’ Stance feature (NetFS) induced a boost of +10 points in terms of F1 Macro for Spanish, and together with NetFM +20 points for Basque. Therefore, we might interpret that contextual information and the network of users are the richest exploitable information in this SD task, and, that the combination of linguistic information with contextual features leads to more explainable results.
In the future, we plan to tailor the ABoD feature (Augmented Bag of Description) exploring in an even ner grained manner the content of the user’s description. We also aim at exploring the contribution of the Dependency -based features for predicting stance in an unsupervised framework. 6
The code, the models and the resources used in this study are freely available online, allowing for an easy replication of the presented results. They can be found in the following repository: https://github.com/mirkolai/WordUp .
? Livio Finos: Implementation of Network-based Features, Methodology, Formal Analysis, User Labeling, Writing. ? Andrea Sciandra: Data Augmentation, Implementation of Network-based Features and Lexical Complexity Features, Formal Analysis, Error Analysis, Writing.
The work of the rst two authors is partially funded by the project (under the 2019 Google.org Impact Challenge on Safety call).
53(1) 30. Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., Cherry, C.: SemEval-2016 Task 6: Detecting Stance in Tweets. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). ACL (2016) 31. Mohammad, S.M., Sobhani, P., Kiritchenko, S.: Stance and sentiment in tweets.
ACM Transactions on Internet Technology (TOIT) 17(3) (2017) 32. Montes, M., Rosso, P., Gonzalo, J., Aragn, E., Agerri, R., `lvarez Carmona, M.A., `lvarez Mellado, E., Carrillo-de Albornoz, J., Chiruzzo, L., Freitas, L., Gmez Adorno, H., GutiØrrez, Y., JimØnez Zafra, S.M., Lima, S., Plaza-de Arco, F.M., TaulØ, M.: In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021). CEUR-WS.org (2021) 33. Pang, B., Lee, L.: Opinion mining and sentiment analysis. Found. Trends Inf. Retr.
2(1-2) (2008) 34. Sciandra, A., Surian, A., Finos, L.: Supervised Machine Learning Methods to Disclose Action and Information in U.N. 2030 Agenda Social Media Data. Social Indicators Research (2020) 35. Sidorov, G.: Should syntactic n-grams contain names of syntactic relations?. International Journal of Computational Linguistics Applications 5(2) (2014) 36. Sidorov, G., Velasquez, F., Stamatatos, E., Gelbukh, A., Chanona-HernÆndez, L.: Syntactic dependency-based n-grams: More evidence of usefulness in classication. In: International Conference on Intelligent Text Processing and Computational Linguistics. pp. 1324. Springer (2013) 37. TaulØ, M., Mart, M.A., Pardo, F.M.R., Rosso, P., Bosco, C., Patti, V.: Overview of the Task on Stance and Gender Detection in Tweets on Catalan Independence. In: Proceedings of the 2nd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017) co-located with 33th Conference of the Spanish Society for Natural Language Processing (SEPLN 2017). CEUR-WS.org (2017) 38. TaulØ, M., Mart, M.A., Recasens, M.: AnCora: Multilevel Annotated Corpora for Catalan and Spanish. In: Proceedings of the Sixth Language Resources and Evaluation Conference (LREC 2008). European Language Resources Association (ELRA) (2008) 39. TaulØ, M., Pardo, F.M.R., Mart, M.A., Rosso, P.: Overview of the Task on Multimodal Stance Detection in Tweets on Catalan #1Oct Referendum. In: Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018) co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018). CEUR-WS.org (2018) 40. Wei, W., Zhang, X., Liu, X., Chen, W., Wang, T.: pkudblab at SemEval-2016 Task 6 : A Specic Convolutional NeuralNetwork System for Eective Stance Detection. In: Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016) (2016) 41. Zarrella, G., Marsh, A.: Mitre at Semeval-2016 Task 6: Transfer Learning for Stance
Detection. arXiv preprint arXiv:1606.03784 (2016) 42. Zubiaga, A., Aker, A., Bontcheva, K., Liakata, M., Procter, R.: Detection and Resolution of Rumours in Social Media: A Survey. ACM Computing Surveys 51(2), 32 (2018)