In this paper we describe the iTACOS submission for the Stance and Gender Detection in Tweets on Catalan Independence shared task. Concerning the detection of stance, we ranked as the rst position in both languages outperforming the baselines; while in gender detection we ranked as fourth and third for Catalan and Spanish. Our approach is based on three diverse groups of features: stylistic, structural and context-based. We introduced two novel features that exploit signi cant characteristics conveyed by the presence of Twitter marks and URLs. The results of our experiments are promising and will lead to future tailoring of these two features in a ner grained manner.
Recently, there is a special interest in the task of monitoring people's stance
towards particular targets; thus leading to the creation of a novel area of
investigation named Stance Detection (SD). Research on this topic could have a
positive impact on di erent aspects such as public administration, policy-making,
and security. In fact, through the constant monitoring of people's opinion,
desires, complaints and beliefs on political agenda or public services, administrators
could better meet population's needs. For example, a practical application of SD
could improve the automatic identi cation of people's extremist tendencies (i.e.
religious extremism [
In 2016, for the rst time a shared task on SD has been held at SemEval-2016,
namely the task 6: Detecting Stance in Tweets3 was organized in the framework
of SemEval. The participating teams were required to determine stance towards
six di erent targets: \Atheism", \Climate Change is a Real Concern", \Donald
Trump", \Feminist Movement", \Hillary Clinton", and \Legalization of
Abortion". Most of the proposed approaches exploited standard text classi cation
features such as n-grams as well as word embeddings. More details about the
participating systems can be found in [
In this paper we present our participation to the Stance and Gender
Detection in Tweets on Catalan Independence task [
Inferring people's traits such as gender, age or native language on the basis
of their written texts is investigated by a eld named Author Pro ling (AP).
From 2013 onwards a shared task on AP has been organized at PAN [
The starting point of our proposal is to be found in the method proposed in
Lai et al. [
Therefore, we propose a supervised approach which consists in determining
stance towards the independence of Catalonia as well as the gender of the author
of a given tweet. In our work, we explored some features that can be grouped
in three main categories: Stylistic, Structural, and Context. In the present paper
we were not able to explore Sentiment features as in [
Bag of Words (BoW )6 Bag of Part-of-Speech labels (BoP )6;7 Bag of Lemmas (BoL)6;7
Bag of Char-grams (BoC )8 Structural Features 4http://stel.ub.edu/Stance-IberEval2017/ 5http://clef2017.clef-initiative.eu/ 6Each tweet was pre-processed for converting it to lowercase. We used unigrams, bigrams and trigrams with a binary representation.
7We used TreeTagger [
Bag of Hashtags (BoH ). We consider the hashtags as terms for building a vector with binary representation.
Frequency of Hashtags (freqHash).
Uppercase Words (UpW ). This feature refers to the amount of words starting with a capital letter.
Punctuation Marks (PM ). We take into account the frequency of dot, comma, semicolon, exclamation and question marks.
Length (Length). Three di erent features were considered to build a vector: number of words, number of characters, and the average of the length of the words in each tweet.
Language (Lan). We create a vector exploiting the labels es for Spanish and ca for Catalan provided by the organizer.
URL (Url ). We observed that tweets containing a URL are common in the training dataset. We decided to take advantage of this by considering di erent aspects extracted from short URLs. First, we identi ed if the web address of reference is or not reachable. Second, we retrieved the words contained on the web address, then we build a bag-of-words using this information. 3
The organizers provided a dataset of 8,638 tweets written in Spanish and Catalan
labelled with stance (against, favor, and neutral) and gender (female and
male). For what concerns gender, the distribution is balanced among female
and male tweets. Regarding stance, the distribution is skew towards favor
for Catalan and skew towards neutral for Spanish (respectively 30.66% and
29.38%). Similar trends were found in Bosco et al. [
It appears, therefore, that language could be a useful feature for stance
detection in the Catalan independence debate concerning a region characterized by
a strong bilingualism and a smoldering nationalism. In fact, Language divides
and unites us. It [...] impinges upon our identity as individuals, as members of
a particular ethnic or national group, and as citizens of a given polity [
In order to assess the performance of the participating systems, a test set of 2,162 unlabelled tweets was provided, and the two tasks were evaluated separately. Two di erent evaluation metrics were used: (1) the macro-average of F-score (favor and against) was used in the case of stance detection and (2) the accuracy was selected as metric to evaluate the performance in terms of gender identi cation. 3.1
iTACOS experiments
In our experiments, we addressed both stance and gender detection as a
classi cation task. The code is available on github for further exploration and for
allow reproducibility of our experiments9. We carried out several experiments10
by combining both the features introduced in Section 2 together with a set of
classi ers composed by: Support Vector Machine (SVM), Random Forest (RF),
Logistic Regression (LG), Decision Tree (DT), and Multinomial Nave Bayes
(MNB). Besides, we exploited a Majority Voting (MV) strategy considering the
di erent predictions of the above mentioned classi ers as described in Liakata
et al. [
We participated in the shared task with ve di erent runs for each language and each subtask. Table 2 shows the obtained results by using both the features and the classi er used in each of the submitted runs. 3.2
O
cial results We ranked as the rst position among 10 participating teams in the subtask of stance detection in both Catalan and Spanish. Table 3 shows the o cial results on the test set. At a rst glance, it is possible to observe that our proposed approach seems to perform slightly better in Catalan than in Spanish. Overall, our submissions performed better in Catalan, in fact our ve runs ranked among the rst 8 positions. In Spanish, on the other hand, our less performing run ranked as the 18th position.
As shown in the table above, the best result in each language was not achieved by the same run. iTACOS.2 performs better for Catalan, while iTACOS.1 for 9https://github.com/mirkolai/iTACOS-at-IberEval2017 10A 10-fold cross-validation setting was used. Spanish. The poorer results in both languages were obtained by using iTACOS.4 and iTACOS.5. As expected the best performing runs (iTACOS.1 and iTACOS.2) contain both context-based features, validating the importance of considering contextual information in stance detection tasks. For example, both runs include the feature Url. We are interested in evaluating the impact of such feature on the performance. For this reason, we carried out experiments on the training set by applying a modi ed version of iTACOS.1 and iTACOS.2 removing the Url feature. Looking at the results, we observed a drop in the performance of -0.029% for Catalan and of -0.002% for Spanish in iTACOS.1; and of 0,004% for Catalan and of -0.002% for Spanish in iTACOS.2.
The BoTM, a novel feature included in the structure-based group, emerges among the relevant features in iTACOS.1 concerning Spanish, but further inquiry on its relevance is matter of future work. For what concerns classi ers, LG and SVM achieved the best performance in both languages. Surprisingly, the approach exploiting MV is not performing. 3.3
A fundamental part of our approach has been that of manually dealing with data. Being the size of the dataset very large, we were able to visualize only a small portion of tweets. Therefore, we focused on the cases of disagreement between the results obtained with iTACOS.1 and the golden labels provided by the organizers11. Below, we report some examples both in Catalan and Spanish: 1. #elecciones #catalunya #NO #27S https://t.co/oBuTDnUEHj ! #elecciones #catalunya #NO #27S https://t.co/oBuTDnUEHj language: catalan golden label: against iTACOS.1: favor 2. Ale @JuntsPelSi, a casa, son solo unas #eleccionescatalanas autonomicas. Man~ana a trabajar que es lunes. Segu s teniendo el mismo DNI. #27S ! @JuntsPelSi, go at home, there is only one autonomous #eleccionescatalanas. Tomorrow, go to work that it'll be Monday. You will have the same DNI (Spanish ID). #27S language: spanish golden label: against iTACOS.1: favor 3. En estas #eleccionescatalanas de decide una posible independencia y un gobierno que vele por los derechos de su pueblo, VOTA @catsiqueespot ! In these #eleccionescatalanas we decide for a possible independence and a government that ghts for the rights of its population, VOTE @catsiqueespot language: spanish golden label: favor iTACOS.1: against Example 1, has been marked as favor from our classi er in (iTACOS.1), probably because of the misleading presence of the token \catalunya", written in Catalan. However, the explicit semantic information carried by the hashtag #NO pointing to against was ignored, thus leading to a wrong classi cation. Considering Spanish, example 2 has been appointed as favor instead of against. The presence of the mention @JuntsPelSi (Catalan independence coalition) could have misdirected our classi cation. On the other hand, the tweet in example 3 was tagged as against whereas it should have been favor as we clearly infer from \VOTA @catsiqueespot" and according to the golden labels.
A manual analysis of this kind helped us to shed some light on the relevance of each single feature we exploited and, after having linguistically analyzed them, to choose which features had to be included in our nal sets. 4
In this paper we presented an overview of the iTACOS submission for the Stance and Gender Detection in Tweets on Catalan Independence task at IberEval-2017. We participated by submitting ve di erent runs in the detection of author's stance and gender both in Twitter messages in Catalan and Spanish. Our approach, chie y based on context and structural features, proved to be highly 11The tweets have been extracted from the training set. successful concerning the task of stance in both languages, as our system ranked as the rst position among ten participating teams. The results show that the addition of two particular features, namely BoTM and Url, produced a signi cant contribution to Stance Detection task. In the future, we plan to tailor these two features we used in an even ner grained manner.