This paper describes the process that we followed to develop our stance analysis tool for IberEval 2018 on MultiModal Stance Detection in tweets on Catalan 1Oct Referendum (MultiStanceCat) task. Our approach is based on the tools provided by the scikit-learn toolkit[3] to develop a system capable of detect the stance of some tweets about the Catalan 1Oct Referendum, using only the text written in the body of the tweet and also using the context formed by the previous and the next tweet of the one we are analyzing.
Nowadays, the large amount of new data produced by services like Twitter brings the opportunity to get useful and interesting information about people opinion and feelings on a wide variety of topics. This high volume of information could be di cult or nearly impossible to handle and process by human operators, requiring to make use of advanced algorithms to obtain the relevant information hidden on the data with low time and economic costs.
This information can be used for tasks that improve the service quality and security (among others) o ered by companies, which are becoming more interested on the text classi cation eld.
For the purpose of obtaining the opinion and feelings of the society about popular
topics, Twitter has become an attractive tool, being used for many studies [
For the MultiStanceCat [
In this task, we had to classify a set of tweets related to the Catalan 1Oct Referendum based on the stance (Favor, Against or Neutral) of the person who wrote it, concerning the independence of Catalonia.
The paper is structured as follows. Section 2 describes our system and the preprocessing we made to the dataset. Next, in Section 3, the obtained results are discussed. Finally, we present our conclusions in Section 4 with a summary of our ndings. 2
The dataset is formed by 6 les, distributed in the following way: 2 set of tweets for training, 2 les with the labels for these tweets, and 2 les for test. One of each of those three types of les is for Spanish language and the other for Catalan.
We joined the two train les (Catalan tweets and Spanish tweets) to compose a bigger corpus and shu ed it, to obtain an homogeneous distribution of the tweets, before dividing it in two parts. The rst one, containing 80% of the texts, used for the training phase and the second one, with the remaining 20%, used to validate the analyzer among training iterations.
Our rst approach was to tokenize the tweets to separate the di erent words before passing them to the vectorizer. Then we obtained a features matrix based on a Tf-idf vectorizer, where we can nd a row for each tweet and a column for the weight of each feature, related with the frequency of each token in the tweet and in the corpus. With this features matrix and the tags obtained from the labels les, we trained a LinearSVC classi er and got a model to classify new tweets based on their stance.
Finally, we used the development set to evaluate the classi er, vectorizing the tweets with the same vectorizer used before and classi ed them with the LinearSVC trained with the training set. The results of this approach can be found in the next section (see Table 1).
Using the previous approach as base, we tried to add new features, with the
intention of improving the obtained results. Our rst hypothesis was that we
could use the similarities between Spanish and Catalan to increase the number
MultiModal Stance Detection
3
of occurrences of each feature in the training case and to decrease the quantity
of di erent features that could appear while training or classifying.
Spanish and Catalan languages have many words that share the same stem but
have di erent ending, for example the Spanish word "television" is treated as
di erent from the same Catalan word "televisio" in our rst approximation. Our
intention is to make our analyzer capable to detect that type of situations.
We started using one of the stemmers included in the NLTK toolkit [
After that, we tried a variation of the last experiment, using a xed length of characters at the end of the word to be removed. The results of this experiment with length 1, 2 and 3 are reported in Table 3. This way of processing each word has the disadvantage of having a big impact on short words and an almost negligible one on long words. For this reason we de ned in Table 4 ranges of lengths and a xed quantity of characters to be removed, causing the deletion length to be proportional to the word length.
For the second part of the MultiStanceCat task, we had to study if adding the previous and the next tweets to the current tweet that we are analyzing could help to increase the stance classi cation accuracy. We used additional information in 3 di erent way: 1. Concatenating these three text bodies, adding a space between them (prev + text + next). 2. Concatenating these three text bodies, trying to give more weight to the text of the current tweet, duplicating it in the concatenated text (prev + text + text + next). 3. Vectorizing each of these three tweets in an independent way and joining their features matrix before training Linear SVC.
Results are given in Table 5 4 3
Experiments using both languages at the same time and using them in an independent way have been carried out (Table 1) having obtained better results in the rst case. Using a too short pre x length showed to worsen the obtained results (see Table 2), but no signi cant variations on F1-macro have been seen when trying di erent values for the xed length su x removal approach. The best results have been obtained with the ranged su x removal length, because the length of the removed su x is proportional to the word's length, resulting in a bigger removal on long words and shorter in the short ones.
Type F1-macro precision recall Both languages together 0.69171 0.71692 0.67917 Only Catalan language 0.51662 0.86154 0.46529
Table 1. Only text without stemmer MultiModal Stance Detection 5 When using the context as additional information, better results have been achieved when duplicating the tweet's text, to increase the weight of this information in relation with the context information. As can be seen comparing the previous results with the ones in Table 6 and Table 7 the macro-F's obtained by us for our results were far better than the ones provided by the organization. This happens because we used a di erent way to evaluate the correction of our stance detector, checking the quantity of right classi cations for all three classes, instead of only looking for the positive and negative stances (discarding the neutral one) as done by organizers.
C. Almendros, C. Cervantes
For this task, we have developed two models to classify tweets according to their stance in two similar languages (Spanish and Catalan). Both were trained with a LinearSVC classi er and the di erence was that on the second one we also made use of the context to extract features.
In the rst model (without context) we observed that using the stem of the words, instead of the whole word, improves the accuracy of results, if these stems are long enough. For this reason, one approach that could be tested, is to translate one of the two corpus in order to work with the same language and extract the features on the basis of a single language.
Looking at the results of the second model, it is proved that the use of context is relevant for this task. This can be seen in the table of results (Table 6 and Table 7) where this executions take a Macro-F of 0.3068 versus the 0.2315 obtained without context.
The good results obtained in IberEval denote that the use of this classi er along with the data preprocessing techniques that we tried are a good election for this task, but there is still a wide range of improvements that can be added to increase the accuracy of our analyzer.