In this paper, we describe our participation at the task on MultiModal Stance Detection in Tweets on Catalan 1Oct Referendum. Tweets are cleaned and represented using the simple Bag-of-Words approach with tf-idf vectors. Then, we explore the most widely used and e cient classi ers in text classi cation. Some algorithms are adapted to be multi-class learning by using one-versus-all strategy because they are naturally binary. For each algorithm, we perform grid search on all combinations of its parameters in order to nd the set of parameters which provides the most accurate model. Our system employing text and context obtains the top macro F1 (28.02%) for spanish tweets.
The goal of the stance detection is to determine the stance of the author of a
text with respect to a speci c topic. The stance can take the following values:
favor (positive), against (negative) or neutral [
We performed an exhaustive evaluation of the most widely used and e cient
classi ers in text classi cation. In particular, we used the following algorithms:
Multinomial Naive Bayes [
Multinomial Naive Bayes classi er has been proven very e ective for text classi cation. It is a probabilistic model based on theorem of Bayes. This classi er calculates the probabilities of each text belonging to each class and then selects the class with the maximum probability. The adjective naive comes from the assumption that all features are independent given class. Although such an independence assumption is not usually true, the algorithm often performs surprisingly well with a fast computational time. Moreover, it requires a small amount of training data, is very easy to implement and is also very scalable. Despite its simplicity, the Naive Bayesian classi er often exceeds more sophisticated classi cation algorithms.
SVM, perhaps one of the most popular and successful classi ers, is a
nonprobabilistic linear classi er that tries to nd the hyperplane that best separates
the classes, maximizing the margin between them while, at the same time,
minimizing the number of misclassi cation errors. The main reason of its success
is that most text classi cation problems are linearly separable [
Logistic Regression is a linear classi er, which can be used to predict the probability of an event. Its main advantage is that its results have an easier interpretation than those obtained by other classi cation algorithms. Moreover, this algorithm provides a regularization parameter to avoid over- tting. Among their disadvantages, it requires much more data than other classi ers to obtain stable and accurate results. Moreover, it is not able to capture complex relationships in the data.
k-NN is one of the simplest classi cation algorithm. It is based on the idea that the closer instances are, the more probability they belong to the same class. In this way, one of its main advantages is that it is a lazy classi er because it does not create a training model from the training dataset, but rather compares the test instance with all instances to determine its class. Moreover, the classi er does not depend on the data distribution.
Random forest is an ensemble classi er of a collection of decision trees by randomly selecting examples from the training data. The nal prediction is calculated by aggregating the predictions of each tree. Learning from di erent trees leads to mitigate the over- tting as well as errors due to bias and variance in the decision trees. Random forests are more robust and generally exhibit better results than decision trees.
Figure 1 shows the distribution of the tweets labelled with stance in the training dataset. Most of the tweets written in Catalan are clearly in favor of holding the referendum. However, for the tweets written in Spanish, the stance seems to be distributed more-or-less equally in the three classes. AGAINST and NEUTRAL have a very close number of tweets (above 3,200), while FAVOR is the class with less tweets (around 2,000). This more balanced distribution may help the learning of the algorithms for Spanish tweets, while the task may be more di cult for tweets written in Catalan because the classes are not balanced 4
I. Segura-Bedmar (there are very few instances for the AGAINST class). The training dataset contains a total of 8,764 tweets written in Spanish and 9,009 written in Catalan.
We performed some experiments in order to determine if the context tweets could help in the task. The results were positive, and thereby, we decided to include the previous and next tweets of each tweet as part of it. We also tried with the StanceCat 2017 dataset, however in this case, the experiments showed that it did not improve the performance. Thus, nally, the StanceCat 2017 dataset was not used for training our system. The tweets were represented using the simple Bag-Of-Word (BoW) approach, but instead of using the word frequencies, we used their inverse document frequencies (tf-idf) to measure the word relevance in the whole collection of tweets. To do this, we used the T dfVectorizer class to convert the tweets into tf-idf values.
As the organizers have not provided any validation set, we randomly generate a test dataset (20% from the training dataset). To do this, we used the Strati edShu eSplit class that provides a random split with same balance of classes. Moreover, we performed grid search on all combinations of the parameters and for each classi er in order to nd best setting (see Table 1). We used the GridSearchCV class. LABDA's early steps toward Multimodal Stance detection 5 dataset. Linear SVM obtained the top F1 for the classes FAVOR (F1=91%) and NEUTRAL (F1=75%). Random Forest and Multinomial NB also achieved the top F1 for FAVOR.
Based on the experiment results, we decided to use Linear SVM to process the test dataset.
We sent two di erent runs: using the context tweets and without using them. The organizers published the nal results and our classi er using text and context information achieved the top macro F1 (0.2802) for Spanish. However, this setting is the fourth place for Catalan with a macro F1 of 0.2876 (the top F1 was 0.3068). 4
Our system is a very simple approach, which only exploits the tweets. The task is very attractive and there is much room for improvement. We will try machine learning classi ers trained using hand-engineered features as well as word embeddings. We also plan to extend our research by using deep learning methods. 6
I. Segura-Bedmar
This work was supported by the Research Program of the Ministry of Economy and Competitiveness - Government of Spain, (DeepEMR project TIN201787548-C2-1-R). LABDA's early steps toward Multimodal Stance detection 7