We present our system LTL_UNI_DUE which participated in the shared task on automated stance detection in tweets on Catalan independence at IberEval 2017. In our system, we combine neural (LSTM) and non-neural (SVM) classifiers to a hybrid approach using a decision tree and heuristics.
Recent political events have shown that political surveys often fail to predict the real
outcome of elections. Examples of this miss-prediction could be observed in the vote
on the UK exit from the European Union (Brexit) or the American presidential election.
As one reason for this failure, it has been discussed that people behave in a socially
desirable1 manner in polling situations [
In order to reliably and efficiently conduct analyzes of such data, systems are needed
that can automatically determine stance. To this end, NLP researchers have recently
begun to systematically address social media stance detection. There were shared tasks on
social media stance detection in English [
In the following, we describe our submission named LTL_UNI_DUE to this shared task. For our participation, we rely on the findings of previous shared tasks and develop a system that uses (almost) no language-specific models or tools and no additional training data.
1 Behaving in a way that is more likely to have social approval. 2
Systems in the previous shared tasks SemEval 2016 task 6 [
Since both approaches require tokenized texts, we apply the Twitter-specific
ArkTokenizer [
We implement a (bi-)LSTM neural network [
As input we translate the training data into sequences of dense word vectors using
the pre-trained vectors in Catalan and Spanish provided by [
The central bidirectional layer follows this layer and has 138 LSTM units, uses tanh
activation and the adam optimizer [
The non-neural system is implemented using the DKPro-TC framework (v0.9.0) [
The consideration of the two strands of approaches in the past shared tasks has shown that they make different errors and also have different strengths. Consequently, the question arises whether one can combine the strengths of both models into a superior, hybrid system. To examine this question we built a third system that automatically decides whether a tweet should be classified with the neural or the non-neural system. Therefore, we first labeled every tweet with whether the SVM’s respectively the LSTM’s prediction was wrong or false.
We than train a classifier for each approach and each language to automate this decision. Since we want to base this decision on a simple set of rules which may be transferrable to other tasks, we use a decision tree for this classification. In detail, we use weka’s J48 as implemented in DKPro-TC.
As features we use characteristics which are suspected of having an influence on the classifiability through the systems. We use the number of tokens per tweet as SVM and LSTM differ in the amount of context they model. As both systems are dependent on lexical redundancy between train and test data, we implement several redundancy features. These features are the type-token ratio and binary features indicating whether the tweet’s n-grams are contained in the training data and whether there are a pre-trained embeddings for its tokens.
Table 1 shows the performance of this classification for both systems and both languages. The rather mediocre results leave huge room for future improvements and more sophisticated machine learning. Based on these classifications we conduct a final decision. In case the system could not derive preference towards one system as both systems are recommend or none, we rely on the SVM as its performance is overall better. Catalan Spanish SVM 0.75 LSTM 0.67 In order to estimate the performance of our models, we evaluate them using a 10-fold cross-validation on the training data. Table 2 shows the performance of these experiments. For both the neural and non-neural approach, we observe better performance for Catalan than for Spanish. For the SVM we perceive an increase of +0:1 and for the LSTM we perceive an improvement of +0:06. Similarly, for both languages, the SVM performs significantly better than the LSTM. The performance decrease is bigger for Catalan (-0.1) than for Spanish (-0.06), which may be attributed to the overall better performance in Catalan.
We performed an ablation test at the level of feature groups to find out which data representation affects the model the most. The results are also shown in Table 2. We do not observe a large drop for any of the groups, which we attribute due to the fact that the modelled properties have strong overlap. For instance, embedding and unigram features model (almost) the same information, i.e. the occurrence of a certain word. However, unigrams are sparse and embeddings are dense word vectors, which both have specific advantages and disadvantages w.r.t. classification.
To quantify the similarity of the models we compute Cohen’s , which is = 0:28 for Spanish and = 0:39 for Catalan. Since the predictions are clearly different, but both models show good performance, we conclude that there is in principle much room for the hybrid model. However, the hybrid system gains a performance similar to that of the SVM. When inspecting the similarity of the hybrid model and the SVM, we obtain = 0:92 (169 different predictions) for Catalan and = 0:90 (230 different predictions) for Spanish. This high degree of agreement between the models explains their similar performance. In order to demonstrate the upper bound of the hybrid system, we also compute a oracle condition in which we assume that the LSTM vs. SVM prediction was done correctly. This oracle condition results in an increase of performance of +0.09 for Catalan and + .13 for Spanish which demonstrates the potential of the approach.
In order to provide a deeper insight into the classification performance of the models, we show the corresponding confusion matrices in Table 3 for Catalan and in Table 4 for Spanish. For both languages, we observe for the SVM a more even error distribution than for the LSTM. However, the LSTM distributes its predictions mainly to the two frequent classes (FAVOR and NEUTRAL for Catalan and AGAINST and NEUTRAL for Spanish). The hybrid model combines these two tendencies by adjusting the prediction of the SVM towards the class distribution. However, thereby a similar proportion of advantageous and disadvantageous adjustments is made. Catalan Spanish SVM 0.80 - embeddings 0.79 - character ngrams 0.78 - word ngrams 0.78 (Bi-)LSTM Hybrid Oracle 0.70 laAgainst tu Favor cANeutral 4
In this section we show, how the systems perform on the test data. We report the results in accordance with the official metric as defined by the organizers. The official metric is the macro-average of F1 and F1 . Note that this metric is beneficial for systems which are similarly good at predicting F1 and F1 , but punishes systems which are more imbalanced.
Table 5 gives an overview on the performance of our submission on the training data.
Catalan Spanish SVM 0.43 (Bi-)LSTM 0.28 Hybrid 0.44
Overall, we again observe that the SVM is superior to the LSTM system. The especially poor performance of the LSTM can also be explained by the used metric, which punishes the LSTMs tendency to ignore the sparse classes (FAVOR for the Spanish data and AGAINST for the Catalan data).
Similar to the results on the training data, we hardly see a difference between the hybrid and the SVM system for both languages. We attribute this again to the used heuristic, which uses the SVM prediction in cases were we cannot be sure about a decision. However, as the the hybrid and the SVM prediction is significantly different, we still see a high potential of hybrid approaches. As described above, future work should focus on a more accurate SVM or LSTM type prediction and more advanced heuristics. 5
In this work, we have described our participation in the shared task on automated stance detection in tweets on Catalan independence at IberEval 2017. The presented system relies on i) neural (LSTM) classifiers, ii) non-neural (SVM) classifiers and a hybrid approach which combines both classification paradigms on the basis of a decision tree and heuristics. On both the train and the test data we could not demonstrate a clear superiority of a hybrid approach. However, the obtained results highlight the potential of hybrid attempts and promising directions for further improvements.
This work was supported by the Deutsche Forschungsgemeinschaft (DFG) under grant No. GRK 2167, Research Training Group ”User-Centred Social Media”.