Stance Detection in Turkish Tweets

Abstract

Stance detection is a classification problem in natural language processing where for a text and target pair, a class result from the set {Favor, Against, Neither} is expected. It is similar to the sentiment analysis problem but instead of the sentiment of the text author, the stance expressed for a particular target is investigated in stance detection. In this paper, we present a stance detection tweet data set for Turkish comprising stance annotations of these tweets for two popular sports clubs as targets. Additionally, we provide the evaluation results of SVM classifiers for each target on this data set, where the classifiers use unigram, bigram, and hashtag features. This study is significant as it presents one of the initial stance detection data sets proposed so far and the first one for Turkish language, to the best of our knowledge. The data set and the evaluation results of the corresponding SVM-based approaches will form plausible baselines for the comparison of future studies on stance detection.

CCS CONCEPTS

• Information systems → Sentiment analysis; Web and social media search; • Computing methodologies → Language resources;

INTRODUCTION

Stance detection (also called stance identification or stance classification) is one of the considerably recent research topics in natural language processing (NLP). It is usually defined as a classification problem where for a text and target pair, the stance of the author of the text for that target is expected as a classification output from the set: {Favor, Against, Neither} [ 12 ].

Stance detection is usually considered as a subtask of sentiment analysis (opinion mining) [ 13 ] topic in NLP. Both are mostly performed on social media texts, particularly on tweets, hence both are important components of social media analysis. Nevertheless, in sentiment analysis, the sentiment of the author of a piece of text usually as Positive, Negative, and Neutral is explored while in stance detection, the stance of the author of the text for a particular target (an entity, event, etc.) either explicitly or implicitly referred to in the text is considered. Like sentiment analysis, stance detection systems can be valuable components of information retrieval and other text analysis systems [ 12 ].

Previous work on stance detection include [ 16 ] where a stance classifier based on sentiment and arguing features is proposed in addition to an arguing lexicon automatically compiled. The ultimate approach performs better than distribution-based and unigram-based baseline systems [ 16 ]. In [ 17 ], the authors show that the use of dialogue structure improves stance detection in on-line debates. In [ 7 ], Hasan and Ng carry out stance detection experiments using diferent machine learning algorithms, training data sets, features, and inter-post constraints in on-line debates, and draw insightful conclusions based on these experiments. For instance, they find that sequence models like HMMs perform better at stance detection when compared with non-sequence models like Naive Bayes (NB) [ 7 ]. In another related study [ 10 ], the authors conclude that topic-independent features can be exploited for disagreement detection in on-line dialogues. The employed features include agreement, cue words, denial, hedges, duration, polarity, and punctuation [ 10 ]. Stance detection on a corpus of student essays is considered in [ 5 ]. After using linguistically-motivated feature sets together with multivalued NB and SVM as the learning models, the authors conclude that they outperform two baseline approaches [ 5 ]. In [ 4 ], the author claims that Wikipedia can be used to determine stances about controversial topics based on their previous work regarding controversy extraction on the Web.

Among more recent related work, in [ 1 ] stance detection for unseen targets is studied and bidirectional conditional encoding is employed. The authors state that their approach achieves state-ofthe art performance rates [ 1 ] on SemEval 2016 Twitter Stance Detection corpus [ 12 ]. In [ 3 ], a stance-community detection approach called SCIFNET is proposed. SCIFNET creates networks of people who are stance targets, automatically from the related document collections [ 3 ] using stance expansion and refinement techniques to arrive at stance-coherent networks. A tweet data set annotated with stance information regarding six predefined targets is proposed in [ 11 ] where this data set is annotated through crowdsourcing. The authors indicate that the data set is also annotated with sentiment information in addition to stance, so it can help reveal associations between stance and sentiment [ 11 ]. Lastly, in [ 12 ], SemEval 2016’s aforementioned shared task on Twitter Stance Detection is described. Also provided are the results of the evaluations of 19 systems participating in two subtasks (one with training data set provided and the other without an annotated data set) of the shared task [ 12 ].

In this paper, we present a tweet data set in Turkish annotated with stance information, where the corresponding annotations are made publicly available. The domain of the tweets comprises two popular football clubs which constitute the targets of the tweets included. We also provide the evaluation results of SVM classifiers (for each target) on this data set using unigram, bigram, and hashtag features.

To the best of our knowledge, the current study is the first one to target at stance detection in Turkish tweets. Together with the provided annotated data set and the corresponding evaluations with the aforementioned SVM classifiers which can be used as baseline systems, our study will hopefully help increase social media analysis studies on Turkish content.

The rest of the paper is organized as follows: In Section 2, we describe our tweet data set annotated with the target and stance information. Section 3 includes the details of our SVM-based stance classifiers and their evaluation results with discussions. Section 4 includes future research topics based on the current study, and finally Section 5 concludes the paper with a summary. 2

A STANCE DETECTION DATA SET

We have decided to consider tweets about popular sports clubs as our domain for stance detection. Considerable amounts of tweets are being published for sports-related events at every instant. Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey. As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs.

In a previous study on the identification of public health-related tweets, two tweet data sets in Turkish (each set containing 1 million random tweets) have been compiled where these sets belong to two diferent periods of 20 consecutive days [ 9 ]. We have decided to use one of these sets (corresponding to the period between August 18 and September 6, 2015) and firstly filtered the tweets using the possible names used to refer to the target clubs. Then, we have annotated the stance information in the tweets for these targets as Favor or Against. Within the course of this study, we have not considered those tweets in which the target is not explicitly mentioned, as our initial filtering process reveals.

For the purposes of the current study, we have not annotated any tweets with the Neither class. This stance class and even finergrained classes can be considered in further annotation studies. We should also note that in a few tweets, the target of the stance was the management of the club while in some others a particular footballer of the club is praised or criticised. Still, we have considered the club as the target of the stance in all of the cases and carried out our annotations accordingly.

At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2. Hence, our data set is a balanced one although it is currently limited in size. The corresponding stance annotations are made publicly available at http://ceng.metu.edu.tr/ e120329/ Turkish_Stance_Detection_Tweet_Dataset.csv in Comma Separated Values (CSV) format. The file contains three columns with the corresponding headers. The first column is the tweet id of the corresponding tweet, the second column contains the name of the stance target, and the last column includes the stance of the tweet for the target as Favor or Against.

To the best of our knowledge, this is the first publicly-available stance-annotated data set for Turkish. Hence, it is a significant resource as there is a scarcity of annotated data sets, linguistic resources, and NLP tools available for Turkish. Additionally, to the best of our knowledge, it is also significant for being the first stance-annotated data set including sports-related tweets, as previous stance detection data sets mostly include on-line texts on political/ethical issues. 3

STANCE DETECTION EXPERIMENTS

USING SVM CLASSIFIERS

It is emphasized in the related literature that unigram-based methods are reliable for the stance detection task [ 16 ] and similarly unigram-based models have been used as baseline models in studies such as [ 12 ]. In order to be used as a baseline and reference system for further studies on stance detection in Turkish tweets, we have trained two SVM classifiers (one for each target) using unigrams as features. Before the extraction of unigrams, we have employed automated preprocessing to filter out the stopwords in our annotated data set of 700 tweets. The stopword list used is the list presented in [ 8 ] which, in turn, is the slightly extended version of the stopword list provided in [ 2 ].

We have used the SVM implementation available in the Weka data mining application [ 6 ] where this particular implementation employs the SMO algorithm [ 14 ] to train a classifier with a linear kernel. The 10-fold cross-validation results of the two classifiers are provided in Table 1 using the metrics of precision, recall, and F-Measure.

The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set. The performance of the classifiers is better for the Favor class for both targets when compared with the performance results for the Against class. This outcome may be due to the common use of some terms when expressing positive stance towards sports clubs in Turkish tweets. The same percentage of common terms may not have been observed in tweets during the expression of negative stances towards the targets. Yet, completely the opposite pattern is observed in stance detection results of baseline systems given in [ 12 ], i.e., better FMeasure rates have been obtained for the Against class when compared with the Favor class [ 12 ]. Some of the baseline systems reported in [ 12 ] are SVM-based systems using unigrams and ngrams as features similar to our study, but their data sets include all three stance classes of Favor, Against, and Neither, while our data set comprises only tweets classified as belonging to Favor or Against classes. Another diference is that the data sets in [ 12 ] have been divided into training and test sets, while in our study we provide 10-fold cross-validation results on the whole data set. On the other hand, we should also note that SVM-based sentiment analysis systems (such as those given in [ 15 ]) have been reported to achieve better F-Measure rates for the Positive sentiment class when compared with the results obtained for the Negative class. Therefore, our evaluation results for each stance class seem to be in line with such sentiment analysis systems. Yet, further experiments on the extended versions of our data set should be conducted and the results should again be compared with the stance detection results given in the literature.

We have also evaluated SVM classifiers which use only bigrams as features, as ngram-based classifiers have been reported to perform better for the stance detection problem [ 12 ]. However, we have observed that using bigrams as the sole features of the SVM classifiers leads to quite poor results. This observation may be due to the relatively limited size of the tweet data set employed. Still, we can conclude that unigram-based features lead to superior results compared to the results obtained using bigrams as features, based on our experiments on our data set. Yet, ngram-based features may be employed on the extended versions of the data set to verify this conclusion within the course of future work.

With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams. The corresponding evaluation results of the SVM classifiers using unigrams together the existence of hashtags as features are provided in Table 2.

When the results given in Table 2 are compared with the results in Table 1, a slight decrease in F-Measure (0.5%) for Target-1 is observed, while the overall F-Measure value for Target-2 has increased by 1.8%. Although we could not derive sound conclusions mainly due to the relatively small size of our data set, the increase in the performance of the SVM classifier Target-2 is an encouraging evidence for the exploitation of hashtags in a stance detection system. We leave other ways of exploiting hashtags for stance detection as a future work.

To sum up, our evaluation results are significant as reference results to be used for comparison purposes and provides evidence for the utility of unigram-based and hashtag-related features in SVM classifiers for the stance detection problem in Turkish tweets. 4

FUTURE PROSPECTS

Future work based on the current study includes the following: The presented stance-annotated data set for Turkish has been created by one annotator only (the author of this study), yet, the data set should better be revised and extended through crowdsourcing facilities. When employing such a procedure, other stance classes like Neither can be considered as well. The procedure will improve the quality the data set as well as the quality of prospective systems to be trained and tested on it.

Other features like emoticons (as commonly used for sentiment analysis), features based on hashtags, and ngram features can also be used by the classifiers and these classifiers can be tested on larger data sets. Other classification approaches could also be implemented and tested against our baseline classifiers. Particularly, related methods presented in recent studies such as [ 12 ] can be tested on our data set.

Lastly, the SVM classifiers utilized in this study and their prospective versions utilizing other features can be tested on stance data sets in other languages (such as English) for comparison purposes. 5

CONCLUSION

Stance detection is a considerably new research area in natural language processing and is considered within the scope of the wellstudied topic of sentiment analysis. It is the detection of stance within text towards a target which may be explicitly specified in the text or not. In this study, we present a stance-annotated tweet data set in Turkish where the targets of the annotated stances are two popular sports clubs in Turkey. The corresponding annotations are made publicly-available for research purposes. To the best of our knowledge, this is the first stance detection data set for the Turkish language and also the first sports-related stance-annotated data set. Also presented in this study are SVM classifiers (one for each target) utilizing unigram and bigram features in addition to using the existence of hashtags as another feature. 10-fold cross validation results of these classifiers are presented which can be used as reference results by prospective systems. Both the annotated data set and the classifiers with evaluations are significant since they are the initial contributions to stance detection problem in Turkish tweets.

References

[1] Isabelle Augenstein , Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva . 2016 . Stance detection with bidirectional conditional encoding . In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).

[2] Fazli Can , Seyit Kocberber, Erman Balcik, Cihan Kaynak, H Cagdas Ocalan , and Onur M Vursavas . 2008 . Information retrieval on Turkish texts . Journal of the American Society for Information Science and Technology 59 , 3 ( 2008 ), 407 - 421 .

[3] Zhong-Yong Chen and Chien Chin Chen. 2016 . SCIFNET: Stance community identification of topic persons using friendship network analysis . KnowledgeBased Systems 110 ( 2016 ), 30 - 48 .

[4] Shiri Dori-Hacohen . 2015 . Controversy Detection and Stance Analysis . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1057 - 1057 .

[5] Adam Faulkner . 2014 . Automated classification of stance in student essays: An approach using stance target information and the Wikipedia link-based measure . In Proceedings of the Twenty-Seventh International Florida Artificial Intelligence Research Society Conference.

[6] M. Hall , E. Frank, G. Holmes , B. Pfahringer , P. Reutemann , and I. H. Witten . 2009 . The WEKA Data Mining Software: An Update . ACM SIGKDD Explorations Newsletter 1 , 1 ( 2009 ), 10 - 18 .

[7] Kazi Saidul Hasan and Vincent Ng . 2013 . Stance Classification of Ideological Debates: Data, Models, Features, and Constraints . In Proceedings of the Sixth International Joint Conference on Natural Language Processing . 1348 - 1356 .

[8] Dilek Küçük . 2011 . Exploiting Information Extraction Techniques for Automatic Semantic Annotation and Retrieval of News Videos in Turkish . Ph.D. Dissertation . Middle East Technical University.

[9] Emine Ela Küçük , Kürşad Yapar, Dilek Küçük, and Doğan Küçük . 2017 . Ontology-based automatic identification of public health-related Turkish tweets . Computers in Biology and Medicine 83 ( 2017 ), 1 - 9 .

[10] Amita Misra and Marilyn A Walker . 2013 . Topic independent identification of agreement and disagreement in social media dialogue . In Conference of the Special Interest Group on Discourse and Dialogue . 920 .

[11] Saif M Mohammad , Svetlana Kiritchenko , Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry . 2016 . A dataset for detecting stance in tweets . In Proceedings of the Language Resources and Evaluation Conference (LREC).

[12] Saif M Mohammad , Svetlana Kiritchenko , Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry . 2016 . Semeval-2016 task 6: Detecting stance in tweets . In Proceedings of the International Workshop on Semantic Evaluation , SemEval.

[13] Bo Pang and Lillian Lee . 2008 . Opinion mining and sentiment analysis . Foundations and Trends in Information Retrieval 2 , 1 - 2 ( 2008 ), 1 - 135 .

[14] John C. Platt . 1999 . Fast Training of Support Vector Machines Using Sequential Minimal Optimization . Advances in Kernel Methods ( 1999 ), 185 - 208 .

[15] Hamid Poursepanj , Josh Weissbock, and Diana Inkpen . 2013 . uOttawa: System description for SemEval 2013 Task 2 Sentiment Analysis in Twitter . In Second Joint Conference on Lexical and Computational Semantics (*SEM) , Volume 2 : Seventh International Workshop on Semantic Evaluation (SemEval 2013 ). 380 - 383 .

[16] Swapna Somasundaran and Janyce Wiebe . 2010 . Recognizing stances in ideological on-line debates . In Proceedings of the NAACL HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text . 116 - 124 .

[17] Marilyn A Walker , Pranav Anand, Robert Abbott, and Ricky Grant . 2012 . Stance classification using dialogic properties of persuasion . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . 592 - 596 .