In this paper we present experimental assessment of a dynamic adaptation of an approach for sentiment classification of tweets. Specifically, this approach enables a dynamic adaptation of the parameters used for three-class classification with a binary SVM classifier. The approach is suited for incremental active learning scenarios in domains with frequent concept alterations and changes. Our target application is in domain of finance and the assessment is partially domain-specific, but the approach itself is not limited to a particular domain.
Copyright c by the paper’s authors. Copying permitted for private and academic purposes.
The presented active learning methodology and the approach for examining the relationship between tweet sentiment and stock prices is explained in detail in our previous studies [Sma14, SGLZˇ13, SGLZˇ14]. In Section 2 we briefly revisit the active learning approach and the concept of the neutral zone. The new extended experiments with the dynamic neutral zone are listed in Section 3 and discussed in Section 4. 2
The aim of our experiments is to discover the best combination of parameters of the developed active learning methodology [Sma14, SGLZˇ14] for sentiment analysis. The initial sentiment model is trained using the smileylabeled Twitter messages [GBH09]1 by employing the Support Vector Machine (SVM) [Vap95] algorithm. We measure the model performance (in terms of the F-measure of the positive class) on a simulated stream of tweets by employing the holdout evaluation approach adjusted for dynamic environments [BK09, IGD11], that is, we evaluate the model on each new batch of data from the Twitter data stream. The simulated data stream consists of tweets which discuss Baidu2 stocks in year 2011. Moreover, active learning is performed, i.e. a selection of tweets from each batch is chosen to be manually labeled and added to the model.
The sentiment model is trained using the positive and negative tweets. However, in the classification phase we adjust the output of the SVM algorithm to detect also the neutral tweets by employing the concept of the neutral zone, that is, examples which are positioned in the neutral zone are marked as neutral. There are various ways of implementing the concept of the neutral zone. For example, the fixed neutral zone is constrained by empirically predefined boundaries [SGLZˇ13, SGLZˇ14, Sma14] as sketched in Figure 1(a), while the relative neutral zone is a function of positive and negative average distances of training examples [SKG+15, Sma14]. The key idea of the latter approach is the following. Given that an example is projected on the positive side of the SVM hyperplane at distance d and the average distance of positive training examples is d¯+, the first step is to calculate the classification reliability by applying the following formula [SKG+15, Sma14]: (1) (2) R = d
¯ 2 ∗ d+
If the calculated reliability is greater than 1, it is transformed to R = 1. The example is labeled as neutral if its classification reliability is below a predefined reliability threshold RT . Figure 1(b) presents an example of classifying an instance (at distance d) in this setting. The same approach (with using the average distances of negative training examples) is applied if an example is projected on the negative side of the SVM hyperplane.
In the active learning environment we dynamically update not only the sentiment model, but also the parameters of the relative neutral zone, i.e. the average training distances. The positive average distance is updated by applying the following formula [Sma14]: d¯+0 = (1 − α) ∗ d¯+ + α ∗ db ¯ where d¯+0 is an updated average distance, d¯+ is the current one, and d¯b is the average distance of the positive examples in the currently processed batch b, which were used for updating the model. Parameter α controls the influence of the new and previous tweets. If α is set to 0, the average distance of initial training examples does not get dynamically updated. Equation 2 is applied accordingly for dynamically updating the negative average distance.
We experimented with the following active learning query strategies [Sma14, SGLZˇ14] to select the most suitable examples from each batch of data for manual labeling: 1. Closest to the neutral zone: the algorithm chooses a selection of tweets whose classification reliability is closest to the reliability threshold. The number of positive/negative examples (according to the classifier’s labeling) must not exceed half of the allocated manual labels. 2. Random: the algorithm randomly selects tweets for manual labeling. 3. Combined approach: combination of two previous approaches, i.e. a certain percentage of tweets is chosen randomly, while the rest of the tweets are chosen according to the ”Closest to the neutral zone” strategy. 1The dataset was obtained from the Sentiment140 Web page, section“For Academics” (http://help.sentiment140.com/ for-students).
2http://www.baidu.com/. (a) Fixed Neutral Zone
(b) Relative Neutral Zone Additionally, we evaluated the scenario without active learning, i.e. without updating the sentiment model or neutral zone.
The Friedman test [Dem06, Fri37, Fri40], the Iman-Davenport improvement [ID80], and the Nemenyi posthoc test [Nem63] were used to rank a selection of the evaluated active learning settings and to find statistically significant differences between them.
The implementation of the methodology uses elements of several libraries: Pegasos SVM [SSSS07] from the sofia-ml library3 [Scu10], SWIG4 to connect sofia-ml C++ implementation with C# programming language, and the LATINO library5 for preparing the features. The learning algorithm for the initial model training in sofia-ml was adapted by implementing sampling which takes examples in succession. 3
In this study we extend the experimental setting from [Sma14] and test the following parameters and their values (the parameters used already in [Sma14] are also included): • Alpha values: 0, 0.05, 0.1, 0.3, 0.5. • Five active learning querying strategies and one without active learning. • Two batch selection strategies: select 10 of 100 (select 10 examples for manual labeling out of 100 examples in a batch) and select 10 of 50. • Reliability threshold: 0, 0.1, 0.2, 0.3, 0.4, 0.5.
The results of experimental assessment for all combinations of the above parameters are presented in Table 1.
From Table 1 it is not straightforward to conclude which combination of parameters is the best one. For that reason, we present the average results of active learning strategies over all values of reliability threshold in Figure 2, where lighter color corresponds to lower values and darker color corresponds to higher (better) values. We exclude the “AL closest to NZ” strategy since the results in Table 1, which are marked with asterisk(s), indicate that this strategy is unreliable as many batches did not have positively classified tweets, which caused missing values of F-measure (see [Sma14] for more details on this phenomenon). The Figure 2 indicates that both 0.1 and 0.3 are reasonable values for the parameter α. However, we focus on α = 0.3, since we already performed the analysis of α = 0.1 in our previous study [Sma14].
3https://code.google.com/p/sofia-ml/. 4http://www.swig.org/. 5https://github.com/LatinoLib/LATINO.
The results of the Friedman test with the Iman-Davenport improvement and the Nemenyi post-hoc test for α = 0.3 are presented in Figure 3. The strategies which are not significantly different are connected with a red line. From the figure it follows that the best active learning settings are: “Select 10 of 100 with AL comb. 20% random”,“Select 10 of 50 with AL comb. 20% random” and “Select 10 of 100 with AL comb. 50% random”.
Finally, we analyze the relationship between sentiment in tweets and stock closing prices of the discussed company. We apply the Granger causality test [Gra69] for different time lags and time periods on two time series: daily change of the positive sentiment probability and daily return in stock closing price [Sma14, SGLZˇ13, SGLZˇ14]. This statistical test indicates whether one time series is useful for predicting the values of another one. The results for top three active learning settings for α = 0.3 are shown in Table 2. The significant results, after applying the Bonferroni correction [Abd07], are marked in bold (which corresponds to values lower than 0.025). 4
We presented an extended experimental assessment of the active learning methodology with dynamic neutral zone in which we were particularly interested in experimenting with the parameter α which dynamically updates the neutral zone as new examples arrive from the data stream. The conclusions of this extended study are in agreement with our previous one [Sma14]. The indications about the characteristics of the neutral zone are now strengthened, but for many aspects still lack a decisive statistical significance.
For α = 0 all the active learning strategies are better than the strategy without active learning. However, the differences between the strategies are not so prominent (see Figure 2). On the other hand, the results with the dynamic neutral zone (α > 0) are more diverse between different strategies and the new results in this setting show that besides completely random query strategy, also combinations with a strong random component (80%) are even worse than not applying active learning at all. In general, the “Select 10 of 100” batch selection seems to be somewhat better than “Select 10 of 50” selection, which is not intuitive, but might be partly caused by partitioning of the batches. Moreover, in larger batches the querying strategies might be more effective as they operate on larger number of different examples. Discovery of the exact cause would be a possible direction for further work. Regarding the query strategies, the combined seem to be the best ones, but the differences among them are usually not significant (see Figure 3). The Granger causality analysis showed that there is a relationship between sentiment in tweets and stock prices in specific time periods, mostly June-August, as already shown in [Sma14]. The relationship also depends on choosing an appropriate active learning setting and the value of reliability threshold. 0.1 0.785 0.469 0.396 0.696 0.837 0.244 0.713 0.683 0.946 0.444 0.413 0.324 Acknowledgements This work was partially funded by the European Commission in the context of the FP7 projects FIRST and FOC (Grant No. 257928 and 255987), by the Slovenian Research Agency through the research program Knowledge Technologies under (Grant P2-0103) and the project Influence of formal and informal corporate communications on capital markets (Grant No. J5-7387). We are grateful to Dragi Kocev for his help in the statistical evaluation of the results and Martin Saveski for his help with the implementation of the active learning algorithms. [Abd07]
Herv Abdi. Bonferroni and Sˇid´ak corrections for multiple comparisons. In Neil Salkind, editor, Encyclopedia of Measurement and Statistics, pages 103–107. Thousand Oaks (CA): Sage, 2007. Albert Bifet and Richard Kirkby. Data stream mining: A practical approach. 2009.
Janez Demˇsar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7:1–30, 2006.
Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701, 1937.
Milton Friedman. A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86–92, 1940.
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, pages 1–12, 2009.
Clive W.J. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: Journal of the Econometric Society, pages 424–438, 1969.
Ronald L. Iman and James M. Davenport. Approximations of the critical region of the friedman statistic. Communications in Statistics-Theory and Methods, 9(6):571–595, 1980.
Elena Ikonomovska, Jo˜ao Gama, and Saˇso Dˇzeroski. Learning model trees from evolving data streams. Data Mining and Knowledge Discovery, 23(1):128–168, 2011.
Svetlana Kiritchenko, Xiaodan Zhu, and Saif M Mohammad. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762, 2014. [Nem63] [Scu10]
David Sculley. Combined regression and ranking. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 979–988. ACM, 2010. [Sma14] [SSSS07]
Jasmina Smailovi´c. Sentiment analysis in streams of microblogging posts. PhD thesis, Joˇzef Stefan International Postgraduate School, Ljubljana, Slovenia, 2014. [Vap95]
Vladimir N. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.