In recent years, traditional economic models failed to forsee several developments resulting in a considerable economic crisis. Other phenomena, such as the increase in Bitcoin value cannot be completely modeled by these traditional means either. As Bitcoin and other cryptocurrencies are a playground for technically interested people, it might be worthwhile to look into other communication channels, such as Social Media to find clues for the development we observe. We hypothesize that sentiment expressed in, for example, might model the development of Bitcoin value better than traditional models. In this work, we present a data set of Tweets covering almost one year, which we annotated for Sentiment. Additionally, we show results from preliminary experiments which support our hypothesis that sentiment information is highly predictive of the value development.
Financial markets sometimes exhibit tendencies, Keynes (1936) describes as Animal Spirits and in the past, traditional models failed to support all that was observable from an economic point of view. Kindleberger (1978) was able to show already in 1978 in the context of a financial crisis that opinions and beliefs of investors are related to news and journal articles. This is especially true for cryptocurrencies such as Bitcoin, which showed a rather erratic behaviour in the past two years. To get new insights into market behaviour, we decide to use Twitter and evaluate whether Tweets can give us more information on the currency’s behaviour than traditional models. The hypothesis behind this is, that people investing in Bitcoin might also voice their opinions and/or beliefs through Social Media channels, such as Twitter and therefore influence the market on a subjective level. To that end, we collect Tweets and perform a sentiment analysis on them. Our main question is whether sentiments expressed in Tweets correlate with the value of the crypotcurrency. Our preliminary results indicate that the degree of sentiment does strongly correlate with the development of the currency and that information found in Tweets could improve traditional economic models.
Our major contributions are1: • A dataset of Tweets related to Bitcoin. • A subset of the main data set that was manually annotated for sentiment. • An evaluation of various off-the shelf machine learning methods to automatically classify sentiment in Tweets. • A preliminary analysis of the development of sentiment in Tweets in correlation to the development of the value of the cryptocurrency Bitcoin.
The paper is structured as follows: Section 2 gives an overview on the relevant related work. In Section 3 we describe the data collection and manual annotation. In Section 4 we describe the machine learning and baseline methods used and the features extracted from the data. Section 5 presents the results and their discussion and we finalize the paper with our conclusions and some pointers for future work in Section 6.
∗The author was doing his final thesis at the University of Applied Sciences Darmstadt.
1The data set and its annotations are available at https: //github.com/mieskes/BitcoinTweets
Work on Sentiment analysis is available in abundance and reviewing the whole field is beyond the scope of this paper. Ma¨ntyla¨ki et al. (2018) present a survey on the topic of sentiment analysis by looking at over 6000 publications, of which 99% were published after 2004. Therefore, we focus on work that was most influential for us.
Gonc¸alves et al. (2013) look into methods for assigning sentiment to five data sets. They test various methods, including lexicon-based approaches. Their results indicate, that machine learning works best for Twitter.
With respect to sentiment analysis of Twitter the
SemEval tasks are of specific interest. Results from
the 2016 installment
In 2017 the subtask A aimed at a three-point
classification (positive, negative and neutral), while
subtask B was the same as in 2016
For 2018 the tasks changed slightly to look at emotions and valence. The annotation for the valence task was done on a 7-point scale, ranging from very positive mental state to -3 very negative mental state.2
With respe
competitions/17751
As Bitcoin values evolve rapidly, we assume that a medium that allows for rapid communication, such as Twitter more closely reflects the development of the currency.
From Twitter we extract Tweets with relation to Bitcoin, by identifying them through their respective hashtags, such as #bitcoin, #btc, #cryptocurrency etc. We collected data from January 2018 until August 2018 and restricted our collection to English Tweets only, to reduce the chance to have a mixed-language data set. The total data set contains over 50 million Tweets3.
Figure 2 shows how often Hashtags related to Bitcoin and cryptocurrencies occur in our data set. We observe that only approximately 17% of the Tweets are actually marked with bitcoin, while a lot of Tweets refer to other cryptocurrencies or deal with general topics related to them, such as mining. To reduce the data set we removed duplicate Tweets, as identified by their ID and also retweeted Tweets. 3.1
We perform a range of preprocessing steps inspired by Mart´ınez-C a´mara et al. (2013) in order to extract features and feed the data to the machine learning algorithms. These preprocessing steps included filtering for stop words, removal of hashtags, UserIDs and URLs within the Tweets. The remaining data only contains plain text. 3.2
To be able to train a machine learning model, we need training data. We handed slightly less than 2000 Tweets to human annotators via Amazon Mechanical Turk to annotate them for sentiment.
Figure 1 shows the task description and the annotation interface as displayed on Amazon Mechanical Turk. We coloured the various levels of sentiment for ease of use. In the description, we refer to positive sentiment as indication of rising value and negative sentiment as indication for dropping value of the currency. Apart from the plain text, Turkers did not get any meta data on the Tweets.
Each Tweet is annotated by 7 Turkers and results were averaged. Average values ≥ 0.15 are considered positive Tweets, ≤ −0.15 are considered 3The set of Tweet IDs are available at https:// github.com/mieskes/BitcoinTweets negative Tweets and results in between are considered neutral. Our final training data set contains 1042 positive Tweets, 727 negative Tweets and 88 neutral Tweets.
We evaluate the annotation quality using Krippendorffs α. As expected, the inter-annotator agreement for the full distinction is fairly low (α = 0.13). As we are primarily interested in positive, negative and neutral sentiment, we collapsed the annotations to represent only the three main classes (Details are described above). Nevertheless, the result (α = 0.43) was considered improvable. A more detailed look at the annotation revealed, that in some cases individual annotators annotated the complete or near opposite of what the
Text @SilverBulletBTC Damn, and I can not buy . . .
Gauthier-Mohammed: I will be a father of . . .
Oh my! So many #scam these days . . .
New #Blockchain marketplace Repayment . . . majority had done. We identified these instances and removed them from consideration. This left us with enough annotations to create a gold standard on it and raised the inter-annotator agreement to α = 0.53, which, considering the complexity of the taks, is a good result.
Table 1 shows example Tweets from the training data. The first column shows the average sentiment value based on all annotations and the last column shows the mapped sentiment classification.
Figure 4 shows the distribution of tweets annotated with a specific sentiment class. We observe, that more tweets receive a positive classification, while fewer receive a negative classification. Most tweets are annotated as Moderately or Very Positive, while on the negative side, the various subclasses are more evenly distributed. It is interesting to note, that very few tweets are marked as Extremely Negative, while on the positive side, a considerable amount of tweets are marked as Extremely Positive. This indicates, that most tweets are positive, up to the degree of being enthusiastic. We experiment with a range of machine learning methods – both classical and deep learning-based. We use SVMs and Random Forest in addition to two deep learning based methods, which we describe in the following. 4.1
We employ two baseline systems in our experiments. Hutto and Gilbert (2014) describe vadersentiment4 as a lexicon and rule-based sentiment analysis tool, which is specifically targeted towards Social Media. On Social Media the authors achieve an overall F1 score for the classification of positive, negative and neutral sentiment of 0.96. The tool is implemented in Python.
Sentimentr5 is implemented in R and is also
vaderSentiment 5https://github.com/trinker/ lexicon-based. The implementation is tested on three different review datat sets (Amazon, Yelp and IMDB) and achieve accuracy rates between 76.5% for the Amazon Review data set and 71.5% for the Yelp data set. 4.2
We also experiment with various machine learning approaches. Two serve as baselines and are traditional machine learning systems, while two are deep-learning based.
We use Random Forest and Support Vector
Machines (SVM) in their implementation in R using
standard features. Using a GridSearch and 10-fold
cross-validation, we experimentally determine the
best parameters for both SVM and Random Forest
and use them to classify the data.
4.2.2 HDLTex
The Hierarchical Deep Learning for Text
Classification has been developed specifically for text
classification
We experimentally adapt the model with respect to the various parameters. Most importantly, we increase the drop out to 65% and use only 15 epochs.
Figure 3 shows how the accuracy of the models using the original (left side) and modified (right sentimentr;https://cran.r-project.org/ web/packages/sentimentr/sentimentr.pdf side) HDLTex architecture develop. We see that both methods reach the plateau measured in accuracy between 5 to 10 epochs.6 But the modified HLDTex architecture achieves a higher accuracy on the test data than the original HDLTex architecture. 4.2.3
We use the Convolutional Neural Network for
Sentence
Similar to the HDLTex we experimentally create a modified architecture, which uses fewer epochs (20), more filters (128) and a higher drop out rate (75%).
Figure 5 shows how the models using the original (left side) and modified (right side) CNNSC architecture develop. While the original architecture shows a somewhat “bumpy” start in the first 5 epochs, the learning curve for the modified architecture is considerably smoother. Furthermore, the modified CNNSC achieves a higher accuracy both in the training and the test data. 5
In the following we present results for the sentment classification and the relation of the sentiment index to the development of the Bitcoin value. 5.1
Table 2 shows the results for the various machine learning methods and the two baselines we used (see Section 4) for details. We observe that all methods are fairly close together in terms of F1 and overall accuracy. For the negative class, the modified CNNSC achieves the best results, while for the positive class vaderSentiment achieves the best results. Both methods perform similarly with respect to overall accuracy. This lack of difference 6The graph on the right is based on fewer epochs. between the two methods might be due to the comparably small data set used for training and that a larger data set might boost the performance of the deep learning-based system. Overall, our results are comparable to what has been reported in the literature.
Figure 6 shows the unigram features ranked by their importance. We observe that the most predictive unigrams are actually easily associated with positive or negative sentiment. Words like join are less clear, but nevertheless rank comparably high for the sentiment classification.
An initial error analysis shows that, as expected, the neutral class, which makes up about 5% of our data set, causes misclassifications. Either because neutral tweets are classified as having positive or negative sentiment or the other way around. Therefore, improving the classification of the neutral class might also improve the overall classification. 5.2
In the next step, we apply the adapted CNNSC to the whole data set in order to classify the data from the complete observed time frame. Figure 7 shows the results for the sentiment development in comparison to the Bitcoin value. The index is normalized to range between 0 and 1. The negative value for the sentiment index at the starting point is an artefact due to lack in previous data. We observe that the Bitcoin value constantly dropped during the observed time-frame, with some bumps in between. The sentiment index closely follows this development and reflects it.
In addition, we perform initial experiments using time-series analysis. For this, we look at the development of the sentiment index and the Bitcoin value on a daily basis. These preliminary results indicate that the sentiment index is a highly significant predictor for the Bitcoin value. But as both Twitter and Bitcoin are rapidly developing and changing, it would be interesting to also investigate shorter time-frames, such as half-day or hourly predictions. 6
We presented a data set of Tweets related to Cryptocurrencies. We manually analysed a subset of the Tweets in order to re-train and evaluate various machine learning and off-the-shelf sentiment classification methods. The main question though was to analyse the development of the sentiment expressed in Tweets in relation to the development of the currency’s value. We found that off-the-shelf tools perform well enough to automatically analyse this type of data. Moreover, the sentiment index closely reflected the Bitcoin value, which indicates that the analysis of social media data could support current economical models in predicting future developments. Initial results using time-series analysis indicate that the sentiment index is highly predictive of the currency development.
Future Work The first next step is to extend the time-series analysis and evaluate if the predictions also hold on a shorter time-frame (i.e., half-day or hourly predictions). Additionally, looking not only at sentiment, but also at emotions and especially extreme emotions might provide additional information.
We currently only looked at positive, negative
and neutral sentiment. Extending this to cover the
whole annotated range could give additional
improvement on the prediction and the currency value
development. Finally, it would be interesting to
evaluate whether these findings also hold in other
areas of economics. Work by
This work was supported by the research center for Applied Computer Science (FZAI) and the Faculty for Mathematics and Natural Sciences, University of Applied Sciences Darmstadt.