Predicting Reactions to Blog Headlines Rel Guzman, Jose Eduardo Ochoa-Luna, Laura Cruz-Quispe, Elizabeth Vera-Cervantes Universidad Nacional de San Agustin Arequipa, Peru {r.guzmanap,eduardo.ol,lvcruzq,elizavvc}@gmail.com Abstract classification provides limited information about sentiment, that’s the reason we also used another This paper describes some experiments type of classifier with which we are able to com- carried out to measure sentiment, which pute the probability of each reaction. we call emotional reaction, on blog head- A reaction is predicted using a predictive model lines. We analyze a text corpus of ti- consisting of Term frequency Inverse Document tles from Facebook entries or posts link- Frequency (TF-IDF), a Linear Support Vector ing to a website. These titles are basically Classifier (SVC) and a Stochastic Gradient De- headlines and we study them to understand scent (SGD) algorithm. Preliminary experiments the relationship between article headlines were run on 14 fan-pages forming a big dataset, and the self-reported reactions of the ar- and results show suitability of our approach. ticles’ readers. We utilize the recently The paper is organized as follow: In Section 2 launched feature, Facebook reactions that related work is described, next in Section 3 the enable people to express their emotional dataset is described, then our proposal is described reactions with five emojis. These reactions in Section 4. We provide experiments and results and headlines are gathered from different of the proposed method in Section 5. Finally, we fan-pages, we analyze them, make an ex- provide concluding remarks in Section 6. The ploratory data analysis and present prelim- source code to reproduce this paper is available inary results of a reaction predictor. online 1 . 1 Introduction 2 Related Work Sharing online content is an integral part of mod- Among recent studies on sentiment analysis ap- ern life, this is somewhat related with social trans- plied on Facebook, most of them analyze public missions which are driven in part by arousal, this posts shared by users (Rastogi et al., 2014), (Gao hypothesis suggests why content that evokes more et al., 2015). Taking into account information such of certain emotions is more shared (Berger, 2011). as a text messages, comments, likes. But, cur- We analyze a text corpus of titles from Facebook rently none of them have studied its new feature, entries or posts linking to a website. These titles “reactions”, neither how a reaction could affect the are headlines and we study them to understand the popularity of content. Predicting the popularity of relationship between article headlines and the self- social media content has been approached from reported reactions of the articles’ readers. We uti- many angles. Some have even been using mea- lize the recently launched feature, Facebook reac- surements of items at their early popularity apply- tions that enable people to express their emotional ing survival analysis (Lee et al., 2010). Typically, reactions with five emojis. Then, we propose a the goal of those researches is to predict the pop- simple approach to predict people’s reactions to ularity like the number of comments that will be textual content by analyzing emotional reactions. generated by an article based on its content, or how As with sentiment detection, this problem is long will it be popular. treated as a simple classification problem and A common line of research focuses on predict- achieve very high accuracy by employing various machine learning algorithms. Although, simple 1 https://github.com/rgap/simbig2016-facebook-reactions 43 ing the spread of ideas and information using con- To predict a specific reaction to a headline, we tent, topology and linguistic features. On twit- defined a target y which is the most voted reac- ter, authors tend to pay special attention to senti- tion among only the reactions, not considering the ment indicators from the content of tweets, includ- number of likes. Furthermore, we added it to the ing counts of emoticons and strength of word-level dataset only if its number of votes was higher than sentiments (Kong et al., 2014). 75% of the total number of votes, so we make sure There are studies that suggest that content that people reacted in just one way, because some of evokes either high-arousal positive emotions (awe) them could have the same number of votes per or negative emotions (anger or anxiety) tends to be reaction. The number of reaction votes per reac- more viral (Berger and Milkman, 2013). There- tion sorted by their means is shown in Figure 1. fore, a headline popularity can also be measured This diagram doesn’t tell too much only that even according to what emotion, sentiment, or reaction taking into account many fan-pages, the sad reac- it produces. And, there are also recent studies tion isn’t the most voted. We took away the head- that deal with sentiment analysis on headlines and lines with a most voted reaction with less than 200 short-texts (Nassirtoussi et al., 2015). votes. 3 Dataset All posts were extracted using the Facebook Graph API, from 14 fan-pages of popular websites about news, science, and entertainment: Buzzfeed, 9gag, Boredpanda, Mashable, Unilad, CNN, CNN international, DailyMail, FoxNews, Huffington- Post, New York Times, IFeakingLoveScience, IMDB, and Natgeo. And because Facebook re- Figure 1: Box plots sorted base on median value, actions were launched after February 2016, the showing the number of reactions votes (total num- dataset was sliced so that it only contains samples ber of reaction votes) per reaction and outliers. within “2016-04-01” and “2016-07-31”. We ob- tained a dataset of 9072 samples. A correlation matrix is shown in Figure 2, it shows how the “number of reactions”, “com- 3.1 Reactions ments”, “shares”, “likes” and the reactions corre- After an administrator posts a link directing to a late among them. website article on Facebook, a thumbnail is gen- erated which contains: image, description, and a 3.2 Headlines title. A user is redirected to the article after click- We needed to get features to predict a specific ing on the thumbnail then reads it, goes back to reaction, these features were extracted from the Facebook and gives it a reaction, or probably does headlines. Some of the headlines are less specific it before even clicking on it. Therefore, we ana- to a special date or event like “numbered lists” also lyzed a text corpus of titles from Facebook posts. known as “listcicles” like: These titles are like news headlines and we stud- ied them to understand the relationship between • 24 Heartbreaking TV Moments That Made them and the self-reported reactions of the articles’ You Cry Your Eyes Out. readers. We utilized the recently launched feature, • 27 Surreal Places To Visit Before You Die. Facebook reactions that enable people to express their emotional reactions with one of five emojis: We selected BuzzFeed also because it creates “love”, “haha”, “wow”, “sad”, and “angry”. We these types of posts, but not so many to take only selected BuzzFeed as one of the fan-pages due to a BuzzFeed into account. A predictor of these types feature its website has, a registered user is allowed of headlines would have been easier to optimize, to comment on an article with a type of reaction but it is harder to know if a title is one of these, at defined by the website, the types of reactions are: least not automatically, also we wanted to make it “love”, “lol”, “fail”, “wtf”, among others but these more general. That’s the reason we gathered data are the most voted. from many fan-pages. 44 4 Predictive Model With the dataset we created we are certainly deal- ing with a multi-class classification problem. And we found a pretty straightforward model to predict the type of emotional reaction a headline will pro- duce. The best predictive model we found was a TF-IDF + Linear SVC, and to get the probability of producing each of the reactions with a TF-IDF + SGD, we describe more about the first model be- cause it predicts the highest reaction with a higher accuracy. 4.1 Feature Extraction Figure 2: Correlation matrix showing how “love” It is very common to follow the bag-of-words is more linearly correlated to the number of reac- (BoW) approach when extracting features from tions and shares. An interesting insight shows how documents. One of the simplest feature model is “angry” is more correlated to the number of com- TF-IDF. Following a BoW representation, where ments, showing that people comment more when we call vectorization to the process of turning a dealing with controversial topics. collection of text documents which will be the headlines, into numerical feature vectors. This strategy includes: tokenization, counting and nor- malization. Then, we figured out the dataset from “buz- In our set of headlines, some words were not so zfeed” contained english and spanish headlines, very present hence carrying very little meaningful spanish ones were taken away from the corpus. information about the actual contents of the head- The headlines were tokenized and we took nor- line. Therefore, we ignored terms that have a fre- mal words, abbreviations and words with internal quency lower than 2. We extracted features for 1- hyphens/apostrophe. We took Unicode words like grams and 2-grams, both were joined and became “ber” and represented them in US-ASCII charac- a unique feature vector of size 9302 for a headline. ters so that it becomes “Uber”. More preprocess- It was not easy to extract features from the head- ing is done like profanity censorship converting lines, we tried with Latent Dirichlet Allocation every bad word in the token “badword”. Then we (LDA) and Doc2Vec, based on Word2Vec, which removed common and some special stopwords, we is kind of a state-of-the-art technique. did lemmatization and stemming. 4.2 Learning Model A histogram with number of headlines per reac- tion in descending order is shown in Figure 3, it A simple but accurate learning model to try first is shows that the dataset is imbalanced. a Linear Support Vector Machine or Support Vec- tor Classifier (SVC). It was trained and evaluated using Stratified K-Fold Cross Validation to deal with the imbalanced dataset and find a good ac- curacy score for this multi-class classification task has to be an accuracy defined by the number of well classified samples which is the “accuracy”. We got an accuracy of 0.7245 with a simple Lin- ear SVC with C = 0.29, it was the highest score among the techniques compared shown in Table 1. Figure 3: Final histogram of the dataset showing the number of samples (headlines) per type of re- 5 Experiments action. We evaluated the final predictor with headlines from recent posts and results of a random sample 45 Table 1: Accuracies per predictive model com- According to the results the most common reac- puted by using Stratified 10-fold cross validation. tion are “haha” and “love”. These are more likely LDA was good enough but it required a very high to produce more reaction votes and therefore be- number of topics. Doc2Vec wasn’t good enough come more popular. However, headlines that pro- with the parameters we used, it could be improved. duce an “angry” reaction are more likely to pro- duce more comments. Predictive Model Accuracy Among the headlines there were other types TF-IDF + Linear SVC 0.7245 of content related with celebrities or news which TF-IDF + Stochastic Gradient Descent 0.6925 0.68 mostly depends on the date the entry is published. TF-IDF + Multinomial Naive Bayes LDA + Linear SVC 0.6525 A more specific type of content could produce bet- Doc2Vec + Linear SVC 0.4278 ter results, but it could is hard to find enough “sad” and “angry” reactions. It could be possible to obtain a better result by of them are shown in Table 2. As expected, ac- changing the parameters of the classifiers and fea- cording to these results we got that headlines that ture models. Also, we tried mostly with BoW contain highly frequent words are more likely to techniques, there should be a better way to extract belong to a specific type of reaction, and it could features and considering the grammar structure. certainly fail awkwardly because of that. Moreover, sentiment analysis techniques could be We created a test dataset containing headlines used to get better assumptions on how to separate created within “2016-08-29” and “2016-09-04” to headlines by the type of reaction. see its drawbacks, it gets a pretty high accuracy for We dealt with a multi-class classification where this task but it fails classifying “wow” and “angry” the target takes one of 5 reactions, the highest reactions, as shown in Figure 4. or most voted reaction. We could have used the number of votes each of those reactions had as a discrete probability distribution. Finally, a deep learning approach might get a better performance. References Jonah Berger and Katherine L Milkman. 2013. Emo- tion and virality: what makes online content go vi- ral? GfK Marketing Intelligence Review, 5(1):18– 23. Jonah Berger. 2011. Arousal increases social trans- mission of information. Psychological science, 22(7):891–893. Figure 4: Confusion matrix showing on what reac- Bo Gao, Bettina Berendt, and Joaquin Vanschoren. 2015. Who is more positive in private? analyz- tions the predictive model has trouble. The more ing sentiment differences across privacy levels and intense the color blue, the more well-classified demographic factors in facebook chats and posts. headlines. In 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 605–610. IEEE. 6 Conclusions Shoubin Kong, Qiaozhu Mei, Ling Feng, Fei Ye, and Zhe Zhao. 2014. Predicting bursts and popularity We built a dataset of 9072 Facebook posts ex- of hashtags in real-time. In Proceedings of the 37th tracted from several fan-pages and from these international ACM SIGIR conference on Research posts we analyzed a text corpus of titles linking & development in information retrieval, pages 927– to a website. Then, we were able to discover fea- 930. ACM. tures that can be used to obtain a good headline Jong Gun Lee, Sue Moon, and Kavé Salamatian. 2010. by considering the type of reaction it produces on An approach to model and predict the popularity of people. online contents with explanatory factors. In Web 46 Table 2: Some predicted values, the first 10 are headlines not used in training, and the last ones are random headlines. ypred is the predicted reaction using the TF-IDF + Linear SVC model, yprob the sorted predicted probabilities per reaction using the TF-IDF + SGD model, and ytrue is the actual reaction. Headline ypred yprob ytrue Huma Abedin And Anthony haha haha:0.519, sad:0.24, love:0.195, angry:0.0461, wow:0.0 haha:86, sad:77, love:38, wow:34, angry:4 Weiner Are Separating Apartment complex warns residents about clown trying sad sad:0.567, haha:0.19, love:0.171, wow:0.0712, angry:0.0 wow:1571, angry:770, sad:66, haha:58, love:28 to lure kids into woods California Lawmakers Pass Bill Requiring Prison angry angry:0.771, love:0.229, wow:0.0, sad:0.0, haha:0.0 love:1958, angry:751, wow:48, sad:42, haha:6 Sentence After Stanford Sex Assault Case Triumphant Tarantula sad sad:0.357, haha:0.303, love:0.227, wow:0.114, angry:0.0 wow:1388, haha:382, love:156, sad:15, angry:5 Survives Being Eaten By Toad Scientists discover oldest wow wow:0.996, sad:0.00439, love:0.0, haha:0.0, angry:0.0 wow:189, love:60, haha:15, sad:2, angry:2 fossils on Earth Wilderness Expert Finds A Giant Black Slug Which Is haha haha:0.527, wow:0.445, angry:0.0285, sad:0.0, love:0.0 wow:837, love:169, haha:144, sad:14, angry:1 Way Bigger And Cooler Than You Thought — 9GAG.tv Georgia teacher dreams up dice game about slavery. It love love:0.855, haha:0.142, wow:0.00315, sad:0.0, angry:0.0 angry:107, wow:96, haha:42, love:9, sad:7 didn’t go well. Latino Trump Supporter Warns Of ’Taco Trucks On haha haha:0.486, love:0.238, angry:0.198, sad:0.0773, wow:0.0 haha:1524, love:231, wow:52, angry:30, sad:4 Every Corner’ Isis has nothing to do with angry angry:0.757, haha:0.243, wow:0.0, sad:0.0, love:0.0 - islam Justin Bieber is dead haha haha:0.61, sad:0.375, angry:0.0141, wow:0.0, love:0.0 - Intelligence and Intelligent Agent Technology (WI- IAT), 2010 IEEE/WIC/ACM International Confer- ence on, volume 1, pages 623–630. IEEE. Arman Khadjeh Nassirtoussi, Saeed Aghabozorgi, Teh Ying Wah, and David Chek Ling Ngo. 2015. Text mining of news-headlines for forex market pre- diction: A multi-layer dimension reduction algo- rithm with semantics and sentiment. Expert Systems with Applications, 42(1):306–324. SSK Rastogi, Rohit Singhal, and Rajeev Kumar. 2014. A sentiment analysis based approach to facebook user recommendation. International Journal of Computer Applications, 90(16). 47