=Paper=
{{Paper
|id=Vol-1743/paper4
|storemode=property
|title=Predicting Reactions to Blog Headlines
|pdfUrl=https://ceur-ws.org/Vol-1743/paper4.pdf
|volume=Vol-1743
|authors=Rel Guzman,José Eduardo Ochoa-Luna,Laura Cruz-Quispe,Elizabeth Vera-Cervantes
|dblpUrl=https://dblp.org/rec/conf/simbig/ApazaLQC16
}}
==Predicting Reactions to Blog Headlines==
https://ceur-ws.org/Vol-1743/paper4.pdf
Predicting Reactions to Blog Headlines
Rel Guzman, Jose Eduardo Ochoa-Luna, Laura Cruz-Quispe, Elizabeth Vera-Cervantes
Universidad Nacional de San Agustin
Arequipa, Peru
{r.guzmanap,eduardo.ol,lvcruzq,elizavvc}@gmail.com
Abstract classification provides limited information about
sentiment, that’s the reason we also used another
This paper describes some experiments type of classifier with which we are able to com-
carried out to measure sentiment, which pute the probability of each reaction.
we call emotional reaction, on blog head- A reaction is predicted using a predictive model
lines. We analyze a text corpus of ti- consisting of Term frequency Inverse Document
tles from Facebook entries or posts link- Frequency (TF-IDF), a Linear Support Vector
ing to a website. These titles are basically Classifier (SVC) and a Stochastic Gradient De-
headlines and we study them to understand scent (SGD) algorithm. Preliminary experiments
the relationship between article headlines were run on 14 fan-pages forming a big dataset,
and the self-reported reactions of the ar- and results show suitability of our approach.
ticles’ readers. We utilize the recently The paper is organized as follow: In Section 2
launched feature, Facebook reactions that related work is described, next in Section 3 the
enable people to express their emotional dataset is described, then our proposal is described
reactions with five emojis. These reactions in Section 4. We provide experiments and results
and headlines are gathered from different of the proposed method in Section 5. Finally, we
fan-pages, we analyze them, make an ex- provide concluding remarks in Section 6. The
ploratory data analysis and present prelim- source code to reproduce this paper is available
inary results of a reaction predictor. online 1 .
1 Introduction 2 Related Work
Sharing online content is an integral part of mod-
Among recent studies on sentiment analysis ap-
ern life, this is somewhat related with social trans-
plied on Facebook, most of them analyze public
missions which are driven in part by arousal, this
posts shared by users (Rastogi et al., 2014), (Gao
hypothesis suggests why content that evokes more
et al., 2015). Taking into account information such
of certain emotions is more shared (Berger, 2011).
as a text messages, comments, likes. But, cur-
We analyze a text corpus of titles from Facebook
rently none of them have studied its new feature,
entries or posts linking to a website. These titles
“reactions”, neither how a reaction could affect the
are headlines and we study them to understand the
popularity of content. Predicting the popularity of
relationship between article headlines and the self-
social media content has been approached from
reported reactions of the articles’ readers. We uti-
many angles. Some have even been using mea-
lize the recently launched feature, Facebook reac-
surements of items at their early popularity apply-
tions that enable people to express their emotional
ing survival analysis (Lee et al., 2010). Typically,
reactions with five emojis. Then, we propose a
the goal of those researches is to predict the pop-
simple approach to predict people’s reactions to
ularity like the number of comments that will be
textual content by analyzing emotional reactions.
generated by an article based on its content, or how
As with sentiment detection, this problem is
long will it be popular.
treated as a simple classification problem and
A common line of research focuses on predict-
achieve very high accuracy by employing various
machine learning algorithms. Although, simple 1
https://github.com/rgap/simbig2016-facebook-reactions
43
ing the spread of ideas and information using con- To predict a specific reaction to a headline, we
tent, topology and linguistic features. On twit- defined a target y which is the most voted reac-
ter, authors tend to pay special attention to senti- tion among only the reactions, not considering the
ment indicators from the content of tweets, includ- number of likes. Furthermore, we added it to the
ing counts of emoticons and strength of word-level dataset only if its number of votes was higher than
sentiments (Kong et al., 2014). 75% of the total number of votes, so we make sure
There are studies that suggest that content that people reacted in just one way, because some of
evokes either high-arousal positive emotions (awe) them could have the same number of votes per
or negative emotions (anger or anxiety) tends to be reaction. The number of reaction votes per reac-
more viral (Berger and Milkman, 2013). There- tion sorted by their means is shown in Figure 1.
fore, a headline popularity can also be measured This diagram doesn’t tell too much only that even
according to what emotion, sentiment, or reaction taking into account many fan-pages, the sad reac-
it produces. And, there are also recent studies tion isn’t the most voted. We took away the head-
that deal with sentiment analysis on headlines and lines with a most voted reaction with less than 200
short-texts (Nassirtoussi et al., 2015). votes.
3 Dataset
All posts were extracted using the Facebook
Graph API, from 14 fan-pages of popular websites
about news, science, and entertainment: Buzzfeed,
9gag, Boredpanda, Mashable, Unilad, CNN, CNN
international, DailyMail, FoxNews, Huffington-
Post, New York Times, IFeakingLoveScience,
IMDB, and Natgeo. And because Facebook re- Figure 1: Box plots sorted base on median value,
actions were launched after February 2016, the showing the number of reactions votes (total num-
dataset was sliced so that it only contains samples ber of reaction votes) per reaction and outliers.
within “2016-04-01” and “2016-07-31”. We ob-
tained a dataset of 9072 samples.
A correlation matrix is shown in Figure 2,
it shows how the “number of reactions”, “com-
3.1 Reactions
ments”, “shares”, “likes” and the reactions corre-
After an administrator posts a link directing to a late among them.
website article on Facebook, a thumbnail is gen-
erated which contains: image, description, and a 3.2 Headlines
title. A user is redirected to the article after click- We needed to get features to predict a specific
ing on the thumbnail then reads it, goes back to reaction, these features were extracted from the
Facebook and gives it a reaction, or probably does headlines. Some of the headlines are less specific
it before even clicking on it. Therefore, we ana- to a special date or event like “numbered lists” also
lyzed a text corpus of titles from Facebook posts. known as “listcicles” like:
These titles are like news headlines and we stud-
ied them to understand the relationship between • 24 Heartbreaking TV Moments That Made
them and the self-reported reactions of the articles’ You Cry Your Eyes Out.
readers. We utilized the recently launched feature, • 27 Surreal Places To Visit Before You Die.
Facebook reactions that enable people to express
their emotional reactions with one of five emojis: We selected BuzzFeed also because it creates
“love”, “haha”, “wow”, “sad”, and “angry”. We these types of posts, but not so many to take only
selected BuzzFeed as one of the fan-pages due to a BuzzFeed into account. A predictor of these types
feature its website has, a registered user is allowed of headlines would have been easier to optimize,
to comment on an article with a type of reaction but it is harder to know if a title is one of these, at
defined by the website, the types of reactions are: least not automatically, also we wanted to make it
“love”, “lol”, “fail”, “wtf”, among others but these more general. That’s the reason we gathered data
are the most voted. from many fan-pages.
44
4 Predictive Model
With the dataset we created we are certainly deal-
ing with a multi-class classification problem. And
we found a pretty straightforward model to predict
the type of emotional reaction a headline will pro-
duce. The best predictive model we found was a
TF-IDF + Linear SVC, and to get the probability
of producing each of the reactions with a TF-IDF
+ SGD, we describe more about the first model be-
cause it predicts the highest reaction with a higher
accuracy.
4.1 Feature Extraction
Figure 2: Correlation matrix showing how “love” It is very common to follow the bag-of-words
is more linearly correlated to the number of reac- (BoW) approach when extracting features from
tions and shares. An interesting insight shows how documents. One of the simplest feature model is
“angry” is more correlated to the number of com- TF-IDF. Following a BoW representation, where
ments, showing that people comment more when we call vectorization to the process of turning a
dealing with controversial topics. collection of text documents which will be the
headlines, into numerical feature vectors. This
strategy includes: tokenization, counting and nor-
malization.
Then, we figured out the dataset from “buz- In our set of headlines, some words were not so
zfeed” contained english and spanish headlines, very present hence carrying very little meaningful
spanish ones were taken away from the corpus. information about the actual contents of the head-
The headlines were tokenized and we took nor- line. Therefore, we ignored terms that have a fre-
mal words, abbreviations and words with internal quency lower than 2. We extracted features for 1-
hyphens/apostrophe. We took Unicode words like grams and 2-grams, both were joined and became
“ber” and represented them in US-ASCII charac- a unique feature vector of size 9302 for a headline.
ters so that it becomes “Uber”. More preprocess- It was not easy to extract features from the head-
ing is done like profanity censorship converting lines, we tried with Latent Dirichlet Allocation
every bad word in the token “badword”. Then we (LDA) and Doc2Vec, based on Word2Vec, which
removed common and some special stopwords, we is kind of a state-of-the-art technique.
did lemmatization and stemming.
4.2 Learning Model
A histogram with number of headlines per reac-
tion in descending order is shown in Figure 3, it A simple but accurate learning model to try first is
shows that the dataset is imbalanced. a Linear Support Vector Machine or Support Vec-
tor Classifier (SVC). It was trained and evaluated
using Stratified K-Fold Cross Validation to deal
with the imbalanced dataset and find a good ac-
curacy score for this multi-class classification task
has to be an accuracy defined by the number of
well classified samples which is the “accuracy”.
We got an accuracy of 0.7245 with a simple Lin-
ear SVC with C = 0.29, it was the highest score
among the techniques compared shown in Table 1.
Figure 3: Final histogram of the dataset showing
the number of samples (headlines) per type of re- 5 Experiments
action.
We evaluated the final predictor with headlines
from recent posts and results of a random sample
45
Table 1: Accuracies per predictive model com- According to the results the most common reac-
puted by using Stratified 10-fold cross validation. tion are “haha” and “love”. These are more likely
LDA was good enough but it required a very high to produce more reaction votes and therefore be-
number of topics. Doc2Vec wasn’t good enough come more popular. However, headlines that pro-
with the parameters we used, it could be improved. duce an “angry” reaction are more likely to pro-
duce more comments.
Predictive Model Accuracy
Among the headlines there were other types
TF-IDF + Linear SVC 0.7245
of content related with celebrities or news which
TF-IDF + Stochastic Gradient Descent 0.6925
0.68
mostly depends on the date the entry is published.
TF-IDF + Multinomial Naive Bayes
LDA + Linear SVC 0.6525
A more specific type of content could produce bet-
Doc2Vec + Linear SVC 0.4278
ter results, but it could is hard to find enough “sad”
and “angry” reactions.
It could be possible to obtain a better result by
of them are shown in Table 2. As expected, ac- changing the parameters of the classifiers and fea-
cording to these results we got that headlines that ture models. Also, we tried mostly with BoW
contain highly frequent words are more likely to techniques, there should be a better way to extract
belong to a specific type of reaction, and it could features and considering the grammar structure.
certainly fail awkwardly because of that. Moreover, sentiment analysis techniques could be
We created a test dataset containing headlines used to get better assumptions on how to separate
created within “2016-08-29” and “2016-09-04” to headlines by the type of reaction.
see its drawbacks, it gets a pretty high accuracy for We dealt with a multi-class classification where
this task but it fails classifying “wow” and “angry” the target takes one of 5 reactions, the highest
reactions, as shown in Figure 4. or most voted reaction. We could have used the
number of votes each of those reactions had as a
discrete probability distribution. Finally, a deep
learning approach might get a better performance.
References
Jonah Berger and Katherine L Milkman. 2013. Emo-
tion and virality: what makes online content go vi-
ral? GfK Marketing Intelligence Review, 5(1):18–
23.
Jonah Berger. 2011. Arousal increases social trans-
mission of information. Psychological science,
22(7):891–893.
Figure 4: Confusion matrix showing on what reac- Bo Gao, Bettina Berendt, and Joaquin Vanschoren.
2015. Who is more positive in private? analyz-
tions the predictive model has trouble. The more ing sentiment differences across privacy levels and
intense the color blue, the more well-classified demographic factors in facebook chats and posts.
headlines. In 2015 IEEE/ACM International Conference on
Advances in Social Networks Analysis and Mining
(ASONAM), pages 605–610. IEEE.
6 Conclusions Shoubin Kong, Qiaozhu Mei, Ling Feng, Fei Ye, and
Zhe Zhao. 2014. Predicting bursts and popularity
We built a dataset of 9072 Facebook posts ex- of hashtags in real-time. In Proceedings of the 37th
tracted from several fan-pages and from these international ACM SIGIR conference on Research
posts we analyzed a text corpus of titles linking & development in information retrieval, pages 927–
to a website. Then, we were able to discover fea- 930. ACM.
tures that can be used to obtain a good headline Jong Gun Lee, Sue Moon, and Kavé Salamatian. 2010.
by considering the type of reaction it produces on An approach to model and predict the popularity of
people. online contents with explanatory factors. In Web
46
Table 2: Some predicted values, the first 10 are headlines not used in training, and the last ones are
random headlines. ypred is the predicted reaction using the TF-IDF + Linear SVC model, yprob the sorted
predicted probabilities per reaction using the TF-IDF + SGD model, and ytrue is the actual reaction.
Headline ypred yprob ytrue
Huma Abedin And Anthony haha haha:0.519, sad:0.24, love:0.195, angry:0.0461, wow:0.0 haha:86, sad:77, love:38, wow:34, angry:4
Weiner Are Separating
Apartment complex warns
residents about clown trying sad sad:0.567, haha:0.19, love:0.171, wow:0.0712, angry:0.0 wow:1571, angry:770, sad:66, haha:58, love:28
to lure kids into woods
California Lawmakers Pass
Bill Requiring Prison angry angry:0.771, love:0.229, wow:0.0, sad:0.0, haha:0.0 love:1958, angry:751, wow:48, sad:42, haha:6
Sentence After Stanford Sex
Assault Case
Triumphant Tarantula sad sad:0.357, haha:0.303, love:0.227, wow:0.114, angry:0.0 wow:1388, haha:382, love:156, sad:15, angry:5
Survives Being Eaten By Toad
Scientists discover oldest wow wow:0.996, sad:0.00439, love:0.0, haha:0.0, angry:0.0 wow:189, love:60, haha:15, sad:2, angry:2
fossils on Earth
Wilderness Expert Finds A
Giant Black Slug Which Is haha haha:0.527, wow:0.445, angry:0.0285, sad:0.0, love:0.0 wow:837, love:169, haha:144, sad:14, angry:1
Way Bigger And Cooler Than
You Thought — 9GAG.tv
Georgia teacher dreams up
dice game about slavery. It love love:0.855, haha:0.142, wow:0.00315, sad:0.0, angry:0.0 angry:107, wow:96, haha:42, love:9, sad:7
didn’t go well.
Latino Trump Supporter
Warns Of ’Taco Trucks On haha haha:0.486, love:0.238, angry:0.198, sad:0.0773, wow:0.0 haha:1524, love:231, wow:52, angry:30, sad:4
Every Corner’
Isis has nothing to do with angry angry:0.757, haha:0.243, wow:0.0, sad:0.0, love:0.0 -
islam
Justin Bieber is dead haha haha:0.61, sad:0.375, angry:0.0141, wow:0.0, love:0.0 -
Intelligence and Intelligent Agent Technology (WI-
IAT), 2010 IEEE/WIC/ACM International Confer-
ence on, volume 1, pages 623–630. IEEE.
Arman Khadjeh Nassirtoussi, Saeed Aghabozorgi,
Teh Ying Wah, and David Chek Ling Ngo. 2015.
Text mining of news-headlines for forex market pre-
diction: A multi-layer dimension reduction algo-
rithm with semantics and sentiment. Expert Systems
with Applications, 42(1):306–324.
SSK Rastogi, Rohit Singhal, and Rajeev Kumar. 2014.
A sentiment analysis based approach to facebook
user recommendation. International Journal of
Computer Applications, 90(16).
47