The abundance of data posted to Twitter enables companies to extract useful information, such as Twitter users who are dissatis ed with a product. We endeavor to determine which Twitter users are potential customers for companies and would be receptive to product recommendations through the language they use in tweets after mentioning a product of interest. With Twitter's API, we collected tweets from users who tweeted about mobile devices or cameras. An expert annotator determined whether each tweet was relevant to customer purchase behavior and whether a user, based on their tweets, eventually bought the product. For the relevance task, among four models, a feed-forward neural network yielded the best cross-validation accuracy of over 80% per product. For customer purchase prediction of a product, we observed improved performance with the use of sequential input of tweets to recurrent models, with an LSTM model being best; we also observed the use of relevance predictions in our model to be more e ective with less powerful RNNs and on more di cult tasks.
In social media, popular aspects of customer relationship management (CRM) include interacting with and responding to individual customers and analyzing the data for trends and business intelligence. Another mostly untapped aspect is to predict which users will purchase a product, which is useful for recommender systems when determining which users will be receptive to speci c product recommendations. Many people ask for input from their friends before buying Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
CBRecSys 2016, September 16, 2016, Boston, MA, USA. c 2016 Copyright held by the owner/author(s).
ACM ISBN 123-4567-24-567/08/06. . . $15.00
DOI: 10.475/123 4
Looking to buy a camera for X-mas, can I get some help?
Mann wish I had an iphone,I wanna be cooolll too! #sadtweet
Now thinking of getting a new phone
My baby arrived #panasonic #lumix - its waterproof but I
daren't try it hah :)
Got a #Xoom tablet today. Already rooted and master of my
domain. Thanks @koush
a higher-priced product, and users will often turn to social
media for that input [
In this paper, we present our investigations of deep learning methods for predicting likely buyers from microblogs, speci cally Twitter tweets. While many social media posts are private, semi-private or transient, and thus hard to obtain, microblogs such as tweets are generally public, simplifying their collection. With the ability to identify likely buyers, as opposed to targeting anyone who mentions a product, advertisements and products can be presented to a more receptive set of users while annoying fewer users with spam.
Microblogs cover a variety of genres, including informative, topical, emotional, or \chatter." Many of a user's tweets are not relevant, that is, indicative of whether a user is likely to purchase a product. Thus, we hypothesize that identifying tweets that are relevant to purchasing a product is useful when predicting whether a user will purchase that product. We also hypothesize that a sequence of tweets contains embedded information that can lead to more robust prediction than classi cation of individual tweets. For example, with the given order of the second and third tweets in Table 1, the user seems interested in buying a new phone, but if the order is reversed, the inference is that the user was thinking of buying a phone, but did not buy one. In this work, we investigate the use of recurrent neural networks to model sequential information in a user's tweets for purchase behavior prediction. Our use of recurrent models enables previous tweets to serve as context. introduce relevance prediction into the model for reducing the in uence from noisy tweets.
This paper is organized as follows. In the next section, we describe related work in deep learning and social media product interest prediction. Afterward, we detail our data collection and annotation process. Then, we explain the deep learning methods, discuss the experiments, and analyze results. Finally, we conclude and propose future work.
There are several works related to identifying customers
with interest in a product. A system developed by [
Our approach also di ers from these works in that access to Facebook pro les is not needed, and features are automatically learned using neural networks. In addition, most of the earlier works classify a single sentence or posting. In contrast, we predict user behavior based on past postings, that is, from a sequence of tweets, which enables preceding tweets in a sequence to provide context for the current tweet.
Our approach draws on the deep learning Recurrent
Neural Network (RNN) models which have been successfully
applied to sequential data such as sentences or speech. For
example, [
For the buy prediction task, we focused on two product categories: (1) cameras and (2) mobile devices, i.e., mobile phones, tablets, and smart watches. These are generally higher-priced products which users do not purchase frequently and therefore are more likely to tweet about.
We created a separate corpus for each category composed
of tweets by users who either: (1) bought a target product
or (2) wanted, but did not buy, a target product [
Due to poor labeling performance by Mechanical Turkers, who often were not familiar with many of the lesser-known mobile devices and cameras, we used an \expert" annotator to whom we gave many examples of labeled bought and want tweets, including trickier cases. For example, a user did not buy a camera if they were given one or if they retweeted (RT) a user who bought a camera. A sample of the labels was checked for accuracy by one of the authors. The annotator determined whether each candidate want user tweeted that they bought the product type of interest; if so, the candidate want user was labeled a buy user (Table 3). Similarly, the tweets of each candidate buy user were examined for at least one tweet indicating that the user really bought the target product type. In total, we annotated tweets from 2,403 mobile device users and 1,252 camera users. The annotator also labeled a separate random sample of tweets as relevant/not to predicting whether a target product was bought. 4.
In this section, we describe the neural network (NN)
models we implemented in Python's Theano toolkit [
The logistic regression (LR) model combines the input with a weight matrix and bias vector, feeding it through a softmax classi cation layer that yields probabilities for each class i. The class i with the highest probability is the output.
A feed-forward (FF) network enables more complex functions to be computed through the addition of a sigmoid hidden layer below the softmax. A natural extension of the FF network for sequences is a recurrent neural network (RNN), in which the hidden layer from the previous timestep is fed into the current timestep's hidden layer: ht = (Wxxt + Whht 1 + b) (1) where ht is the hidden state, xt is the input vector, W is a learned weight matrix, and b is a learned bias vector.
Thus, information from early words/tweets is preserved
across time and is still accessible upon reaching the nal
word/tweet for making a prediction. However, typically the
error gradient vanishes as the sequence becomes increasingly
long, which in practice causes information loss over long time
spans. To compensate for this, long short-term memory [
Cft = tanh(Wcxt + Ucht 1) The new memory cell Ct is generated from the combination of the candidate memory cell Cft, controlled by the input gate it through element-wise multiplication, and the previous memory cell Ct 1, modulated by the forget gate ft:
Ct = it Cft + ft Ct 1 The output gate and new hidden layer are computed by: ot = (Woxt + Uoht 1 + VoCt)
ht = ot tanh Ct
To preprocess the data, each tweet was rst tokenized
by the TweeboParser [
To predict tweet relevance, we compared the LR, FF, RNN, and LSTM models. Since the RNN and LSTM are sequential, the input was an array of token embedded vectors; for the LR model and FF networks, we summed the token embedded vectors as input. For regularization of the LR and FF models, we employed an early stopping technique, and for the RNN and LSTM networks, we incorporated dropout. 4.2
To predict whether a user will buy a product based on their tweets, we propose a con guration of neural networks that uses predicted tweet relevance in purchase prediction. The input for each user is a sequence of tweets (instead of words, as is more commonly used) enabling the preceding tweets to provide context for the current tweet. To model the information in a tweet sequence, a recurrent network (e.g., RNN or LSTM) is intuitively a good choice.
In our proposed joint model (Figure 1), tweets from a user are input as a sequence where each tweet is represented as the sum of the embedded vectors representing its words. The (optional) lower sub-network predicts the relevance of each tweet; we use the best of the four types of relevance classi cation models, the feed-forward neural net.
The buy sub-network at each time step (i.e., tweet) is composed of either an RNN or an LSTM memory cell, which is fed a tweet vector along with its predicted relevance. The maximum across each dimension of all the tweets' hidden layer outputs are fed into the softmax classi er for the nal buy/not buy prediction. The softmax will generalize well to future work on predicting other labels besides buy/not-buy.
For all experiments, each data set was split into 10 partitions for 10-fold cross-validation. Within each fold, 10% was for validation, 10% for testing, and the rest for training. In order to incorporate the FF classi er for predicting tweet relevance as a sub-network in a joint network, we trained a separate classi er for each of the 10 cross-validation partitions. We selected all the users in the training and validation sets for that partition, and trained the relevance classi er on all those users' tweets for which we had a relevance label. The predicted relevance for each tweet was used as an additional feature to predict the user's purchase behavior.
We conducted two sets of experiments: rst, predicting whether a tweet is relevant to a user's purchase behavior; second, predicting whether a Twitter user will eventually purchase a product within 60 days of tweeting about it.
For all the networks, we set the hidden layer size to 50.
The weight matrices were initialized randomly, and bias
vectors were initialized to zero. We used RMSprop [
We observe from the results shown in Table 4 that the FF model performed best. The task is harder for combined mobile device and camera data than for the two individual products, likely due to di erences between domain-speci c relevance indicators for cameras versus mobile devices.
Model Logistic
FF RNN (25%) LSTM (50%)
Mobile
Camera 79.7 81.2 80.1 80.2 78.8 80.4 79.2 77.0
Both 74.7 78.0 77.7 77.0
We explored RNNs and LSTMs for predicting whether or not a Twitter user would buy a product, since these models sequentially scan through a user's tweets. We incorporated the FF tweet relevance prediction model as a sub-network in a deep network; this model predicts a feature indicating each tweet's relevance, which is appended to the input tweets.
For each Twitter user, we used all tweets containing a product mention within a 60-day span, limited to users who wrote between ve and 100 product-related tweets; the upper limit was used to lter out advertisers. The 60 days are motivated by the assumption that companies are not interested in promoting products for longer periods, and by Figure 2, which shows that about half the users purchase what they want within 60 days.
We evaluated our models on mobile devices, cameras, and the two combined. Negative examples included Twitter users who wanted a product but did not mention buying it. We trained on their tweets from within the 60-day window before their most recent tweet that mentions wanting a product. Positive examples included users who eventually tweeted about buying a product (the buy users), but did not include the \bought" tweet or any tweets written afterward.
The 10-fold cross-validation results averaged over several runs (due to random initialization) on the expert-labeled training data are shown in Table 5. As the baseline, we trained a FF model with sums of all tokens across all tweets for each user (*-sum) as the input. We observe better performance when tweet information is input sequentially to a model with memory (*-seq) than when the sequence information is lost by summing (*-sum) tweets. That is, it is important to capture information embedded in the sequence of tweets. As expected, the LSTM consistently outperformed the RNN because it has the ability to retain information over longer time spans. We also observe that the addition of predicted relevance probabilities to the RNN model (*+Rel) improved performance over the simple RNN; however, adding tweets' relevance only improved the LSTM's performance for the harder combined product task, which may indicate that the vanilla LSTM is powerful enough to learn which tweets are relevant or not when trained on a single product.
Model FF-sum
RNN-seq RNN-seq+Rel
LSTM-seq LSTM-seq+Rel
Mobile 73.6 80.8 81.3 83.9 83.8
Camera 66.3 78.0 80.5 81.4 80.9
Both 73.4 79.3 80.1 81.5 81.7
In this work, we investigated deep learning techniques for predicting customer purchase behavior from Twitter data that recommender systems could leverage. We collected a labeled corpus of buy/not buy users and their tweets.
A FF neural network performed best at predicting whether a tweet is relevant to purchase behavior, with an accuracy of 81:2% on mobiles devices and 80:4% on cameras. We found that the use of a deep learning model that incorporates sequential information performed better than ignoring sequential information for the purchase prediction task.
Our initial work in this area has many possible extensions. While we used a 60-day window, it would be interesting to observe user purchase probability changes over time. We will also predict related behaviors, e.g., product comparison.