Recommendation systems provide personalized product and service recommendations by learning latent user preferences. Most recommendation systems nowadays are static, and do not consider real-time factors when making recommendations. However, purchasing behaviors are easily influenced by realtime events happening in society. Such real-time events can be extracted from social media, as previous works have shown. In this paper, we propose using social media as a background information source to improve e-commerce recommendation models. In contrast to previous works that created shallow representations of social media, we propose two representations of real-time social media information, that captures the dynamics of word usage trends and evolving semantic word relations. Taking a popular neural recommendation system as the base system, we show that the attention mechanism allows us to integrate the rich, matrix-like representation of social media. We conduct experimental evaluations on a real-world e-commerce dataset and a Twitter dataset. The results show that our method of social media background representation and integration is efective in integrating social media predictiveness in recommendation models, and the representation is superior compared to several other representations.
Using a recommendation system to provide personalized service has become a popular practice
in e-commerce and online shopping platforms [
The second weakness is related to temporal context awareness. In the traditional interaction
data, there are records of users purchasing or rating items, but these data are not timed.
Intuitively, one might think that user preferences can be influenced by real-time events, and data
closer to the time of recommendation may better indicate the user’s current preference [
Some existing works have proposed to use social media to improve recommendation systems.
However, these works rely on the assumption that common users exist in social media and the
recommendation domain and can be identified [
While some interesting cases can be observed, Given the large number of words used on
Twitter, it is hard to handpick which word correlates to which e-commerce product. There
will be a lot of noise. A found correlation may be fake (false positive) and real related words
may not be found (false negative). Instead of using heuristics to discover positive cases, we
propose a more general approach. We convert social media into matrix-like representation and
fuse them with a neural network recommendation system through the well-known attention
mechanism [
Our work focuses on an e-commerce dataset collected by a specific platform, provided by
our industry partner1. However, we argue that our proposed method has generality that can
be adopted and applied to other e-commerce platforms. First, social media platforms such
as Twitter make data access available to the general public, and data can be easily collected.
Second, nowadays more and more recommendation systems use a deep neural network as the
recommendation model [
They are also shown to be superior to other representations.
The remainder of this paper is organized as the following. In Section 2, we will discuss related work. Section 3 will introduce the problem and the base solution. Then in Section 4, we will present our method for representing social media background and use attention to integrating it into the base solution. Section 5 will present our experimental evaluations, including an analysis of the dataset and experiment setups, followed by result discussions. Finally Section 6 will conclude this paper.
A number of research eforts have been made to address the problem of cold-start
recommendation using contextual information [
In addition, some works attempted to use social media as the context in recommendations.
For example, Alahmadi and Zeng proposed using linked Twitter accounts to address cold-start
recommendation [
Outside of the research field of recommendation systems, social media as a background has
been used in diferent kinds of data analysis and applications. For example, Wei et al. found
that Twitter volume spikes could be used to predict stock options pricing [
In this section, we will first introduce the problem of using social media background to support dynamic neural recommenders. Next we will briefly introduce an existing dynamic neural recommendation system, in which we will integrate social media information.
Technically, the problem a dynamic neural recommendation system tries to solve is to rank candidate items based on user preferences at a certain time. Given a product description (), a user description (), and the time , the system should make a prediction = ((), (), ), where is a score indicating the strength of preference, and is the recommendation model. This model is normally learned through supervised learning. In the implicit feedback recommendations, the training dataset normally contains a number of positive triples, ((), (), ) = 1, if user has purchased product at time , and a number of randomly sampled negative triples ((), (), ) = 0, from all triples where user have not purchased product at time .
To use social media background to support the system is to add the temporal background as an extra input to the model, so that the prediction becomes = ((), (), , ). We assume preprocessing has been done on social media texts and can be a vector or a matrix, depending on the representation method.
There is a large number of proposals for learning the recommendation model . We select the
recommendation system proposed by Wang et al. [
Assuming for each item, there is no user-item interaction past records available, and also
assuming from the contextual data, vector representations have been learned for users and
items, which are treated as () and (). The task of the cold start recommendation model
is thus to learn preference relationships between users and items based on their embeddings.
Wang et al. generalize a neural matrix factorization (NeuMF) model [
ŷui Fuly-connected
Concatenation GMF
MLP GMFItem Embedding
GMFUser
Embedding MLP ItemEmbedding
MLP UserEmbedding ItemEmbedding
UserEmbedding
This model is an ensemble of generalized matrix factorization (GMF) and multi-layer perceptron (MLP). Two copies of user embeddings and item embeddings are input into the GMF and MLP components, both of which produce an output embedding in their last layer. NeuMF concatenates the two output embeddings and runs them through a fully-connected layer to produce a prediction. The functions in this process are defined as the following: z = p ⊙ q, ︂[ p ]︂ q
z = ( (− 1(...2(2
+ 2)...)) + ), ˆ = (h · [︂ z ︂] ), z where p and p denote user embeddings for GMF and MLP, while q and q denote item embeddings for the two components. ˆ denotes prediction results.
Since the dataset contains only observed interactions, i.e., user purchase records of items, when training the model, it is necessary to bring up some negative samples, for example, by randomly choosing some user-item pairs that have no interaction. They defined the loss function as the following: =
∑︁ (,)∈∪− log ˆ + (1 − ) log(1 − ˆ), (1) (2) (3) (4) Social Media
Frequency change analysis Word co-occurrence graph construction
TrendRepresentation
Attention
Attention SemanticRepresentation
TransformedItem
Embedding GMFItem Embedding Ful y-connected Concatenation
GMF
MLP ItemEmbedding where = 1 if user purchased item , and 0 otherwise. denotes observed interactions and − denotes negative samples.
Although it is possible to make recommendations at any moment when a purchase intention is detected, we follow a more realistic scenario by changing the recommendation three times a day, i.e., in the morning, afternoon, and evening, which correspond to hour 10, 16, and 22 of the day. Since in our e-commerce dataset, each purchase is associated with a time, we can modify the target variable to incorporate timing. Specifically, we change to such that = 1 if user purchased item in the next time segment following , and 0 otherwise. The length of next time segment following hour 10 and 16 is set to 6 hours, and for hour 22 it is set to 12 hours2. In our dataset, these three time segments separate purchases records evenly, with 29,799, 28,266 and 27,898 purchases in each of the three segments.
It is an important problem to determine whether the user has purchase intention at hour or not, before making the recommendation. Here we assume this information is already obtained, for example, from the fact that the user visited the e-commerce website. We use the training label such that there is guaranteed to be an item that user will purchase for time . In other words, time s when user made no purchase at all are ignored.
The social media background in our scenario is a collection of social media posts. They need to
be transformed before they can be added to a neural recommendation model. We fuse social
media background with the base recommendation system in two steps. First we convert the
social media background to machine-readable matrix inputs. Then we use attention [
We design a method that aims to capture the ℎ or of the social media words
that comes from interesting phenomenon happening in real-world. When something interesting
or stimulating happens, some topics on social media may become trending. When this happens,
we say certain social media aspects are emerging. We capture this emergence by observing the
change in word frequency. The frequency of the social media words is taken as the count of
messages that contains the word. Thus we first obtain the frequency table of social media words
against time units. We then devise a method for emergence detection based on word frequencies.
Following an approach of previous works on social media event detection [
otherwise
With this formula, we aim to capture two phases of surges of words in social media. First, captures a new surge. Second, AND () > () captures the sustenance of a previous surge. Both phases can be considered a part of an emergence. With this calculation, we obtain for each time unit the emerging words in product sales and social media. This representation can be called bag-of-emerging-words (BOEW).
Practically, the time unit for the prediction can be set to a day, while the emergence calculation can be done on a smaller unit, such as an hour. In this way, for each day, we obtain a vector of length 24 for each word, indicating its emergence status in each hour of the day. Assuming we have || words in the dictionary . For each day, we can have a matrix that has || rows, each row is an emergence vector of a word across 24 hours.
The above method can capture the trends of individual word usages. However, words have inter-relationships, and treating words independently would lead to loss of information. As the semantics of words also change in real-time, and such changes need to be captured. Therefore we propose a graph-based method to capture the inter-relationship between words that evolves in real-time. Here we will first introduce the graph construction, and then propose a graph embedding method that can be applied in real-time.
Our graph is constructed based on the co-occurrence of words in tweets. Suppose at time we have a new collection of social media tweets . We use mutual information [32] to represent the co-occurrence behavior. Specifically, we have (1, 2) = ︂( (1, 2)| | )︂ , (1) (2) where (1, 2) is the frequency of co-occurrence of words 1 and 2, | | is the total number of tweets, and () is the frequency of occurrence of a single word . We use a threshold to determine the co-occurrence relation, such that if (1, 2) > , we create (1, 2) = _. For time units, we would have graphs, each of which can be represented as an adjacency matrix .
Graph convolutional network (GCN) [
= (ˆ ) where ˆ is a normalization of , and is the activation function (typically ReLU) for all but the output layer. Utilizing the spectral graph theory, the validity of this approach has been shown in several existing works [33, 34].
Given a series of graphs across several time units, a naive solution would be applying GCN
and generating a set of node embeddings in each graph. However, the embeddings in diferent
times would then be irrelevant to each other, and thus cannot be used for learning a single
recommendation model. Recently, a technique called EvolveGCN has been proposed to address
this problem [
While EvolveGCN can be used to generate node embeddings peculiar to each time segment, it has a drawback. The training of this network requires holistic data, with each training epoch processing all time segments, thus it cannot directly be used in real-time. To address this problem, we propose an incremental version of the EvolveGCN (IEGCN). The basic idea is that instead of processing all time segments in one epoch, we process one time segment at a time. Algorithm 1 shows the process.
Updating parameters (line 5) can be done through some pseudo learning tasks such as link prediction. While in the first few time segments, the embeddings are not accurate due to the lack of information, in later time segments the embeddings should have similar representativeness as the native EvolveGCN. With this algorithm, for each time , we can obtain a matrix , each row of which is the embedding of a word in the dictionary .
If the social media is represented as a vector, such as the average embedding of all hours, a simple way to integrate it into the model is by concatenating the vector with outputs of a middle 3This is also known as the version of EvolveGCN. There is also a less-known version. We omit it here for simplicity, but the details can be found in the referred paper.
Algorithm 1 Learning an incremental EvolveGCN (IEGCN) model INPUT: for all ∈ , nEpoch OUTPUT: +1 for all ∈ 1: for each ∈ do 2: = LSTM(− 1) 3: for each in nEpoch do 4: +1 = GCONV(, , ) 5: update parameter through a loss function 6: end for 7: end for and the loss function defined in Equation (4) is modified so that the temporal aspect is considered layer. If we denote the vector as b, a possible position to add it is in the concatenation layer where the outputs of GMF and MLP are jointed. If we do so, the prediction from the final layer becomes ⎡z ⎤ ˆ = (h · ⎣z ⎦),
b =
∑︁ (,,)∈∪−
log ˆ + (1 − ) log(1 − ˆ).
However, the above method aggregates information in a higher dimension to a lower
dimension (i.e., from a matrix to a vector), which can lead to information loss. If social media is
represented as a matrix, we can use better methods, for example, by using the attention. In recent
years, the attention mechanism in deep learning has been shown to be helpful by allowing
the model to focus on some aspects of input data [
For clarity, we will focus on the frequency-based representation matrix first. Denote the matrix as for time , The output of an attention module is thus a context vector for item c = ∑︁
a = softmax(ℎ, )
(9)
(10)
(11)
(12)
where is the vector in row , and is called attention weights. The attention weights can
be generally obtained using the following formula
where ℎ is the embedding of item , and is an attention score function calculated on ℎ and
. Several ways have been proposed to calculate attention weights. In this paper we choose a
simple approach called the attention function [
We consider that the attention mechanism is what we need for finding relevant real-time aspects with respect to an item. The model after fusing the real-time information through the attention mechanism is shown in Figure 2. The gray components are unchanged components in the base model, while the bright components are new additions or components that have changed because of the fusion.
With this extension, the concatenation layer takes the output of the encoder and thus becomes ⎡z ⎤ ˆ = (h · ⎢⎢⎣zcc ⎥⎥⎦), (14)
Essentially, fusing with real-time social background information this way allows the final layers of the model to learn the latent relationship between the social media background and user-item pairs. More specifically, the co-occurrence of real-time aspects and positive/negative instances will be captured. In the case of social media word , for example, when the model receives many times the co-occurrence of the emerging word and the purchase behavior of products containing the word , it will reinforce this pattern. By allowing attention on diferent rows in the matrix, we also capture finer context such as diferent delays in the causality between the social media background and purchase behaviors.
We conduct experiments with real-world data to test the efectiveness of our approach. For this purpose, we implement the base model and the new model with the extension to incorporate real-time social media background. The first set of evaluations is conducted by comparing these two models. We then test the advantage of our model compared to other methods that have diferent ways to represent the social media background. In this section we present the experimental setup, including datasets and implementation details, and discuss the evaluation results.
We are provided with an e-commerce dataset by our industry partner for the purpose of testing recommendation methods. The dataset is collected from a flash sales platform, which ofers discount coupons that are made available for a limited period of time, usually between 7 and 14 days, essentially making them flash sales. The dataset contains the information of several thousands of products and users who purchased the product during the flash sales events. Since the available periods of the products are short, the market is rapidly changing, and thus the majority of the products can be considered cold items that have no purchase records in an earlier period. The products include several categories of items, such as food, cosmetics, home appliances, hobby classes, and travel packages. All products are associated with text descriptions written in Japanese. Each user is associated with some user information including gender, age, and the prefecture code of their home address. The dataset is for a period of four months, between June and September 2017.
Since the products are associated with text descriptions written in Japanese, we can use word embeddings to represent products [35]. We use a natural language processing package called kuromoji4 to process the Japanese text. The package can efectively perform segmentation and part-of-speech (POS) tagging for Japanese text. We use the package to tokenize the text description and run POS tagging to select only nouns in the text. The vector representation of a product is thus obtained as the average word embeddings of the nouns in the description.
We obtain a social media dataset by collecting Japanese tweets through Twitter API5. To align with the period of the e-commerce dataset, we develop a procedure to search past tweets. In addition to the time requirement, it is also desirable that the tweets are talking about Japanese domestic afairs, which reflects the background in which the e-commerce business was operated. Our procedure is thus as the following. First, we collect a list of Japanese politician Twitter accounts6. From them we remove a few top politician accounts such as Abe Shinzo as they would attract foreign followers. Next we collect the follower of these politicians, who are expected to be Japanese citizens. Then we select from these citizen accounts whose earliest tweets are dated earlier than June 2017. This is to ensure that the accounts are active during the entire period of the e-commerce dataset. Finally, we collect tweets in the said period from these selected accounts. These tweets become our social media data in this study. In total this dataset contains about 2,464,645 tweets from 33,443 accounts. Intuitively, this social media dataset would only be weakly related to user purchase behaviors, since consumer products are not its topic of interest. But messages in this dataset are more similar to typical social media discussions. If we can use this dataset to improve recommendation performance, we can say the totality of social media indeed contains predictive hints.
We use Pytorch7 to implement the base and the extended models. Model parameters are randomly initiated. We follow the base model and use a tower pattern for the MLP component, which halves the layer size for each successive higher layer. The sizes of three layers in the MLP are thus [200, 100, 50]. The output size of GMF is set to 50. The size of the fully connected layer is set to 100. The number of hours, , to consider as embeddings in time is set to 24.
We use the Adam optimizer with an initial learning rate of 0.001. We run 50 training epochs for each model, before which model performances generally become stable. When training the model, we randomly sample 4 negative instances for each positive instance. For the base model, the negative instances (, ) are user and item that have no interaction. While for the new model, the negative instances (, , ) are user and item that have no interaction for time .
5https://developer.twitter.com/en/docs 6Such a list can be found online as political social media accounts are usually public. An example list is provided by the website Meyou with the url https://meyou.jp/group/category/politician/ 7https://pytorch.org/
We divide that dataset into a training set and a test set. The training set is for a period from June 1 to September 16, 2017, and the test set is between September 17 to 30, 2017, a period of two weeks. We remove what are so-called free items, which are time-limited discount coupons of 0 price, which anyone can get during the active period without paying a fee. These items occupy a large portion of the dataset, but they do not reveal user-item preferences, so we consider them noises. After removing the free items, the number of users, items, and interactions are shown in Table 1.
Considering the consistency of the evaluation, for each interaction in the test dataset (positive instances), we randomly sample 99 negative instances from items available of the same time segment. Combining the positive and negative instances, we have 100 candidate items in each recommendation.
We use hit-rate (HR) and Normalized Discounted Cumulative Gain (NDCG) to measure recommendation performance. HR@K is calculated as number of hits in top K recommendation number of recommendations .
HR@K measures whether the correct item is in the recommended items, but it does not consider the rank of the item. NDCG on the other hand counts the position of the correct item. It is calculated as: @ = ∑︁ 2 − 1 =1 log2( + 1) , where = 1 if the correct item is ranked in the -th position, and 0 otherwise. NDCG@K will be higher if the correct item is ranked higher in recommended items. To simulate a realistic scenario, where the recommended items are shown on a single web page to e-commerce website visitors, we choose a number of K values between 3 and 10.
We compare the base model (base) with three variations of our extended model, the one with only trend information captured by bag-of-emerging-words (BOEW), the one with only semantic information captured by incremental Evolve-GCN (IEGCN), and the one with both parts of information (BOEW + IEGCN). Recommendation performances measured as HR@K and NDCG@K of diferent models are shown in Table 2. For each measurement of three variations of the proposed model, the relative performance increases are shown below that measurement. The best-performing results are highlighted in bold font. (15) (16)
The main insight from these results is that considering social media background with our approaches generally improves the recommendation prediction. Both the trend information and semantic information are helpful in improving the accuracy, while the combination of the two achieves even better accuracy than using them separately. When using the trend information, the HR@K is improved by 1.65% to 5.9% for diferent K values. Using semantic information achieves better accuracy, with HR@K improving between 5% to 11.89% for diferent K values. Using both parts of information achieves the best accuracy in our evaluation, with HR@K improved between 7.5% to 15.7% with diferent K values. Similar trends are observed in HDCG@K results. Thus from the results we can see that both trend information and semantic information contribute to the predictiveness of social media background. Furthermore, incorporating the combination of them achieves better accuracy than using them separately, indicating that their contributions are complementary to each other.
We have shown that our method for representing and fusing the social media background can improve recommendation accuracy compared to the base model. We acknowledge that there are other ways to represent social media background, from simple to complex methods. In this set of experiments, we compare our method to several other real-time representation methods for social media background. We briefly introduce them as the following.
Bag-of-words (BOW). One of the most common methods for text representation, BOW has been the baseline in many previous researches in text mining [36, 37]. This method is time independent and word independent. To integrate it with our model, we produce a BOW representation for each word and each time , so that , is the frequency count of the word in that time period. This will generate a frequency vector in each time of length ||, which we concatenate in the concatenation layer in Fig 1.
incremental tf-idf. Term frequency-inverse document frequency (tf-idf) has been shown to be a more efective representation of texts, with the additional information of total document count [38]. The formula for tf-idf is: tf-idf(, ) = , · log
| ∈ : ∈ | where , is the frequency of word in the document , and is the total collection of documents. To apply it, we consider social media text posted in one time unit as a document. For the total set of documents, we set a look-back window of length ℎ so that = {− ℎ, ..., }. Again we can have a larger time unit and a small time unit, i.e., a day and an hour. After applying tf-idf, we can have a matrix, each row is a vector representing the tf-idf value of a word across 24 hours of the day. The matrix has || rows. Then it is integrated with the model using the same method as the trend matrix or the semantic matrix in Section 4.3. This method is time-dependent because the total set of documents depends on the current time.
incremental bi-term topic model (IBTM). Over the past two decades, topic-modeling such as Latent Dirichlet Allocation (LDA) has become a popular method for text representation [39]. Such methods consider the co-occurrence of words in texts, thus can capture word semantics. Bi-term topic model (BTM) is a new variation of LDA that use bi-term instead of single-term for calculation [40]. The representation is based on the generative assumption below: 1. Draw ∼ Dirichlet( ). 2. For each topic ∈ [1, ] draw ∼ Dirichlet( ).
3. for each biterm ∈ , draw ∼ Multinomial( ), and draw ,1, ,2 ∼ Multinomial( ). The parameters in this generative process can be learned through techniques such as Gibbs Sampling. As the result of learning, the parameter represents a distribution of probabilities on which a document is drawn from topics. The parameter represents a distribution of probabilities a topic is drawn on || words. The incremental version of the algorithm, provided by the same authors, trains a single model over a bi-term stream using an incremental Gibbs sampler. When applying to our problem, a tweet is considered a document, and all tweets posted in time are considered the total collection of documents. As the result, for each time step , we can get , which is a matrix with || rows, each of which is a vector of the probability value a word has for topics. Since it is a matrix input, we can integrate it using the same method presented in Section 4.3.
Emergence with Embedding (E-EMB). In a previous work, we proposed a model that
tries to combined both trends and semantics [
incremental graph convolutional network (GCN). GCN is the base form of Evolve GCN
[
We apply all baseline methods to the same Twitter data and generate diferent representations. And then we input the representation, either in vector form or in matrix form, to the recommendation model. Using the same test settings in the last set of experiments, the accuracy results of all compared representations are shown in Table 3.
As we can see from the table, our method steadily outperforms all other representations in all measurements. The best compared model seems to be IBTM, which achieves 0.139 for NDCG@10. Our method, however, outperforms this value by about 5%. Looking at other baselines, we see that BOW is the least efective representation. E-EMB, the previous proposed method, is comparable to GCN, but is worse than our new method. The tf-idf method, while being simple to implement, achieves a good HR@10, only 1% less than our method, although its HR@3 is poor. To conclude, while each representation has its strength and weakness, our method achieves the best performance, mostly due to its capability to consider trends and semantics at the same time.
In this paper, we propose a method to integrate social media background in dynamic recommender systems in real-time. Our method consists of a representation of social media, and an attention-based fusing method. The representation takes into account both real-time trends and evolving semantics of words. Experimental evaluations with real-world e-commerce and social media datasets show that our method is feasible, with steadily improved accuracy results achieved by the extended model. The representation is also shown to have an advantage over several other real-time representations of the social media background. Our method is suitable to be deployed practically because social media data can be easily obtained, and there is no requirement to link user accounts. We would like to make further investigations, however, because it is still dificult to tell which social media contexts are predictive for which products. To make our method easier to explain, in the future, we plan to find ways to make the relationships between social media background and product purchasing behavior more explicit. We also plan to deploy our system on real e-commerce platforms and view its impact on product sales in real-time.
This research is partially supported by JST CREST Grant Number JPMJCR21F2. microblogs, in: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, ACM, 2013, pp. 43–52. [32] H. Peng, F. Long, C. Ding, Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, IEEE Transactions on pattern analysis and machine intelligence 27 (2005) 1226–1238. [33] W. Dai, O. Jin, G.-R. Xue, Q. Yang, Y. Yu, Eigentransfer: a unified framework for transfer learning, in: Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 193–200. [34] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, M. Welling, Modeling relational data with graph convolutional networks, in: European semantic web conference, Springer, 2018, pp. 593–607. [35] V. Lampos, B. Zou, I. J. Cox, Enhancing feature selection using word embeddings: The case of flu surveillance, in: Proceedings of the 26th International Conference on World Wide Web, International World Wide Web Conferences Steering Committee, 2017, pp. 695–704. [36] W. Stokowiec, T. Trzciński, K. Wołk, K. Marasek, P. Rokita, Shallow reading with deep learning: Predicting popularity of online content using only its title, in: International Symposium on Methodologies for Intelligent Systems, Springer, 2017, pp. 136–145. [37] Y. Zhang, M. Shirakawa, T. Hara, Predicting temporary deal success with social media timing signals, Journal of Intelligent Information Systems (2021) 1–19. [38] A. Ramisa, F. Yan, F. Moreno-Noguer, K. Mikolajczyk, Breakingnews: Article annotation by image and text processing, IEEE transactions on pattern analysis and machine intelligence (2017). [39] D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet allocation, Journal of Machine Learning
Research 3 (2003) 993–1022. [40] X. Cheng, X. Yan, Y. Lan, J. Guo, Btm: Topic modeling over short texts, IEEE Transactions on Knowledge and Data Engineering 26 (2014) 2928–2941.