Making session-based recommendations, i.e., recommending items solely based on the users' last interactions without having access to their long-term preference proles, is a challenging problem in various application elds of recommender systems. Using a coarse classication scheme, the proposed algorithmic approaches to this problem in the research literature can be categorized into frequent paern mining algorithms and approaches that are based on sequence modeling. In the context of methods of the laer class, recent works suggest the application of recurrent neural networks (RNN) for the problem. However, the lack of established algorithmic baselines for session-based recommendation problems makes the assessment of such novel approaches dicult. In this work, we therefore compare a state-of-the-art RNN-based approach with a number of (heuristics-based) frequent paern mining methods both with respect to the accuracy of their recommendations and with respect to their computational complexity. e results obtained for a variety of dierent datasets show that in every single case a comparably simple frequent paern method can be found that outperforms the recent RNN-based method. At the same time, the proposed much more simple methods are also computationally less expensive and can be applied within the narrow time constraints of online recommendation.
•General and reference →Evaluation; •Information systems →Recommender systems; •Computing methodologies →Neural networks; Rule learning;
Making recommendations solely based on a user’s current session
and most recent interactions is a nontrivial problem for
recommender systems. On an e-commerce website, for instance, when
a visitor is new (or not logged in), there are no long-term user
models that can be applied to determine suitable recommendations
for this user. Furthermore, recent work shows that considering the
user’s short-term intent has oen more eect on the accuracy of the
recommendations than the choice of the method used to build the
long-term user proles [
e problem of predicting the next actions of users based solely
on their sequence of actions in the current session is referred to
in the literature as session-based recommendation. A number of
algorithmic approaches have been proposed over the years to deal
with the problem. Early academic approaches, for example, rely
on the detection of sequential paerns in the session data of a
larger user community. In principle, even simpler methods can be
applied. Amazon’s “Customers who bought . . . also bought” feature
represents an example that relies on simple co-occurrence paerns
to generate recommendations, in that case in the context of the
very last user interaction (an item view event). A number of later
works then explored the use of Markov models [
Today, RNNs can be considered one of the state-of-the-art
methods for sequence learning tasks. ey have been successfully
explored for various sequence-based prediction problems in the past
[
While these results indicate that RNNs can be successfully
applied for the given recommendation task, we argue that the
experimental evaluation in [
e goal of this work is to shed light on these questions and in the remainder of this paper we will report the detailed results of comparing a state-of-the-art RNN-based method with a number of computationally more ecient paern mining approaches in dierent dimensions.
In session-based recommendation problems, we are given a
sequence of the most recent actions of a user and the goal is to nd
items that are relevant in the context of the user’s specic
shortterm intent. One traditional way to determine recommendations
given a set of recent items of interest is to apply frequent
pattern mining techniques, e.g., based on association rules (AR) [
A dierent way of nding item-to-item correlations is to look
for sessions that are similar to the current one (neighbors), and to
determine frequent item co-occurrence paerns that can be used in
the prediction phase. Such neighborhood-based approaches were
for example applied in the domains of e-commerce and music in
[
Dierently from such paern- and co-occurrence-based
techniques, a number of recent approaches are based on sequence
modeling using, e.g., Markov models. e main assumption of
Markovmodel-based approaches in the context of session-based
recommendation is that the selection of the next item in a session is
dependent on a limited number of previous actions. Shani et al. [
More recent approaches to sequence modeling for session-based
recommendation utilize recurrent neural networks (RNN). RNNs
process sequential data one element at a time and are able to
selectively pass information across sequence steps [
In the work presented in this paper, we will compare dierent existing and novel paern-mining-based approaches with a stateof-the-art RNN-based algorithm. 3 3.1
3.1.1 RNN Baseline. gru4rec is an RNN-based algorithm that
uses Gated Recurrent Units to deal with the vanishing or exploding
gradient problem proposed in [
scoreknn¹i; sº = Σn 2Ns sim¹s; nº 1n ¹iº (1)
In our experiments, we tested dierent distance measures to
determine the similarity of sessions. e best results were achieved
when the sessions were encoded as binary vectors of the item space
and when using cosine similarity. In our implementation, the set
operations, similarity computations, and the nal predictions can
be done very eciently as will be discussed later in Section 4.2.2.
Our algorithm has only two parameters, the number of neighbors
k and the number of sampled sessions m. For the large e-commerce
dataset used in [
3.1.3 kNN Temporal Extension – tknn. e knn method, when using cosine similarity as a distance measure, does not consider the temporal sequence of the events in a session. To be able to leverage the temporal information within the knn technique, we designed an additional temporal-ltering heuristic for it. e proposed tknn method uses the same scoring scheme as the knn method (Equation 1). e only dierence is that, given the current session s, we consider item i as being recommendable only if it appears in the neighbor session n directly aer a certain item. In our implementation, that certain item is the last item of the current session s. Technically, we therefore use a slightly dierent implementation of the indicator function of Equation 1: 1n ¹iº = 1 if neighbor session n contains i and ¹j; iº is a subsequence of n, where j is the last item of the current session and thus the basis to predict the next item.
3.1.4 Simple Association Rules – ar. To assess the strength of simple two-element co-occurrence paerns, we included a method named ar which can be considered as an association rule technique with a maximum rule size of two. Technically, we create a rule rp;q for every two items p and q that appear together in the training sessions. We determine the weight, wp;q , of each rule simply as the number of times p and q appear together in past sessions. Given the current session s, the ar score of a target item i will be then computed as scorear¹i; sº = wi; j 1AR ¹ri; j º (2) where j is the last item of the current session s for which we want to predict the successor and AR is the set of rules and their weights as determined based on the training data. e indicator function 1AR ¹ri; j º = 1 when AR contains ri; j and 0 otherwise.
3.1.5 Simple Sequential Rules – sr. e sr method is a variant of ar, which aims to take the order of the events into account. Similar to the ar method, we create a sequential rule for the co-occurrence of every two items p and q as rp;q in the training data. is time, however, we consider the distance between p and q in the session when computing the weight of the rules. In our implementation, we use the multiplicative inverse as a weight function and set wp;q = 1x , where x is the number of items that appear between p and q in a session. Other heuristics such as a linear or a logarithmic function can also be used. In case that those two items appear together in another session in the training data, the weight of the rule in that session will be added to the current weight. We nally normalize the weight and divide it by the total number of sessions that contributed to the weight. Given the current session s, the sr score of a target item i is then computed as where j is the last item of session s and SR is the set of sequential rules. e indicator function 1S R ¹rj;i º = 1 when SR contains rj;i and 0 otherwise.
3.1.6 Hybrid Approaches. We made additional experiments with several hybrids that combine dierent algorithms. At the end, a weighted combination of the two normalized prediction scores of the algorithms led to the best results in our experiments. 3.2
We performed experiments using datasets from two dierent domains in which session-based recommendation is relevant, namely e-commerce and next-track music recommendation. e source code and the public datasets can be found online.2
3.2.1 E-commerce Datasets. For the e-commerce domain, we
chose the ACM RecSys 2015 Challenge dataset (RSC) as used in
[
3.2.2 Music Datasets. We used (a) two datasets that contain listening logs of several thousand users and (b) two datasets that comprise thousands of manually created playlists.
Listening logs: ese used datasets are (almost one-year-long)
sub-samples of two public datasets. First, we created a subset of
the #nowplaying dataset [
Playlists: Generally, music playlists are dierent in nature from
listening logs and e-commerce user logs in various ways.
Nonetheless, they are designed to be consumed in a listening session and the
tracks are oen arranged in a specic sequence. e used playlist
datasets come from two dierent music platforms. e
Art-of-theMix dataset (AotM) was published by [
e dataset statistics are shown in Table 1. e total number of sessions is larger for the e-commerce datasets. However, the number of unique items in the music datasets, which corresponds to the number of tracks included in the playlists or the number of played tracks in the listening sessions, is higher than the number of items in e-commerce datasets. scoresr¹i; sº = wj;i (3) 2hp://ls13-www.cs.tu-dortmund.de/homepage/rectemp2017 e music sessions are on average longer than the e-commerce sessions.3 e last row of Table 1 shows the average number of unique items in each session (“Avg. I/S”). Comparing this value with the average session length (“Avg. E/S”) indicates what we call the item repetition rate in each dataset. Including the same track more than once in a playlist is comparably uncommon. Listening to a track more than once during a listening session is, however, common. e dierence between the average session length and the average number of items in each session for the e-commerce dataset indicates that re-occurring of the same item in a session is common in the e-commerce domain.
3.2.3 Evaluation Protocol. e general task of the session-based
recommendation techniques in our experiment is to predict the
nextitem view event in a shopping session or to predict the next track
that is played in a listening session or is included in a playlist. To
evaluate the session-based algorithms, we use the same evaluation
scheme as in [
Our rst experiment used the exact same setup as described in [
In the subsequent sections, we then report the results of our
comparison using the sliding-window validation scheme described
above with recommendation list lengths varying from 1 to 20. In all
experiments, we tuned the parameters for the dierent algorithms
using grid search and optimized for HR@20 on validation sets
(subsets of the training sets). gru4rec was only optimized with
100 layers as done in [
4.1.1 Results Using the Original Evaluation Setup. Table 2 shows
the results ordered by the hit rate (HR@20) when using the
original setup. We could reproduce the hit rate and MRR results from
[
RSC 3Note that each session in the TMall dataset is dened as the sequence of actions of a user during one day which results in relatively larger average session length. 4Remember that the sessions of the TMALL dataset cover the events of one day, as the time stamps in this dataset are given only in the granularity of days.
MRR@1 MRR@2 SR AR KNN
4.1.2 E-commerce Datasets. Figure 1 shows the MRR results for the algorithms on the two e-commerce datasets, RSC and TMall. For both datasets, we can observe that most of the frequent paern methods lead to higher or at least similar MRR values as gru4rec. ere is, however, no clear “winner” across both datasets. e sr method works best for the RSC dataset. On the TMALL dataset, the knn method outperforms the others, an eect which might be caused by the longer list session lengths for this dataset.4 In both cases, however, the dierence between the winning method and the best-performing gru4rec conguration is statistically signicant. is is indicated by a star symbol in Figure 1.
4.1.3 Listening Logs Datasets. Figure 2 shows the accuracy performance of the algorithms on two selected listening logs datasets. #nowplaying AotM 0.3 SR
MRR@1 MRR@2 MRR@3 MRR@5 MRR@7 MRR@10 MRR@15 MRR@20 AR KNN TKNN GRU4REC(100,TOP1) GRU4REC(100,BPR) 30Music MRR@1 MRR@2 MRR@3 MRR@5 MRR@7 MRR@10 MRR@15 MRR@20 AR KNN TKNN GRU4REC(100,TOP1) GRU4REC(100,BPR) 8tracks * * * Similar to the e-commerce datasets, in all measurements, a frequent paern approach, namely the sr method, outperforms gru4rec. Here again, for MRR@20, the recommendations of sr are signicantly more accurate than the recommendations of gru4rec. Note that on the music datasets, we apply gru4rec(100,top1) and gru4rec(100,bpr), which use 100 hidden units and the TOP1 and BPR’s pairwise ranking loss function, respectively.5
e tknn method – the time-aware extension of knn– works always signicantly beer than the knn method on the listening logs datasets. tknn also outperforms both gru4rec congurations on the #nowplaying dataset for list lengths larger than 1.
Another observation on the listening logs datasets is that the sequence-based approaches (sr, tknn and gru4rec) work significantly beer than methods that do not consider the temporal information in data (knn and ar).
4.1.4 Playlists Datasets. Figure 3 shows the MRR results of the algorithms on the playlists datasets. On both datasets, the temporal extension of kNN, tknn, leads to the best results across all recommendation list sizes and signicantly outperforms both variants of gru4rec. e performance of all other methods, however, seems to depend on the specics of the datset. e sr method works well on both datasets. e relative performance of the ar method, however, depends on the dataset and the list length at which the measurement is made.
One interesting observation that we made for the music datasets is that the relative performance of knn strongly improves in terms of the hit rate6 when the recommendation list length is increased. is can, for example, be seen in Figure 4, which shows the hit rate results for the #nowplaying dataset. e hit rate of knn on the #nowplaying dataset that is about 3% for list length one increases 5Repeating the experiments with 1000 hidden layers for the gru4rec methods did not lead to any beer results on the music datasets. 6Generally, the hit rate results for the experiments, which we do not include here for space reasons, are similar to the MRR results.
SR
MRR@1 MRR@2 MRR@3 MRR@5 MRR@7 MRR@10 MRR@15 MRR@20
AR KNN TKNN GRU4REC(100,TOP1) GRU4REC(100,BPR) SR
AR to 24% for list length 20. At the same time, the hit rate of some of the other methods only slightly increases, e.g., from 6% to 15%. As a result, across all four investigated music datasets, knn outperforms all other algorithms in terms of HR@20. A similar trend can also be seen for ar, the other non-sequential approach.
4.1.5 Aggregated Ranking of Algorithms. To determine the
ranking of dierent algorithms based on their accuracy results (MRR@20)
across all six datasets, we applied the Borda Count (BC) rank
aggregation strategy [
4.1.6 Hybrid Approaches. We conducted a variety of additional experiments with dierent hybridization methods as described in Section 3.1.6 to analyze the eect of combining the algorithms. In general, a weighted combination of the two normalized prediction scores of a neighborhood-based and a sequence-based method led to the best results in our experiments. For instance, the combination of knn and sr with a weight ratio of 3 to 7, wh(knn,sr:0.3,0.7), outperformed all other individual algorithms on the 30Music dataset. 0.285 0.142 0.275 0.298 0.261 Another example is combining the normalized score of knn and gru4rec(100,top1), which can outperform other algorithms in terms of HR@20. e dierences between the winning hybrid approaches (printed in bold face in Table 3) and the best performing individual methods in each measurement were statistically signicant. Similar results were also achieved for the other datasets, which we do not include here for space reasons. 4.2
Since prediction accuracy might not be the only possible relevant
quality criterion in a domain [
4.2.1 Popularity Bias and Catalog Coverage. As in [
Additionally, we analyzed the catalog coverage of each algorithm by counting the number of dierent items that appear in the top-20 recommendation lists of all sessions in the test set. Overall, the recommendation lists of gru4rec and sr include more dierent items than the other algorithms. e recommendations of neighborhood methods, knn and tknn, on the other hand, focus on smaller sets of items and show a higher concentration bias. is can be explained by considering the sampling strategy of knn which focuses on a smaller subset of the sessions, e.g., those of the last few days.
4.2.2 Computational Complexity and Memory Usage. We measured the training time as well as the needed memory and time to generate recommendations for each algorithm. On a desktop computer with an Intel i7-4790k processor, training gru4rec on one split of the RSC dataset with almost 4 million sessions and in its best conguration takes more than 12 hours. is time can be reduced to 4 hours when calculations are performed by the GPU (Nvidia GeForce GTX 960).7 e knn method needs about 27 seconds to build the needed in-memory maps, see Section 3.1.2. e well-performing sr method needs about 48 seconds to determine the rule weights. A specic advantage of the laer two methods is that they support incremental updates, i.e., new events can be immediately incorporated into the algorithms. Creating one recommendation list with gru4rec needed, on average, about 12 ms. knn needs about 26 ms for this task and sr only 3 ms. 7Training the model for 6 month of data using the GPU lasts about 8 hours. #nowplaying
RSC
e raw data used for training the algorithms in this specic experiment (one split of the RSC dataset) occupies about 540 MB of main memory. e data structures used for training sr and knn occupy about 50 MB and 3.2 GB, respectively. e model created by gru4rec needs about 510 MB. Note that memory demand of gru4rec depends on the algorithm parameters and signicantly increases with the number of items. For the music and Tmall datasets, the memory demand of gru4rec exceeded the capacity of our graphics card. Running gru4rec using the CPU is multiple times slower than when a graphics card is used. 5
Our work indicates that comparably simple frequent-paern-based approaches can represent a comparably strong baseline when evaluating session-based recommendation problems. At the end, we could nd at least one paern-based approach that was signicantly beer than a recent RNN-based method. In particular the sr method was surprisingly eective, despite the fact that both learning and applying the rules is very fast.
Our results also indicates that the “winning” strategy seems to
strongly depend on the characteristics of the data sets like average
session lengths or repetition rates. Further research is still required
to understand this relationship. In our future work, we will
investigate the performance of additional session-based algorithms. ese
algorithms include both ones that are based on Markov models, e.g.,
Rendle et al.’s factorized Markov chains [