Next-track music recommendation is the task of automatically determining the next song to play in a music listening session. Almost all music streaming platforms on the web provide their users with such a feature today. In this work, we propose the use of language modeling techniques for this task and investigate how well these techniques perform in the context of popular and also more diverse music. For this, we implement two basic language models, one based on n-grams and the other based on a recurrent neural network. We evaluate these models on two datasets, one limited to popular music and one consisting of more diverse tracks. Further, we also compare them with a nearest-neighbor model. Our results suggest that language models perform well in the context of popular music and can be used both as a basis for more sophisticated models and as a strong comparative baseline.
H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval; H5.5 [Information Interfaces and Presentation]: Sound and Music Computing
Nowadays, almost all music streaming services on the web provide their users with a feature to automatically determine the next song to listen to, based on the user's current listening session and possibly their past listening behavior and general preferences. This feature is an example of a recommender system, and the speci c problem is known as next-track recommendation.
A single listening session (or playlist) Xi is given as a temporally ordered sequence of tracks, i.e., Xi = [t1; t2; :::; tn]. The task of a next-track recommender is then to take the current listening session of the user and predict the most likely|in other words, most appropriate|next track to continue the session with. E ectively, such a recommender can be considered as a model learning a probability distribution p(Xj ) over all possible listening sessions and the task of recommending the most likely continuation as nding the solution to continuation(Xi) = argmaxt2T p(Xi + t) (1) where T is the set of all available tracks and Xi+t is the listening session resulting from appending track t to session Xi, i.e., Xi + t = [t1; t2; :::; tn; t]. This can further be generalized to not only determine the single most likely continuation, but a list of the n most likely continuations.
Many di erent approaches for solving this task have
been proposed over the years. Quadrana et al. [
Existing approaches can be divided into those that view
listening sessions as pure lists of track identi ers, and
those that make use of additional features of tracks, including
track metadata, audio features or lyrical features. In this
work, we take the former approach|i.e., we do not make
use of any features of the tracks in a session. Speci cally, we
regard listening sessions as sentences, formed from words
given by track identi ers, as has been done before for example
by McFee and Lanckriet [
With this work, we attempt to answer the following research questions:
RQ1: Can language models perform well for the nexttrack recommendation task? RQ2: Are there di erences in performance for language models between di erent types of listening sessions/playlists and if yes, can we explain them?
The remainder of this paper is structured as follows: In Section 2, the di erent language models we propose for solving the next-track recommendation problem are explained. Section 3 details the experiments we performed to evaluate our approaches as well as the datasets we used for the evaluation. In Section 4, the results of these experiments are given and discussed. Lastly, we give a conclusion of those results in Section 5. 2.
In this section, the language models we use to tackle the next-track recommendation problem are explained. We implemented two di erent types of language models. The rst model, explained in Section 2.1, is an n-gram based model using backo smoothing. The second model, presented in Section 2.2, is a neural language model using a recurrent architecture. 2.1
An n-gram is a subsequence of length n of a longer sequence. Consider for example the listening session X = [t1; t2; t3; t4]. This session can be decomposed into three 2grams: [t1; t2], [t2; t3], and [t3; t4]. An n-gram language model is a language model which makes the assumption that a given word in a sentence only depends on the n 1 previous words in the sentence, i.e., p(tijt1; t2; :::; ti 1) = p(tijti (n 1); :::; ti 1) (2) In other words, such a language model can be trained on all n-grams of a set of sentences instead of the complete sentences. The probabilities can then be determined as p(tijti (n 1); :::; ti 1) = c(ti (n 1); :::; ti 1; ti) c(ti (n 1); :::; ti 1) (3) where c(s) is the number of occurences of sequence s in the training set. For real language models, these probabilities are often discounted slightly to \set aside" probability mass for words that were not encountered during training. We make the simplifying assumption that all tracks available to the system will be seen during training and therefore don't use discounting. Such models have the advantage of being very easy and fast to train even on very large data sets and are known to work remarkable well considering their simplicity.
One limitation of a simple n-gram model is that it is limited to a xed sequence length n. This limitation can be overcome by training multiple models for di erent values of n and then combining them via a procedure called backo . In a simple backo model, the probability p(ti) is rst determined using the largest value for n for which a model was trained. If this probability is 0, because the sequence ti (n 1); :::; ti 1; ti was never encountered in the training data, the probability is instead determined using the nextlower value for n for which a model was trained etc.
For our approach, we decided to train n-gram models
for n 2 f1; 2; 3; 4g. We also implement a backo model,
for which we further introduce a parameter kn, inspired by
Katz' backo model [
This was implemented in Python. For the implementation, a further simpli cation is possible. Since we don't actually need probability values, but only want to output candidate continuations in order of probability, it is possible to simply save lists of all continuations for pre xes of length n 1, ordered by frequency, and draw predictions from these lists. 2.2
The second kind of language model we propose is a neural language model based on a recurrent architecture. A neural language model is a neural network which produces a vector of probability values as its output. The vector has the same number of elements as there are distinct words in the modeled language (i.e., the language's vocabulary size). Every element of the vector thus corresponds to one word, and the value of that element is the probability that this word is the next work in a text, given the history of previous words that the network got as its input. In other words, a neural language model learns the probability distribution p(tijti n; :::; ti 1)
For our approach, we use a recurrent architecture. An
illustration of this architecture is given in Figure 1. Our
network takes a sequence of track identi ers as input. The rst
layer of the network calculates a semantic vector space
embedding of the inputs. This embedding is then fed into a
recurrent layer using GRU cells [
The network was implemented in Python using Keras1. For training, categorical_crossentropy was used as loss function and Adam as optimizer.
For training the network, the listening sessions in the training data are split into subsequences of a prede ned length, and all of those subsequences are fed to the network for training, using the last element of the sequence as the target value and the elements before that (the pre x ) as the input to the network. The length of those subsequences is controlled via a hyperparameter specifying how long the pre xes
should be (the subsequences are then one element longer then the pre x).
In this section, the experiments we conducted to test the applicability of language models for the next-track recommendation task are described. First, in Section 3.1, the datasets we used for the experiments are presented. Section 3.2 discusses the nearst-neighbor baseline we use to compare our models against. After that, we explain our experimental setup in Section 3.3. 3.1
As we attempt to assess the applicability of language
models to the next-track recommendation problem for di erent
types of playlists/listening sessions, we use two di erent
datasets for our experiments. As our results should also be
comparable to already existing work, we decided to use
data from sources that are used extensively in the literature [
The Last.fm dataset we used is based on the LFM-1b [
We then ltered this dataset as follows to make it meet our purposes. First, we dropped all sessions with fewer than three tracks. Then, we extracted a chunk of the rst three million remaining sessions. Lastly, we dropped all the sessions containing tracks that occur in fewer than 840 sessions in this chunk. This has two bene ts: First, it restricts the da
taset to listening sessions consisting of popular music only, which is exactly the type of sessions for which we hypothesize that language models should work well. Second, it makes the dataset small enough so that the hardware available to us can handle training our models on it.
As the second dataset, we used the AotM-2011 dataset
[
kNN Baseline
To have a baseline to compare the performance of our
models against, we also train and evaluate a nearest-neighbor
model (kNN) [
A kNN is a simple supervised machine learning algorithm. It works by nding|using some arbitrary distance metric| the k nearest neighbors in feature space to a given data point and then determining the class label (in case of classi cation) or the value (in case of regression) for that given data point based on the labels/values of those neighbors. In the case of classi cation, this is often done by (weighted) majority vote among the neighbors.
To implement this, we used the Python package implicit4. This package provides di erent distance metrics for determining neighbors. We used cosine similary and item-item for the baseline, as preliminary experiments showed them to
N-Gram 0.757 0.733 0.843 0.820 0.871 0.854 0.882 0.878 work best. As with the language models, the kNN did not make use of any audio features; for the similarity computations, the list of song identi ers in a listening session was used as the vector representation of that listening session. 3.3
We performed one experiment for every combination of model (n-grams, neural language model, and kNN baseline) and dataset (LFM-1b, AotM-2011). This gives a total of six experiments.
For each experiment, we performed a grid search over a set of reasonable hyperparameter values to nd the best parameter settings for every model-dataset combination. To eliminate possible bias resulting from a single, xed traintest split, we used 5-fold cross validation for the grid search. To make results reproducible, a xed seed was used for the cross validation. For the n-gram model, we decided to always train the model on n-grams ranging from length one to four and grid search over the backo tresholds kn ranging from zero to six. For the neural language model, we performed the grid search over the dropout rate (0.2, 0.4), and the pre x length (2, 3, and 4). For the kNN baseline, we performed a grid search over k, the number of neighbors, with possible values of 10, 20, 50, 100, 200, and 300.
In every experiment (and every fold within it) the model is rst trained on the training portion of the data. After that, the evaluation is performed as follows. For every listening session in the test portion of the data, the last track of the session is removed and the trained model is asked to produce a ranked list of candidate continuations for the shortened session. Based on these predictions and the known real continuations|those that were removed|we then calculate the metrics recall (R) and mean reciprocal rank (MRR), taking into account only the rst, the top ve and the top twenty of the predicted candidate continuations, respectively.
In this section, we present and discuss the results of our experiments. The results for the evaluation on the LFM-1b dataset are summarized in Table 2. In this table, kNNi is the kNN model with item-item distance, using k = 20, and kNNc is the same with cosine similary, using k = 50. We can see that both language models (n-grams and the neural model) work well on this dataset and signi cantly outperform the kNN baseline in all but one metric (Recall@20). Thus, we can already positively answer RQ1|language models can N-Gram Neural kNNi kNNc indeed perform well for the next-track recommendation task.
The better performance for recall at lengths 1 and 5 suggest that the language models are better than kNNs at learning and correctly ranking the exact patterns in people's music listening behavior|at least for people leaning heavily towards popular music|and are therefore more likely to nd the correct continuation as one of their rst predictions. This is further supported by the results for the mean reciprocal rank (MRR), where the language models beat kNNs event at length 20, suggesting that they tend to rank the correct continuation higher on average. For recall at length 20, the kNN model using item-item distance shows the same performance as the n-gram model and slightly better performance than the neural model. This suggests that the kNN model does indeed learn the relevant patterns, but is not as good at ranking them correctly.
Between the two language models, the n-gram model consistently outperforms the neural one on this dataset, despite its simplicity. Looking at the di erent metrics, we identify an apparent trend|the gap between the n-gram model and the neural model seems to become smaller at larger lengths. To further investigate this, we also calculated the recall for both models on the LFM-1b dataset for lengths 10 and 100. The results for this are given in Table 3. They support this trend and show that the neural model even overtakes the n-gram model at some point and outperforms it at length 100, suggesting that the neural model is better at detecting patterns, but worse at ranking them.
The results for the experiments on the AotM-2011 dataset show a di erent picture. They are summarized in Table 4. Here, the kNN model using item-item distance consistently outperforms both language models. Due to this, we can partially answer RQ2|there apparently are di erences in performance for language models on di erent types of listening sessions. The fact that language models perform better than nearest-neighbor models for our ltered Last.fm dataset but worse for the AotM-2011 dataset supports our hypothesis that language models work well in the context of users leaning towards popular music, where we deal with a limited number of frequently occurring patterns, but not so well for scenarios where users listen to more diverse music. 5.
In this work, we proposed to use common language modeling techniques to solve the next-track music recommendation task. We were able to show that these approaches perform well in the context of popular music, but not so well for scenarios in which users prefer playlists that are more diverse. Both of the language models we proposed are simple and easy to train and are therefore also well suited for use as a baseline for the development of more sophisticated models.
For future work, we will have to perform further experiments to test our hypothesis|namely that language models work well for popular music contexts|by evaluating our models on additional datasets obtained from di erent sources. Another possible direction for future work is the use of more sophisticated language models for the task. We believe that a language model that is better optimized for this speci c task could show signi cantly better performance still.