High quality user feedback data is essential to training and evaluating a successful music recommendation system, particularly one that has to balance the needs of multiple stakeholders. Most existing music datasets sufer from noisy feedback and self-selection biases inherent in the data collected by music platforms. Using the Piki Music dataset of 500k ratings collected over a two-year time period, we evaluate the performance of classic recommendation algorithms on three important stakeholders: consumers, well-known artists and lesser-known artists. We show that a matrix factorization algorithm trained on both likes and dislikes performs significantly better compared to one trained only on likes for all three stakeholders.
Music recommendation algorithms play a major role in what music gets listened to on streaming platforms [
Much has been written about competing classes of algorithms and metrics. Unfortunately, the growing number of proposed metrics makes it dificult to answer straightforward questions related to the satisfaction of the stakeholders of a music streaming platform. For example, what proportion of recommendations is actually liked by users? Do the algorithms serve well-known artists better than lesser-known artists? Are music consumers more or less satisfied with well-known or lesser-known recommendations? There is often a tension between metrics that aim to measure familiarity, relevance and predictability, and metrics that aim to measure fairness, diversity and serendipity. Combining these metrics into objective functions that represent the interests of the aforementioned stakeholders is challenging.
At the heart of this problem are three common assumptions in the data used for training and testing recommendation algorithms: (i) unheard songs are assumed to be disliked, though in reality unheard songs are often excellent and (ii) played songs are assumed to be liked, though they may have been passively listened to on a playlist (iii) rated songs are randomly presented to users, while they are often self-selected by the user. All of these assumptions tend to favor the well-known artists, who are more often heard, more often recommended on playlists and more often remembered at the search bar. To mitigate these concerns we built Piki 1, a music discovery tool that collects ratings through a binary choice (like/dislike) on song clips that cannot be searched or skipped. We present the Piki Music dataset with the goal of enabling researchers and practitioners from the RecSys community to explore the above challenges. We highlight the following contributions: ∗Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Presented at the MORS workshop held in conjunction with the 15th ACM Conference on Recommender Systems (RecSys), 2021, in Amsterdam, Netherlands. 1www.piki.nyc • We collect the Piki Music dataset in a way that incentivizes users to give truthful binary ratings to songs, thus mitigating the self-selection and noisy feedback biases inherent in most recommendation datasets. (Section 3). • We quantify the value of a dislike by training a matrix factorization algorithm on this binary dataset. We find that it performs significantly better than when trained on positive-only feedback for all three stakeholders (Section 4). 2
Datasets with Explicit and Implicit User feedback. Datasets consisting of user-item interactions are typically
elicited in two ways: explicitly or implicitly. Explicit feedback refers to an action that a user performs with the intention
of giving their opinion on the quality of an item, for example, giving a rating to a watched movie. Two well-known
datasets for recommendations, the Netflix Prize dataset [
Implicit feedback refers to an action that a user performs with an intention other than giving their opinion on the
quality of the item, such as clicks, purchases, etc. Implicit feedback collected by streaming apps is commonly used
to train modern recommendation systems [
Recommendations for multi-stakeholders. A recent paper calls for a paradigm shift from user-centric metrics
towards modeling the value that recommendation systems bring to their multiple stakeholders [
In the music domain, lesser-known artist have expressed many concerns, which include reaching an audience,
transparency in recommendations, localizing discovery, gender balance and popularity bias, according to a qualitative
study [
Manuscript submitted to ACM Through the Piki interface (Fig. 1), we collect binary data while incentivizing users to provide feedback in a way that is aligned with their individual tastes. The Piki Music dataset currently consists of 2723 anonymized users, 66,532 anonymized songs and 500K binary ratings and the data collection is on-going. Figure 2 illustrates the distribution of like rates across users and songs. The columns of the dataset are as following: • timestamp: a datetime variable • user_id: an anonymized user id • song_id: an anonymized song id • liked: this is the binary indicator, 1 if the song is liked, or 0 if the song is disliked. Note that the feedback consists of 39% likes and 61% dislikes. The superlike indicator, labeled 2, is included in the data, though we treat it as a like in our experiments. • personalized: this is 1 if the song was recommended based on their previous choices or 0 if the song was selected randomly. Note that the songs recommended are 66% personalized and 34% random songs. We have included this flag in the dataset, to allow mitigation of the recommendation bias of the data, though this question is not within the scope of our study. • spotify_popularity: this is the song’s artist’s popularity, a value between 0 and 100, with 100 being the most popular. It is published by Spotify for each artist, through their publicly-available API 2. The average value of the Spotify popularity in our data set is 52, so we classify songs as coming from well-known artists if the value is above this mean and as a lesser-known artist if it is below the mean. Note that this threshold corresponds to artists that have approximately 350,000 monthly listeners, which on average generates around $2000 per month, assuming this is a solo artist without a label. 3.1
Users on Piki provide explicit binary feedback, by liking or disliking music clips. Users do not have access to a search bar and thus cannot control what songs they will hear. They are asked to like or dislike 30 second music video clips. Figure 1 shows how the interface presents the songs in batches, much like a social media story. The binary nature of the Piki music data set addresses our first concern with the training data, namely, that we won’t need to treat unheard songs as disliked, since we have a set of disliked songs to train and test the algorithm.
The dislike button is unlocked after 3 seconds (see the first and second images in Fig. 1), the like button is unlocked after 6 seconds (see the 3rd image in Fig. 1) and the superlike button is unlocked after 12 seconds (see the 4th image in Fig. 1). The clip starts 40 seconds into the song and when the clip is over, 30 seconds later, the user may replay the clip before rating it. This mechanism aligns the users’ ratings with their preferences. The 3 second lock period for the dislike button ensures that all songs are given a fair chance. The 6 and 12 second lock periods guarantee that positive ratings are backed up by a meaningful time investment in the song. Only songs that truly capture a user’s attention get liked or superliked. Piki users are rewarded with micropayments each time they complete a set of ratings. Both the timing and the financial rewards help mitigate our second data concern, namely that liked songs are actively liked, not just played passively on a playlist. 4
We split the dataset into a training set T and an evaluation set E using random sampling according to 80%/20% splits stratified by user and average the results across 5 runs. Algorithms are trained on T and scores are computed on interactions in E. 2https://developer.spotify.com/documentation/web-api/reference/category-artists 4.1
Given a set of users U, songs I and ratings R, collaborative filtering aims to learn -dimension latent user vectors
∈ R|U |× and latent item vectors ∈ R|I |× from the sparse user-item rating matrix through singular value
decomposition (SVD) [
A classic algorithm from this framework is the Weighted Regularized Matrix Factorization (WRMF) [
ˆ = . min Õ ( − ˆ )2 + (, ), ∗∗ , (1) (2) (3) where is the ground true preference score for user to item , is the weight put on each observation, (, ) = || ||2 + || ||2 represents the Frobenius norms for regularizing user and items matrices, and is regularization parameter. Note that with implicit feedback datasets, is assumed to be {0, 1}, where a click is treated as 1 and missing data from the rating matrix is treated as 0.
In a setting where negative feedback is available, a generalized framework is proposed to incorporate negative
feedback during training [
2 ˆ + Õ
2 ˆ + (, ), (,) ∈T (,) ∈T (,) ∈T where T , T , T refer to positive, negative and missing user feedback, , , are weights assigned to the corresponding sets of user feedback. We highlight two types of weight schemas: • WRMF with Likes: = = 0.5, = 0. This means that we only sample from the positive and missing feedback, while ignoring the negative feedback. • WRMF with Likes and Dislikes: = = 0.5, = 0. This means that we only sample from positive and negative feedback, without the need to sample from missing data.
We implemented the WRMF algorithm based on the OpenRec library [
The evaluation dataset E can be segmented into E , E , E referring to positive, negative and missing user feedback. Popular metrics like Recall@ and Precision@ aim to quantify the relevance of playlists of length . However, they implicitly assume that E and E are indistinguishable from each other.
For a trained algorithm, all scores above a given threshold are classified as recommendations. Without loss of generality, we use the median of scores from model outputs as the threshold in our experiments. We use R to denote the songs recommended by an algorithm from E. The set of recommended songs from the well-known artists is R and the set of recommended songs from the lesser-known artists is R . It is obvious that R = R ∪ R . For each song in R, we have a corresponding binary rating of {0, 1} from the consumers. For simplicity, we use R to represent the vector for binary ratings on songs in R. Similarly, we have R and R .
• CÍonRs. uTmheeirnst:uthiteiopnroisptohratitoanhoifgthheer rperceocmismioennidseadlisgonnegdswfriothmbtehteteervuasluerateioxnpesreite nthcaetfaorrethacetcuoanllsyulmikeerds.by users: |R | • Well-known artists: the proportion of recommendations coming from the well-known artists that are actually liked by users: Í|R R | . A higher precision leads to more efective song exposure for well-known artists. • Lesser-known artists: the proportion of recommendations coming from the lesser-known artists that are Í R . A higher precision leads to more efective song exposure for lesser-known artists.
actually liked by users: |R | 4.4
We evaluate the performance of the three stakeholders on the following baselines: • Popularity: A naive baseline that always recommend more popular songs from well-known artists, i.e., R = ∅. • Anti-popularity: A naive baseline that always recommend less popular songs from lesser-known artists, i.e.,
R = ∅.
• WRMF with Likes: A matrix factorization algorithm trained on likes [
The Popularity recommender achieves a precision of 41.1% for consumers, while the Anti-popularity recommender results in a lower precision of 36.3%. As a comparison, WRMF with likes improve the consumer metric by 21.9% (table 1). Moreover, we find that WRMF with Likes and Dislikes outperforms the WRMF with Likes for all three stakeholders. The consumer metric was lifted by 18.9%, with an increase of 18.1% for well-known artists and 19.5% for lesser-known artists. This highlights the importance of binary feedback in improving the training and evaluation of recommenders. 5
We present the Piki Music Dataset and argue that it was collected in a way that addresses many of the biases of other publicly available datasets. More importantly, since the ratings are binary (in the form of likes and dislikes), we can define performance metrics for recommendation algorithms from the perspective of various stakeholders.
There are a few directions that we think future researchers using this dataset may want to explore. In the spirit of the Netflix challenge, we encourage researchers to test the accuracy of more advanced RecSys algorithms on the dataset. For example, it would be interesting to determine if a neural recommender performs better than matrix factorization for consumers, well-known artists or lesser-known artists. It may also be valuable to explore other metrics to measure Manuscript submitted to ACM the interests of the stakeholders in this study. Researchers may be also be interested in other ways to segment the artist stakeholders, across genres or other metadata associated with the songs. We are particularly interested in modeling how the algorithm’s objectives are tied to the business objectives of other important stakeholders such as streaming platforms and record labels.