Figure 1: Distribution of users and ratings in the MovieLens dataset. 16% of users rated less than 10 movies, 26% rated between 10 and 19 movies. In most MovieLens releases (100k, 1m, 10m, ...), these 42% of users and their ratings are not included. 3% of users have 500 or more ratings and contribute 28% of all ratings in MovieLens.
ACM RecSys 2019 Late-breaking Results, 16th-20th September 2019, Copenhagen, Denmark Copyright ©2019 for this paper by its authors. Use permited under Creative Commons License Atribution 4.0 International (CC BY 4.0).
INTRODUCTION ’Data pruning’ is common practice in recommender-systems research. We define data pruning as the removal of instances from a dataset that would not be removed in the real-world, i.e. when used by recommender systems in production environments. Reasons to prune datasets are manifold and include user interests when publishing data (e.g. data privacy) or business interests. ’Data pruning’ difers from ’data cleaning’ as such that data cleaning (e.g. outlier removal) is typically a prerequisite for the efective training of recommender-system and machine-learning algorithms, whereas data pruning is not afecting the algorithm performance in itself.
A prominent example is MovieLens in most of its variations1 [
Not only the creators of datasets may prune data, but also individual researchers may do so. For
instance, Caragea et al. pruned the CiteSeer corpus for their research [
We criticized the practice of data pruning previously, particularly when only a fraction of the
original data remains [
Our criticism was based more on ’gut feeling’ than scientific evidence. To the best of our knowledge, no empirical data exists on how widely data pruning is applied, and how pruning afects recommendersystems evaluations. Also, the recsys community has not discussed if, and to what extent a) datasets should be pruned by their creators and b) whether individual researchers should prune data. We conduct the first steps towards answering these questions. Ultimately, we hope to stimulate a discussion that leads to widely accepted guidelines on pruning data for recommender-systems research.
. 3We might have missed some relevant information on pruning if that information was provided in a section other than the Methodology section.
special technique for evaluations. Please also note that using a random sample is not an example of data pruning.
To identify how widespread data pruning is, we analyzed all 112 full- and short papers published at the ACM Conference on Recommender Systems 2017 and 2018. 88 papers (79%) used ofline datasets, the remaining 24 papers (21%) conducted e.g. user studies or online evaluations. For the 88 papers, we analyzed, which datasets the authors used, whether the datasets were pruned by the original creators and whether authors conducted pruning. To identify the later part, we read the Methodology sections of the manuscripts or similarly named sections.3 This analysis was done by a single person rather quickly. Consequently, the reported numbers should be seen as ballpark figures.
To identify the efect of data pruning on recommender-system evaluations, we run six collaborative filtering algorithms from the Surprise library, namely SVD, SVD++, NMF, Slope One, Co-Clustering, and the Baseline Estimator. We use the unpruned ’MovieLens Latest Full’ dataset (Sept. 2018), which contains 27 million ratings by 280,000 users including data from users with less than 20 ratings. Due to computing constraints, we use a random sample with 6,962,757 ratings made by 70,807 users.4
We run the six algorithms on three sub-sets of the dataset, i.e. for a) the entire unpruned dataset, b) the data that would be included in a ’normal’ version of MovieLens (users with 20+ ratings) c) the data that would be ’normally’ not included in the MovieLens dataset (users with less than 20 ratings). We compare how algorithms perform on these diferent sets, and measure the performance of algorithms by Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). As the two metrics led to almost identical results, we only report RMSE. Our source code and analysis of the manuscripts is available at htps://github.com/BeelGroup/recsys-dataset-pruning.
The authors of the 88 papers used a total of 64 unique datasets, whereas we counted diferent variations of MovieLens, Amazon and Yahoo! as the same dataset. Our analysis empirically confirms what is common wisdom in the recommender-system community already: MovieLens is the de-facto standard dataset in recommender-systems research. 40% of the full- and short papers at RecSys 2017 and 2018 used the MovieLens dataset in at least one of its variations (Figure 3). The second most popular dataset is Amazon, which was used by 35% of all authors. Other popular datasets are shown in Figure 3 and include Yelp (13%), Tripadvisor (8%), Yahoo! (6%), BookCrossing (5%), Epinions (5%), and LastFM (5%). 11% of all researchers used a proprietary dataset, and 2% used a synthetic dataset.
50% of the authors conducted research with a single dataset, 31% used two datasets, and only 2% used six or more datasets (Figure 4). The highest number of datasets being used was 7. On average, researchers used 1.88 datasets. 40% of the authors used a pruned dataset, and 15% pruned data themselves. In total, 48% of all authors conducted research at least partially with pruned data. 5The actual numbers in the pruned MovieLens versions may somewhat difer given that we just used a sample, and the diferent MovieLens versions difer due to the fact that they include data from diferent time periods.
the data in this case. Follow-up research is needed to confirm the numbers, and investigate diferent options.
For users with few ratings (<8), other algorithms performed best than for users with more ratings.
The user-rating distribution in MovieLens follows a long-tail distribution (Figure 1). 42% of the users have less than 20 ratings, and these users contribute 5% of all ratings. The remaining 58% of users, with 20+ ratings, contribute 95% of all ratings. The top 3% of users – those with 500+ ratings – contribute 28% of all ratings in the dataset. Consequently, using a pruned MovieLens variation (100k, 1m, ...) is equivalent to ignoring around 5% of the ratings and 42% of users.5
There are notable diferences for the three data splits in terms of algorithm performance (Figure 2). Over the entire unpruned data, RMSE of the six algorithms is 0.86 on average, with the best algorithm being SVD++ (0.80), closely followed by SVD (0.81). The worst performing algorithm is Co-Clustering (0.90). For the subset of ratings from users with 20+ ratings – that equals a ’normal’ MovieLens dataset – RMSE over all algorithms is 0.84 on average (2.12% lower, i.e. beter). In other words, using a pruned version of MovieLens will lead, on average, to a 2.12% beter RMSE compared to using the unpruned data. But, to make this clear, the algorithms do not actually perform 2.12% beter. The results only appear to be beter because data for which the algorithms tend to perform poorly was excluded in the evaluation. The ranking of the algorithms remains the same when comparing the pruned with the unpruned data (SVD++ performs best, followed by SVD, and Co-Clustering performs worst).
We also looked at the users grouped by the number of ratings per user6. Figure 5 shows the RMSE for users with 1-9 ratings, 10-19 ratings, ... 500+ ratings. There is a constant improvement (i.e. decrease) in RMSE the more ratings user have. On average, the six algorithms achieve an RMSE of 1.03 for users with less than 20 ratings (1.07 for users with <=9 ratings; 1.02 for users with 10–19 ratings). This contrasts an average RMSE of 0.84 for users with 20+ ratings. In other words, RMSE for users in a pruned MovieLens dataset is 23% beter than RMSE for the excluded users. For SVD and SVD++, the best performing algorithms, this efect is even stronger (+27% for SVD; +25% for SVD++).
Data Pruning is a widespread phenomenon with 48% of short- and full papers at RecSys 2017/2018 being based at least partially on pruned data. MovieLens nicely illustrates an issue that probably applies to many datasets with user-rating data. In the pruned MovieLens datasets, the number of removed ratings is rather small (5%). However, these 5% ratings were made by 42% of the users. For researchers focusing on how well individual ratings can be predicted, the removed data has probably litle impact. For researchers who focus on user-specific issues, ignoring 42% of users is probably not ideal, particularly as their RMSE is 23% worse than the RMSE of the users with 20+ ratings.
When discussing data pruning, the probably most important question is whether pruning changes
the ranking of the evaluated algorithms. Other research has already shown that the ranking may
change, though that research was not conducted in the context of data pruning [
Given the current results, we propose that data pruning should be applied with great care, or, if possible, be avoided. We would not generally consider pruning as a malpractice, but certainly not a best-practice either. In some cases, especially when large parts of data are removed, data pruning may become a malpractice, though the community yet has to determine how much removed data is too much. As a starting point, we would recommend the following guidelines, though this is certainly not a definite recommendation, and more discussion in the community is needed: (1) Publishers of datasets should avoid pruning – if possible. If there are compelling reasons to prune data (e.g. ensuring privacy), these should be clearly communicated in the documentation. (2) Researchers using pruned datasets should discuss the implications in their manuscript. (3) Individual researchers should not prune data. If they feel that an algorithm may perform well on a subset of the data, they should report performance for both the entire dataset and the subset. If researchers conduct pruning anyway, they should clearly indicate this in the manuscript, provide reasoning, and discuss the implications.
It may not always be obvious where data cleaning ends, legitimate data pruning begins, and when data pruning becomes a malpractice. The community certainly needs more discussion about this issue. We are confident that with widely agreed guidelines on data pruning, recommender-systems research will become more reproducible, more comparable and more representative of ’the real world’.