Data science challenges allow companies, and other data holders, to collaborate with the wider research community. In the area of recommender systems, the potential of such challenges to move forward the state of the art is limited due to concerns about releasing user interaction data. This paper investigates the potential of privacy-preserving data publishing for supporting recommender system challenges. We propose a data masking algorithm, Shufle-NNN, with two steps: Neighborhood selection and value swapping. Neighborhood selection preserves valuable item similarity information. The data shufling technique hides (i.e., changes) ratings of users for individual items. Our experimental results demonstrate that the relative performance of algorithms, which is the key property that a data science challenge must measure, is comparable between the original data and the data masked with Shufle-NNN.
1htp://www.mymedialite.net/examples/datasets.html Recommender systems, privacy-preserving data publishing, data masking
We propose a masking approach, Shufling Non-Nearest-Neighbors (Shufle-NNN), which changes (i.e.,
hides) a large proportion of the values of the ratings in the original user-item matrix, R, to create
a masked data set, R′. Our work is a step towards masked data that can be publicly released for
data science challenges. For this reason, we adopt a relative-rank success criterion: given a set of
recommender algorithms to be compared, the relative ranking of the algorithms trained and tested on
R and on R′ must be maintained. Shufle-NNN is motivated by the observation that privacy-preserving
techniques [
Shufle-NNN generates a masked data by changing a large portion of values of the ratings in
a user’s profile. We chose a data shufling technique because of its previous success in masking
numerical data in other domains [
Shufle-NNN works as follows. Our overall strategy is to shufle ratings in a way that maintains key item-item similarities in the data set. First, we determine the item neighborhoods, i.e., the most similar item for every item, and join them, giving us an overall set of critical items. Then, we shufle ratings for items not in this set (non-critical items, i.e., the rest of the items). Shufle-NNN has two parameters. We fix the neighbhorhood size k = 40, since this value has been shown to be efective in practice1. Given k, we set θ (a minimum threshold on item-similarity that must be met for inclusion in a neighborhood) by running some test rankings. Our exploratory experiments showed that a range of θ values can be efective, and that θ can be determined using a subset of the algorithms to be ranked (meaning that it can be set in-house before releasing data).
Recall that the goal of this paper is not to demonstrate the absolute performance of the algorithms,
but rather to evaluate if the relative performance of algorithms is the same on the original and on
the masked data. For this purpose, we need a selection of classic recommender algorithms, ranging
from baselines that are known not to yield state-of-the-art performance, to current algorithms.
We carry out experiments on both ranking prediction, and on classic rating prediction tasks. The
algorithms for ranking prediction are: Most Popular, KNN is user/item-based K-Nearest Neighbor.
(BPRFM) [
The experiments are implemented using WrapRec [
We train and test the algorithms for ranking prediction both on the original data and and the masked data. Results are given in Table 2. It can be seen that the comparative algorithm ranking is maintained. In other words, for all cases, the best algorithm on the masked data is also the best algorithm on the original data and the worst algorithm on the masked data is also the worst algorithm on the original data. These results demonstrate the success of Shufle-NNN. In Table 2, we report results for specific values of θ . However, relative ranking is actually maintained for a range of values of θ (not shown here). In addition to Recall@5 and Recall@10, we found also that Precision@5/ Precision@10 maintains the same relative ranking of algorithms.
Interestingly, when we tested our algorithms on empirical data (which we generate by replacing individual values with values drawn randomly from the global distribution of ratings in the original data), the comparative ranking was also preserved. This means that Shufle-NNN is suficient, but
3 4 Movielens data
5 1 2 3 4 GoodBook data
5 not actually necessary in the case of Top-N recommendation. Our conclusion from these results is that (at least for these data sets) the values of the ratings are not important if the goal is a relative ranking among algorithms. What is important is that our masked data sets maintain information on which items were rated.
To dig deeper, we carry out rating prediction experiments. Results are reported in Table 3. Here, we observe that the relative ranking is well maintained between the original data and masked data (although not perfectly). Our focus here is on relative performance, but it is interesting to note that the absolute RMSE on the masked data remains within 5% of its value on the original data. In contrast to the ranking-prediction results, we found that the empirical distribution does less well in maintaining the ranking than Shufle-NNN.
Next, we discuss the ability of Shufle-NNN to hide ratings. A rating is considered hidden if its value
changes between the masked data and the original data [
Then, we look at the impact of masking at the user level. Figure 1 illustrates the relationship between average user ratings before and after masking. Figure 2 shows, for diferent level of hidden ratings, for how many users that level was achieved. The average user rating of the masked data is 80 70 10 0 17500 a t a15000 d k o12500 o B od10000 o G f 7500 o s r 5000 e s U2500 Figure 2: Average user ratings original vs. masked data (masked using θ = 0.4 for MovieLens; θ = 0.15 for GoodBook). impacted, but still correlated with the original values. The protection varies per user, but is relatively high. For the GoodBook data, there is a large number of users have a hiding level equal to zero. On average these users have rated less than 30 books.
Our overall conclusion is that data masking has great potential for data science challenges. It is possible to develop a masking approach, such that masked data can be used to train and test algorithms with litle impact on the relative performance of algorithms. Shufle-NNN provides valuable evidence about what information can be removed from the user-item matrix and what information should be maintained. When the critical items list is larger than the list of items that is shufled, it is easier to reconstruct the original values. Future work will investigate the dificulty of reconstructing the original values from the shufled data, which is an issue important to consider in cases where the critical items are in the majority. We note that even modest levels of rating hiding can support deniability.