The numerical outcome of an ofline experiment involving diferent recommender systems should be interpreted also considering the main characteristics of the available rating datasets. However, existing metrics usually exploited for comparing such datasets like sparsity and entropy are not enough informative for reliably understanding all their peculiarities. In this paper, we propose a qualitative approach for visualizing diferent collections of user ratings in an intuitive and comprehensible way, independently from a specific recommendation algorithm. Thanks to graphical summaries of the training data, it is possible to better understand the behaviour of diferent recommender systems exploiting a given dataset. Furthermore, we introduce RS-viz, a Web-based tool that implements the described method and that can easily create an interactive 3D scatter plot starting from any collection of user ratings. We compared the results obtained during an ofline evaluation campaign with the corresponding visualizations generated from the HetRec LastFM dataset for validating the efectiveness of the proposed approach.
Being able to correctly interpreting the results obtained during
an ofline evaluation of diferent recommender systems is of
paramount importance for understanding the quality of the suggested
items [
Diferent metrics have been proposed in literature to summarize the main characteristic of a rating dataset, i.e. sparsity or entropy. However, we argue that such metrics are not suficient for comparing datasets in a reliable way, as many other facets should be taken into account. For example, it is not possible to understand the rating behaviors of specific groups of users nor the popularity of the most rated items by only looking at some general statistics computed on the whole dataset.
A possible solution to this problem could be represented by
data visualization techniques [
In this paper, we propose a novel qualitative approach based on data visualization for creating a graphical summary of any collection of user preferences. This method is useful for visually identifying similarities and diferences among the available datasets. In fact, we argue that if two datasets result in similar visualizations, the behavior of diferent recommender systems relying on them will be consistent. Furthermore, we present a Web-based tool, named RS-viz, for easily constructing the proposed visualization and comparing rating datasets in an intuitive way. RS-viz is freely available on GitHub1 and its usage is described in an introductory video.2
Diferently from the plotting capabilities already available in specialized software like Matlab or Scilab, our approach is more general, as it can be applied in a consistent way by diferent users on any dataset and it can be exploited on many devices without the need of installing specific tools.
The remainder of this paper is structured as follows: in Section 2 we review existing visualization techniques in the context of rating datasets, while in Section 3 we present the approach used to construct the scatter plot and we describe the implementation details of the Web-based tool RS-viz. In Section 4, we comment on the outcome of an evaluation campaign designed to validate the proposed method. Finally, in Section 5, we provide the conclusions and we outline possible future works. 2
Diferent authors have proposed to create interactive visualizations for qualitative evaluating the goodness of the recommended items or helping the users to identify the most relevant suggestions.
For example, Kunkel et al. [
Çoba et al. [
Gil et al. [
In contrast, Cardoso et al. [
All the reviewed approaches are based on popular recommender systems. To the best of our knowledge, this paper represents the ifrst formal attempt to visualize the ratings available in an ofline dataset independently from a specific recommender system. 3
In this section, we first describe the algorithm that we devised for creating a scatter plot that represents a rating dataset (Section 3.1), then we introduce the implementation details of RS-viz (Section 3.2). 3.1
In order to visually represent the rating matrix associated with a generic dataset we opted for a 3D scatter plot. The rationale behind this choice is that each point in the visualization could intuitively represent a single rating from the dataset: the value of the x-axis is the identifier of the user, the value of the y-axis is the identifier of the item, while the value of the z-axis is the rating itself, if it is expressed on a numerical scale.
However, it is easy to foresee that this approach cannot handle complex datasets with many preferences, as it requires one point for each rating. If the ratings available are only binary, a traditional scatter plot would sufice.
For these reasons, we decided to create a more compact representation of the rating matrix before visualizing it. In details, we first associated the users and the items with internal numerical identiifers according to their frequency of appearance in the dataset. For example, we associated the most rated item with the value of 1, and the second most rated item with the value of 2. The same approach was followed for ordering the identifiers of the users according to the number of ratings that they expressed.
Then, we linearly normalized such identifiers within an interval ranging from 0 to a user provided value, which represents the size of a squared rating matrix in a transformed space. Finally, we binarized the ratings from the original dataset according to a user provided threshold and we counted, for each cell of the transformed matrix, the number of positive preferences associated with that cell.
For example, if the user 40 expressed a preference for the item 360 in a dataset where the number of users is 941, the number of items is 1446, and the number of normalized users and items is equal to 100, that rating would be associated with the cell (4, 24) because ⌊40 ÷ 941 × 100⌋ = 4 and ⌊360 ÷ 1446 × 100⌋ = 24.
Therefore, the value of the z-axis represents the number of positive ratings associated with a sub-matrix of the original dataset, sorted by item popularity and user activity. In order to enhance the readability of the visualization, we also represented the value of the z-axis using a logarithmic color scale.
As an example of the proposed method, we report in Figure 1 and Figure 2 the scatter plots obtained from the MovieLens 100K and MovieLens 1M datasets, when the rating threshold is equal to 3, and the number of normalized users and items is equal to 100.
By looking at the values of the z-axis, it is possible to observe
in an intuitive way that MovieLens 1M contains a higher
number of popular items and of very active users. This conclusion is
consistent with the findings of other works that analyze the main
characteristics of the MovieLens datasets [
We realized a software implementation of the proposed approach as a Web-based tool, called RS-viz, which is freely available. Our visualization framework has been developed using the JavaScript programming language and it runs entirely in a user’s browser. For this reason, it can also be exploited for analyzing private datasets, as no information about them is sent to remote servers.
The user needs to visit the Web-page of RS-viz3 and select one of the built-in datasets or provide her own dataset as a CSV file. Then, she needs to specify the threshold between positive and negative ratings and the number of normalized users and items, which should be selected also considering the rating scale of the input dataset and the desired visualization density. A screenshot of the form containing the configuration parameters of RS-viz is reported in Figure 3.
After a few seconds, an interactive 3D scatter plot is constructed on the right side of the page. The user can inspect the plot by rotating the camera and finally save the result as a PNG file. 4
In the following, we report the numerical outcomes of an evaluation campaign conducted on the HetRec LastFM dataset using diferent recommendation approaches with the purpose of understanding if our visualization technique is capable of capturing the diferent characteristics of a rating dataset and to what extend they influence the recommendation coverage and accuracy. 4.1
We performed two diferent experiments with the HetRec LastFM
dataset and our evaluation framework RecLab [
In the first one, we set the rating threshold equal to 0, while in the second one, we set it equal to 1,000. For the other parameters, we used the default values of the framework: we selected a random 3http://datascience.ismb.it/rs-viz/ 4http://datascience.ismb.it/reclab/ splitting protocol, the test set size as the 20% of the dataset, and the length k of the recommended lists equal to 10.
We considered diferent recommendation approaches, namely
the most popular and random baselines and the MyMediaLite [
We computed the metrics of coverage, precision, recall, and NDCG. The results of these experiments are reported in Table 1. The same datasets obtained from HetRec LastFM by varying the rating threshold were exploited for creating two scatter plots using RS-viz, as displayed in Figure 4. 4.2
From the visualization provided in Figure 4a, we can observe that the HetRec LastFM dataset has a very diferent structure from the one of the MovieLens datasets. In fact, a limited number of items are associated with the preferences of almost all users, as it can be deduced by considering only the ratings expressed for popular items, that is the ones with low identifiers. Please note that such ratings seem not related to the identifier of the user, resulting in a scatter plot that resembles the shape of a half cylinder.
Furthermore, less popular items seem to be liked by less active users. This behavior can be observed by looking at the lower part of Figure 4a. Users with a high identifier have rated a more widespread set of items, while users with a low identifier have rated popular items more frequently.
These diferences can be easily explained if we consider the collection protocol and the domain of the dataset under analysis. The ratings in the LastFM datasets represent the number of times a user listened to a particular artist: therefore, they were collected in an implicit way and their values range from one to tens of thousands.
Also the strange area in the plot with almost no preferences is a direct result of the collection protocol, which relied on the LastFM website to obtain the top artists for a set of users. In fact, the list of artists available in the dataset is limited to 50 items for each user.
If we increase the value of the rating threshold, we can observe that the resulting scatter plot represented in Figure 4b is more similar to the ones of the MovieLens datasets, resulting in a very typical long tail distribution with respect to both the items and the users. This outcome is due to the fact that we removed ratings produced by more casual listeners.
From the numerical outcomes of the experiments, we can deduce that the User KNN and WRMF algorithms are the most appropriate ones with both the diferent rating thresholds. In general, all the recommenders available perform worse with an higher threshold. In fact, from the visualizations it is clear that the number of available preferences is much lower with respect to the MovieLens 100K dataset, as the scatter plot represented in Figure 4b is sparser than the one available in Figure 1. Because user preferences are more limited in number and fragmented, the task of any recommender system is necessarily harder.
Interestingly, the Item KNN, diferently from the User KNN, experienced a dramatic drop in all the metrics considered. This result may have been caused by the fact that a very low number of users is available for each item of the dataset. Also this characteristic can be observed from the generated scatter plot by looking at the
In this paper, we proposed a method for creating graphical summaries of any rating dataset for the purpose of enabling researchers and practitioners to better interpret the results of an ofline evaluation campaign. Furthermore, we introduced RS-viz, a Web-based tool capable of creating an interactive 3D scatter plot according to the aforementioned approach starting from a user provided CSV dataset or a built-in collection of ratings.
We validated the capabilities of such visualizations to reveal useful information by comparing the graphical representations of the HetRec LastFM dataset constructed with diferent rating thresholds with the numerical outcomes of two ofline experiments involving various recommendation techniques.
As future work, we would like to quantitatively characterize rating datasets according to diferent dimensions and place them in various categories, for example by analyzing the diversity of user preferences or the tendency to rate popular items only. This empirical categorization would enable the users of our tool to better understand the ratings available and to select the most appropriate recommendation approach according to such proprieties.
Furthermore, we would like to improve RS-viz by developing other visualization methods to enable more comprehensive analysis and to evaluate its efectiveness by checking if researchers and practitioners are able to correctly use it to explain the performance of diferent recommender systems on a particular dataset.
Finally, additional studies are needed to better understand how the proposed approach could be extended for also visualizing nonconventional datasets, for example the ones enhanced with contextaware information like spatial and temporal data.