The vast volumes of open data pose a challenge for users in finding relevant datasets. To address this, we developed a hybrid dataset recommendation model that combines content-based similarity with item-to-item co-occurrence. The features used by the recommender include dataset properties and usage statistics. In this paper, we focus on fine-tuning the weights of these features. We experimentally compare two feature weighting approaches: a uniform one with predefined weights and a user-driven one, where the weights are informed by the opinions of system users. We evaluated the two approaches in a study, involving the users of a real-life data portal. The results suggest that user-driven feature weights can improve dataset recommendations, although not at all levels of data relevance, and highlight the importance of incorporating target users in the design of recommender systems.
The adoption of open data policies by research institutions and
government agencies has led to a dramatic increase in the volume
of open data. Although open data brings numerous benefits, the
proliferation and the diversity of data make it dificult for users to
ifnd relevant datasets. Current data repositories primarily support
keyword and faceted search modes. These may benefit users, who
can precisely express their needs and are familiar with the data
repository, but may pose a challenge otherwise. In addition, the
search may return a long list of loosely related results, which may
aggravate the dataset discovery task. All this raises the issue of
delivering personalized dataset recommendations to users.
Recommender systems were applied in the past to assist the discovery of
scholars, articles, and citations [
In this paper, we focus on the feature weights. We deploy and evaluate two weighting models. The first uses fixed uniform weights, 2
Given a target dataset d examined by a user, we recommend its
n most relevant datasets (d1, . . . , dn ) that are ranked according to
their similarity to d. The similarity between of d and di is:
overall _sim(d, di ) =
n
X (ωi · simi (d, di )) ,
i=1
(1)
where ωi is the weight associated with a feature i and simi (d, di ) is
the similarity of d and di with respect to i. In total, we consider ten
features: title, description, keyword, activity, research field, creator,
contributor, spatial, search, and download [
As shown in Equation 1, feature-based similarity scores simi (d, di ) are aggregated linearly by using feature weights ωi . However, how should these weights be set? Will diferent weighting models afect the quality of the recommendations? We consider two weighting models. The first uses a fixed set of weights, which are defined heuristically by the system designers. For the sake of simplicity, no domain knowledge is applied, and the weights of all ten features are set to ωi = 0.1. We refer to this as the uniform weighting model. The second weighting model is a user-driven one, as it is informed by the feature importance perceptions of the target system users. We conducted a survey, which involved 151 users of a real data repository. These users were shown the 8 eight features in the above list and asked to rate their importance on a 5-Likert scale. The survey revealed that title, description, and keywords were more important features, while creators and contributors were deemed less important. These importance scores were mapped onto the feature weights, e.g., ωt itl e = 0.123 and ωcr eator s = 0.086.
Experimental setup: We evaluated the uniform and user-driven weighting models in two intra-group user studies that were approximately 2 months apart. In both studies, we showed to users a target dataset d they were familiar with, as well as with a list of 5 recommended datasets, at fixed ranks i = 1, 3, 20, 80, 100 in the list of datasets most similar to d. We showed the recommended datasets in a random order and asked the users to rate their relevance to d on a 4-Likert scale, ranging from ‘very similar’ to ‘dissimilar’. We obtained the judgments of 50 users who participated in both studies and jointly rated 82 target datasets. Thus, our results are based on 410 judgments obtained in each study. Note that in both studies every user judged recommendations by referring to the same target dataset d. That said, the 5 recommended datasets might have changed due to the diferent feature weighting model.
Results: Figures 1-left and 1-right depict the distribution of the users’ relevance judgments assigned to the recommendations produced by the uniform and user-driven weighting models, respectively. The horizontal axis represents the rank i of the recommended dataset and the vertical axis indicates the distribution of the judgments. Since the results of the two studies are similar, we also include the exact judgment distributions below the plots. It can be observed that the user-driven weighting achieves a slight improvement for datasets at rank 1. Here, 81.7% of the datasets were judged ‘highly similar’ or ‘similar’, compared to the 79.3% obtained for the uniform weighting. The diferences are more pronounced at rank 3 where the user-driven weighting was judged ‘highly similar’ or ‘similar’ in 62.2% of cases, while the uniform weighting resulted in 47.5%. The obtained judgments at ranks 20, 80, and 100 are predominantly negative, so these datasets cannot be recommended and are excluded from the analysis. We compared the user judgments obtained across the two studies by using a pairwise t-test for means. We observed statistically significant diferences at rank 3, p < 0.001 while at rank 1 the diferences were not significant.
Discussion: Although the results of both studies were comparable at most ranks, our findings suggest that the user-driven feature weighting improves the quality of the recommendations at ranks 1 and 3. To acquire a better understanding of this, we plot in Figure 2 the average similarity of the recommended datasets at various ranks. This similarity exhibits a long-tail distribution. We believe that the recommended datasets at rank 1 were related regardless of the fine-tuned weights, as the strong user support of about 80% suggests. Hence, the improvement was insignificant. However, at rank 3, the average similarity is about 10% lower than at rank 1, as reflected by the lower user support dropping to the 50-60% mark. Hence, the improvement introduced by the user-driven weighting was found to be strongly significant. We conclude, therefore, that user-driven feature weights turn out to be particularly critical in the borderline areas where the relevance of the datasets is unclear. We believe that this finding reflects the importance of the target system users’ opinions.
In this paper, we studied the importance of user-driven feature weights in producing open data recommendations. We compared their performance against the baseline of heuristically set uniform weights. The results have showed that user-driven feature weights have a positive efect on user judgments, although this finding may not necessarily be applicable at all ranks. We consider this work to provide an important argument in favor of incorporating target users in the early stage of designing a data recommender system.