In this paper, we address the essential yet complex task of evaluating Recommender Systems (RecSys) across multiple datasets. This is critical for gauging their overall performance and applicability in various contexts. Owing to the unique characteristics of each dataset and the variability in algorithm performance, we propose the adoption of efect-size-based meta-analysis, a proven tool in comparative research. This approach enables us to compare a “treatment model” and a “control model” across multiple datasets, ofering a comprehensive evaluation of their performance. Through two case studies, we highlight the flexibility and efectiveness of this method in multi-dataset evaluations, irrespective of the metric utilized. The power of forest plots in providing an intuitive and concise summarization of our analysis is also demonstrated, which significantly aids in the communication of research findings. Our work provides valuable insights into leveraging these methodologies to draw more reliable and validated conclusions on the generalizability and robustness of RecSys models.
Within the fast-evolving domain of Recommender Systems (RecSys), broad adoption across
various industries has prompted researchers to strive for improvements in general-purpose
methods. Often, however, the efectiveness of improvements introduced by these novel methods
is confined to highly specific experimental settings, encompassing particular datasets,
evaluation measures, and baselines. As such, these enhancements do not necessarily translate into
broad applicability across diferent contexts or problem domains [
Evaluating recommender systems using multiple datasets is a complex process due to each
dataset’s unique characteristics and the algorithms’ variability. The performance of an algorithm
Perspectives on the Evaluation of Recommender Systems Workshop (PERSPECTIVES 2023), September 19th, 2023,
can vary greatly, excelling in one dataset while falling short in another, particularly when
there are significant diferences in the properties of the datasets, such as the ratio of users
to items or the density of ratings [
In this study, we examine the value of efect-size-based meta-analysis as a tool for comparative
research [
In summary, this paper makes the following key contributions: • We propose the use of efect-size-based meta-analysis as a robust approach for multidataset evaluations. • We demonstrate the practical utility of this approach through two case studies, one involving an incremental treatment model and the other involving a more sophisticated update. • We provide code and data of our experiments publicy available under:
https://github.com/MeteSertkan/meta-analysis-based-recsys-eval
Evaluating recommender systems poses inherent challenges, including varying algorithm
performance across diferent datasets, divergent evaluation goals, and the complexity of choosing
appropriate metrics for comparison [
Yet, ofline experiments still serve as a crucial initial step towards comprehensive evaluation. Several toolkits like RecPack [12], Elliot [13], Cornac [14], and RecBole [15] have been introduced for reproducible experimentation and evaluation. These tools provide the capabilities for easy, reproducible experiments, ofer built-in baselines, models, data, and facilitate the evaluation and comparison of models using various metrics. Evaluating recommender systems over multiple datasets is essential for a comprehensive understanding of their robustness and generalizability. However, the unique characteristics of datasets, metrics, and performance variations across datasets complicate drawing reliable conclusions and can introduce subjective biases. For example, it’s not valid to average scores across multiple datasets as scores are task-dependent and can only be compared within one task.
Despite their invaluable contributions, these tools do not fully address the need for statistically
robust comparison and aggregation methods across diverse datasets and scenarios. This work
proposes the use of efect-size-based meta-analysis [
We utilize efect-size-based meta-analysis to contrast the eficacy of a treatment model with a
control model across multiple datasets. The treatment model could be an updated version of the
control model, a new model, or one trained with additional data, while the control model acts
as the standard for comparison. We consider the raw mean diference and the standardized
mean diference , as defined by Borenstein et al. [
Raw Mean Diference . In RecSys experiments, performance metrics are typically calculated for each user, item, or session. Averaging the metrics enable researchers to compare the efectiveness of diferent models. Thus, the mean diference, a direct and intuitive measurement of efect-size, aligns with the scale of the underlying metric. We compute the raw mean difference by averaging the pairwise diferences between treatment and control metric and use the standard deviation ( dif ) of the pairwise diferences to compute its corresponding variance as follows ( is the number of compared pairs): = − , = d2if ,
Standardized Mean Diference . We might consider standardizing the mean diference (i.e., convert it into a “unitless” form) to make the efect-size comparable and combinable across studies, for example, in case of
and diferent ratings scales. The standardized mean diference is computed by dividing the raw mean diference by the within-group standard deviation within calculated across the treatment and control metrics.
=
within within is determined by the standard deviation of the pairwise diferences dif and the correlation (1) (2) of the corresponding pairs as follows: The variance of standardized mean diference is within =
dif
√2(1 − )
= ( +
1
2
2
)2(1 − ),
where is the number of compared pairs. In small samples, tends to overestimate the absolute
value of the true standardized mean diference , which can be corrected by factor to obtain
an unbiased estimate called Hedges’ [
3 4 − 1 , = × , = 2 × , where is degrees of freedom which is − 1 in the paired study setting with number of pairs.
Combined Efect ∗
. After calculating the individual efect-sizes (
) and corresponding
variances ( ) for experiments (i.e., datasets), the final step is to synthesize them into one
combined efect. We assume, as in [
=1
, ∑ 2 − = ∑ − (∑=1 2)2
∑ =1
, ∑ 2 ∑ compute the weighted average of the individual efect-sizes, i.e., the summary efect where the weight of the individual experiments = 1/ . We adjust the weights by 2 and ∗, and its corresponding variance ∗ as follows: ∗ =
1 + 2 , ∗ =
∑ =1
∗ ∑ =1 ∗ , ∗ =
1 ∑ =1 ∗ result of an individual experiment ( ) or the summary efect ( ∗), as follows: by the lower limit, , and the upper limit, ) for a given efect-size , which can be the = √ , = − × , = + × , sponding to the desired significance level . Given we compute = (1 − where is the standard error, the variance of the efect-size, and the Z-value correis the percent point function (we use scipy.stats.norm.ppf1). For example, = 0.05 yields the 2 ), where () 1https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html (3) (4) (5) (6) (7) (8) ⊢ ♦ ⊣. The size of the diamonds corresponds to the weight of the experiments ( ∗). The dotted line at zero denotes the absence of an efect. Suppose the confidence interval of an observed efect-size crosses this line. In that case, it indicates that the efect-size is not significant at the given confidence level, meaning that the efect is not detectable.
For the efect-size-based meta-analysis, we compute pairwise performance metrics for the
models under comparison on a user basis. Therefore we utilize models and data from the Cornac
framework [14]. We use the explicit-feedback datasets listed in Table 1. We split the data in
each experiment with 80% for training and 20% for testing. To conduct the efect-size-based
meta-analysis, we utilize Ranger [
1) We compare matrix factorization with (MF-bias) and without bias terms (MF) [18] to demonstrate an incremental update. Our meta-analysis is based on highlighting the diferences in matrix completion and ranking tasks. For and @10 , we use the standardized mean diference as the efect-size index, accounting for varying scales in each
, we utilize the raw mean diference.
2) We compare matrix factorization (MF) [18] with Bayesian probabilistic ranking (BPR) [19] to illustrate a more sophisticated treatment focusing on ranking. We perform the meta-analysis across all datasets, movie-only datasets, and retail-only datasets, ofering insights into overall and domain-specific efects. mean diference as our efect-size index.
We use the default (hyper)parameters as provided by Cornac [14]. Note that our study focuses on evaluation rather than the models themselves. For replicability, we provide all runs, i.e., is the base metric in each case, and we use the raw MF-bias vs. MF Effect Size −3 −2 −1 0 Standardized Mean Difference of RMSE
MF-bias vs. MF
Effect Size
In our first case study, we examine the impact of incorporating bias terms (user, item, and global) - designated as the treatment - against a basic matrix factorization model - the control. We measure their performance on user-item-rating matrix completion via and ranking through @10 .
Figure 1 summarizes the comparison. All confidence intervals (CI) lie on the left of the zero-efect (dotted) line, with none crossing it, suggesting that the use of bias terms consistently and significantly improves matrix completion performance (smaller error). Notably, nearly all experiments contribute similarly to the overall efect calculation, except for the amazon-clothing dataset, which shows a less confident efect estimation. This reflects the inverse relationship between the variance of an experiment’s efect-size and its weight in calculating the summary efect, as detailed in Equations 6 and 7.
Regarding ranking performance (refer to Figure 2), we generally observe a decline when bias terms are introduced, suggesting potential overfitting. While introducing bias terms enhance matrix completion, it seems to negatively impact ranking performance. Notably, for individual experiments using amazon-ofice and netflix-small datasets, we find no significant ranking performance diferences, as their corresponding confidence intervals cross the zero-efect line.
In our second case study, we evaluate the efect of using Bayesian probabilistic ranking (BPR) treatment - versus matrix factorization (MF) - control - on ranking performance ( @10 ). This comparison is conducted across all datasets, as well as exclusively on movie and retail datasets. Looking at the summary efect in Figure 3, we generally expect a boost in ranking performance when choosing BPR over MF. Although the summary efect estimate is less certain than in the prior meta-analysis, it is still significant. The diference in ranking performance between individual experiments with retail and movie datasets is immediately noticeable. Focusing on retail data only (Figure 4), we expect a performance decline for amazon-clothing, no significant diference for amazon-toy, and performance gains for amazon-digital-music and amazon-ofice . On the whole, we do not expect significant diferences when choosing BPR over MF for retail datasets. In contrast, for movie datasets, we expect consistent and significant BPR vs. MF Effect Size improvements in ranking performance when employing BPR (Figure 5).
Evaluating recommender systems across multiple datasets is crucial for understanding their generalizability and robustness. Yet, given the unique characteristics of diferent data and algorithms, making comparisons across datasets can be challenging. For instance, it’s not straightforward to compare RMSE values or average NDCG values across multiple datasets to determine a model’s overall capacity.
This is where efect-size-based meta-analysis comes in handy. It allows for a statistically robust comparison between two models across multiple datasets, providing a reliable synthesis of results. This mitigates subjective interpretations and fosters more valid conclusions on the overall treatment efect.
We have outlined the theoretical underpinning of this method and, through two case studies,
demonstrated its utility in multi-dataset evaluations. This method isn’t metric-dependent and is
applicable even when scales vary across datasets. Unlike a p-value from a hypothesis test, which
indicates the likelihood of correctly rejecting a null hypothesis (a retrospective view), confidence
intervals predict future outcomes in similar experiments. They enable distinguishing between
the magnitude of an efect and the probability of its recurrence [
Since recommender system evaluations often involve more than two models, our future work will adapt this approach for multi-dataset, multi-model settings. All runs and code - for replicating the results - can be found under https://github.com/MeteSertkan/ meta-analysis-based-recsys-eval.
This research is supported by the Christian Doppler Research Association (CDG), and has received funding from the EU’s H2020 research and innovation program (Grant No. 822670).