Despite the pivotal role ensembling played in the success of BellKors' Pragmatic Chaos Team in winning the Netflix Prize challenge in the early 2000s, it never became a standard practice in recommender systems. In contrast, ensembling, particularly greedy ensemble selection, has become a standard practice in machine learning to enhance performance compared to a single model. Despite the success of greedy ensemble selection in classification and regression tasks, it has not been adapted for top-n prediction tasks. Hence, in this study, we aim to analyze the potential of greedy ensemble selection to boost the performance of recommender system models for top-n prediction tasks. We adapt the concept of greedy ensemble selection for top-n prediction tasks, train and optimize ten factorizationand neighborhood-based models on five datasets, and compare the performance of the ensemble to that of the individual models. Our experiments reveal that greedy ensemble selection always performs better than a single model and enhances performance by an average to quantify the performance and robustness improvements Workshop Proceedings Attribution 4.0 International (CC BY 4.0).
Ensembling played a pivotal role for BellKors’ Pragmatic
Chaos Team, enhancing their recommender system to win
the Netflix Prize challenge in the early 2000s [
In comparison, in machine learning, ensembling,
particularly greedy ensemble selection, is a standard practice [
However, despite the success of greedy ensemble
selection for regression and classification, it has never been
adapted for top-n ranking prediction tasks [
Therefore, we want to analyze the potential of greedy ensemble selection for top-n prediction tasks and answer the question: RQ: How does the ensembling of factorizationand neighborhood-based models impact performance and robustness compared to a single optimized model?
In this work, we adapt greedy ensemble selection for top-n prediction tasks to assess its potential for enhancing
The use of ensembling techniques in recommender systems
is not new and has been covered in the literature [
Today, primarily hybrid recommender systems use
ensembling to mitigate the weaknesses of individual algorithms
[
As in our work, researchers have applied ensembling
techniques that do not require manual model selection for
ensembling. For example, they have used standard machine
learning ensemble techniques, such as bagging and boosting
to recommender systems [
ceur-ws.org © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License ecommendations
Additionally, researchers have analyzed ensembling for
various other aspects of recommender systems. Researchers
ensemble predictions of models trained on diferent datasets
containing diferent user feedback types [
Recent work has focused on greedy ensemble selection
for recommender systems [
Greedy ensemble selection, as implemented in machine learning, cannot be directly applied to top-n recommendations in recommender systems. Greedy ensemble selection for classification applies majority voting on predictions. However, majority voting fails for top-n recommendations since the number of repeating item recommendations across users is often insuficient. Similarly, taking the mean for regression tasks is not applicable, as top-n recommendations deal with ranked lists instead of single numeric values returned by each model. Therefore, we focus on aggregating and re-ranking the prediction scores of multiple models to generate an ensemble of their outputs.
To apply greedy ensemble selection, we assume we have a set of trained and optimized models , each predicting ′ items and their validation performance. We aim to aggregate the ′ predictions of every model ∈ into one ranked list of length . This requires the length ′ of every predicted list to be at least the same length as .
In ensembling, we can utilize more predictions ( ′) than the desired output list length ( ). There is a chance that a prediction ranked at position + 1 or beyond still holds relevance or contributes valuable information, even though it does not make it into the top predictions of the model. Taking ′ predictions into account enables the ensemble process to utilize a broader range of data, potentially improving the performance of the final ensembled recommendations. Furthermore, utilizing ′ predictions does not increase the prediction cost of the base models since all models score all predictions anyway before selecting the top .
We normalize all prediction scores of ′ and multiply each prediction score by the validation performance of the respective model (Algorithm 1). The normalization ensures that all models have an equal impact in the ensembling, while the validation performance multiplication weights the impact based on the models’ performance. Consequently, the impact of models with good validation performance will be increased relative to those of poorly performing models.
Then, we initiate the greedy search by examining all subsets of ∈ and aggregating the ′ predictions by aggregating and re-ranking the prediction scores of all in the subset included models (Algorithm 1). If items appear in the ′ lists of multiple models, the item scores are summed to reflect their collective relevance across models. We then select the top- predictions of the ensembled list. The performance of every subset of all models ∈ is evaluated on the validation set. The best-performing ensemble of models is then selected as the final result of the greedy ensemble selection, and their top- predictions are returned.
We conducted all of our experiments with ten diferent factorization- and neighborhood-based algorithms and greedy ensemble selection on five datasets. The hardware includes AMD EPYC 7452 CPU processors, each with 32 cores and a CPU frequency ranging from 2.35 to 3.35 GHz.
In our experimental pipeline, we apply five-fold
crossvalidation to all five datasets, randomly splitting each fold
into three sets: 60% for training, 20% for validation, and
20% for testing. With the training and validation sets, we
optimize all included algorithms with two hours of
random search to select the best hyperparameter configuration.
With the test set, we evaluate the final performance of the
single models as well as the greedy ensemble selection. This
combination of five-fold cross-validation and random search
allows each algorithm to be finely tuned on every subset of
the data while mitigating the efects of randomness in data
splits and hyperparameter selection [
We measure performance using NDCG@k for = 5, 10, 20
to evaluate top-n ranking predictions for diferent list
lengths. The NDCG@k model performance on the validation
set, obtained from the random search optimization process,
is later used to weight the model predictions (Section 3).
4.1.1. Datasets
We include five distinct datasets of diferent sizes in our
experiments and refer to Table 1 for a detailed overview.
We transform convert datasets with user ratings, specifically
Movielens-1M, Movielens-100k, and CiaoDVD, into binary
user feedback datasets as it is done in related work [
We run greedy ensemble selection using various prediction input list lengths ( ′) to examine the impact of predictions ranked higher than on the ensembling process (Section 3). We set ′ to 5, 10, 15, 25, 50, 75, 100, 125, and 150. This wide range of ′ values helps us identify trends in the impact of longer input list lengths. All ensemble configurations are evaluated on the validation set. Ultimately, we select the ensemble configuration that performs best on average across all folds, with the optimal ′ value.
Our experiments reveal that greedy ensemble selection enhances performance by an average of 8.8% on NDCG@5, 8.6% on NDCG@10 (Table 2), and 16.3% on NDCG@20 compared to the single best model. Since NDCG@10 is the most commonly used evaluation metric with a cutof of = 10 and the trends are consistent across all values, our analysis will focus on the NDCG@10 results.
In general, the algorithm performance ranking varies across datasets. While Popularity always yields the lowest NDCG@10 score (Table 1), the best-performing algorithm changes. UserKNN performs best on CiaoDVD, CiteULikeA, and MovieLens-1M, while ItemKNN performs best on Hetrec-LastFM and ALS on MovieLens-100k.
In contrast to the single algorithm performances, greedy ensemble selection consistently outperforms all algorithms across all datasets for all NDCG@ values and adverts the algorithm selection problem. Greedy ensemble selection efectively identifies and aggregates a subset of models that outperforms the single best model, resulting in the highest
NDCG@k scores on all five datasets. This approach reliably enhances performance and presents robust results compared to the single best model respectively.
On average, the overall performance advantage of greedy ensemble selection compared to the single best algorithm is 8.6% on NDCG@10 (108% vs. 99% for UserKNN, Table 2), but varies across datasets. Greedy ensemble selection shows a performance increase as high as 15.4% on CiaoDVD (0.03 vs. 0.026 for UserKNN, Section 5) and as low as 0.7% on HetrecLastFM (0.175 vs. 0.174 for ItemKNN, Section 5). On datasets like CiteULike-A, Movielens-1M, and MovieLens-100k, the performance increase is approximately 5%.
Longer prediction input lists of length ′ (Section 3) improve the overall model performance (Fig. 1). Predictions that do not make it into the final predictions of the single models still contribute valuable information to the ensemble process. While ensembling ′ = predictions already enhance performance, increasing ′ can further improve results. We tested using up to 150 predictions per user from each model and observed that the ensemble’s performance plateaued beyond ′ = 100 predictions (Fig. 1). Additionally, increasing ′ beyond this point incurs higher computational costs during the ensembling process without yielding significant performance gains.
To comprehensively answer our research question: How does the ensembling of factorization- and neighborhoodbased models impact performance compared to a single optimized model? We conclude that greedy ensemble selection of factorization- and neighborhood-based models enhances the performance, on average, up to 16.3% compared to the single best model averaged over all datasets.
Our experiments show that greedy ensemble selection enhances performance and avoids the need for manual algorithm selection. By ensembling a subset of all available algorithms, greedy ensemble selection consistently achieves better results than any single algorithm across all included datasets. However, ensembling introduces an additional step in the recommender systems pipeline.
Despite its performance boost, greedy ensemble selection for top-n recommendations is expensive compared to single factorization- and neighborhood-based models. In addition to adding complexity to the pipeline, ensembling requires the training and optimization of multiple models to utilize their predictions for the top-n recommendations. Increasing the overall complexity and computational cost.
Nevertheless, the research community appears willing to accept higher computational costs for better performance, as evidenced by more sophisticated algorithms and the growing use of deep-learning approaches. While greedy ensemble selection involves an exhaustive search, further research could optimize the ensembling process for greater eficiency.
Currently, there is an ongoing debate in the field about
whether recommender systems are truly making progress
[
Future work can investigate the contribution of more advanced deep-learning algorithms to the ensembling process. This includes assessing the models’ potential performance enhancements and comparing the overall performance to state-of-the-art deep-learning methods. Furthermore, analyzing more eficient strategies to build an efective ensemble is valuable. Finally, examining the impact of ensembling across diferent domains within recommender systems helps better understand domain-specific trends.