In the field of recommender systems, an important issue within the current state-of-the-art is the inconsistency in item rankings produced by models initialized with diferent weight seeds. Despite these models achieve convergence and obtain similar average performance metrics, their item rankings difer significantly. This phenomenon is quantitavely demonstrated using metrics such as Rank List Sensitivity (RLS) and Normalized Discounted Cumulative Gain (NDCG) across diferent model pairs. In this paper, we reafirm the existence of this problem and provide new insights by analysing models with common item embeddings but diferent network initialization, and diferent item embeddings but common network initialization, to identify which network components most influence ranking variability. To address the general issue, we propose an ensemble approach that averages the output of multiple models. Our ensemble maintains the NDCG of the original model while significantly improving ranking stability: the RLS FRBO@10 value shows an approximate increase of 30.82%.
In the recent years, neural sequential recommender systems
have gained importance due to their ability to model user
behavior over time, providing more accurate and
personalized recommendations[
Despite the advancements in neural sequential
recommender systems, a significant issue persists: the
variability in item rankings generated by models initialized
with diferent weight seeds[
In both initializations, when the models arrive at convergence, Item A is consistently ranked as the top item, which is expected as it is the correct positive item that the model should prioritize. However, there is significant variability in the ranking of the other items.
For instance, Item B is ranked 2nd in Initialization 1 but drops to 3rd in Initialization 2. Similarly, Item C rises from 3rd in Initialization 1 to 2nd in Initialization 2. This variability can occur because the loss function used in training 1 2 3 4 5
typically focuses on ensuring the correct positive item is ranked highest, but does not enforce a specific order for the remaining negative items.
Even when models converge and predict the same
topranked item for a given user sequence, the subsequent items
in the ranking often difer. This inconsistency can afect
tasks that require multiple item predictions simultaneously,
generate relevance-ordered rankings, and impact the
explainability of the model’s recommendations. In particular,
we are referring to these related works that address the
sensitivity and robustness of recommender systems[
Our experiments show that ensemble methods efectively reduce ranking variability. Specifically, models with diferent initialization seeds obtain an average RLS-FRBO@10 of 0.542, while an ensemble of models achieves a score of 0.709. Furthermore, we show that the average NDCG@10 across models with diferent seeds is 0.129, while an ensemble approach improves it to 0.132. This results show that ensemble methods not only reduce variability but also maintain or even slightly enhance recommendation quality. Our contributions can be summarized as follows: • We identified the significant impact of weight initialization on the variability of item rankings in recommender systems. • We proposed the use of ensemble models to reduce this variability, demonstrating that shared embeddings can further improve ranking consistency. 2.
The experimental framework employs the SASRec
model[
v − 1 =1 The next item ˆ+1 is predicted by applying a softmax function over the dot products between z and the embedding matrix E: ˆ+1 = argmax softmax z⊤E)︁ ︁( ︁( For example, if the user interacted with items [1, 2, . . . , ], the model will generate predictions for the next likely item ˆ+1 based on the context-aware embeddings. Assume 1 corresponds to "Book A", 2 to "Book B", and so on. If the attention mechanism strongly associates "Book B" with "Book D" in the past, "Book D" might be ranked higher as the next recommendation ˆ+1.
For this study, the SASRec model configuration includes an embedding dimension of 50, a single self-attention head, and a dropout rate of 0.2, the latter being employed to reduce the risk of overfitting.
The dataset employed for this study is the MovieLens 1M
(ML-1M) dataset[
The experimental dataset is curated with careful consideration of the following preprocessing steps: • Rating Threshold: To ensure meaningful interaction data, only users who have rated at least 5 items, and items rated by at least 5 users, are retained. • Data Partitioning: A leave-one-out strategy is adopted for data splitting. In particular, for each user, the most recent interaction is used for testing, while the one before the last is reserved for validation. The remaining data constitutes the training set. • Negative Sampling: Negative sampling is employed to balance the dataset. During training and validation, one negative sample per positive instance is generated, whereas, in the testing phase, all noninteracted items are considered as negative samples.
The training pipeline is meticulously designed to optimize performance and eficiency: • Batch Processing: Training is conducted with a batch size of 128, ensuring the model can eficiently process substantial data per iteration. • Model Checkpointing: To capture the most optimal model configuration, checkpoints are saved at intervals. The best model is determined based on the highest NDCG@10 score observed during validation. • Training Setup: The model undergoes training for up to 200 epochs, utilizing the Adam optimizer. The objective function is the Binary Cross-Entropy Loss, which is particularly suited for this task. It is a suitable choice because it computes probabilities over two classes, the positive one and the negatives, since the model has to guess if the items is or not the next one in the sequence. Additionally, BCE Loss outputs probabilities that can be used directly to rank items by their likelihood of being the next interaction.
The evaluation of model performance is based on several metrics, addressing both predictive accuracy and robustness. • Performance Metrics: The primary metric for assessing predictive performance is NDCG (Normalized Discounted Cumulative Gain), calculated at various cutofs (5, 10, and 20). NDCG evaluates the ranking quality of the recommended items, giving higher importance to items ranked closer to the top of the list. • Robustness Metrics: To evaluate the robustness of the recommender system, we use the Rank List Sensitivity (RLS) metric.
= 1
∑︁ |test| ∈test sim( , ) ℳ ℳ′ Here, test is the set of test items, and ℳ ℳ′ are the rank lists generated by the model ℳ and a perturbed version ℳ′ for item . The function sim(, ) measures the similarity between two rank lists and . The RLS metric gives an average similarity score over all test items. RLS measures how much a model’s recommendations change in response to small perturbations in the training data. There are two versions of RLS: the RLS-RBO (RankBiased Overlap) and the RLS-JAC (Jaccard Similarity). RLS-RBO, based on the Rank-Biased Overlap, computes the similarity between two ranked lists, but it is tailored for infinite rankings, which can limit its applicability in finite settings. (, ) = (1 − ) ∑︀|| =1 − 1 |[1:]∩[1:]| The Rank-biased Overlap (RBO) measures the similarity of orderings between two rank lists and . The parameter (typically set to 0.9) controls the weighting, with higher weights given to the top ranks. || is the total number of items, and [1 : ] represents the top- items in list . The RBO score lies between 0 and 1, where higher values indicate more similarity between the rank lists. RLS-JAC, on the other hand, uses the Jaccard similarity coeficient, focusing on the overlap of items between two sets without considering their order.
Jaccard = | ∩ |
| ∪ |
where and are the sets of items in the two
ranked lists. Given the limitations of RLS-RBO,
we utilize an enhanced version called Finite
RankBiased Overlap (FRBO)[
The RBO value is now normalized to its maximum possible value, ensuring that the metric reaches 1 when the two rankings are identical.
FRBO addresses the shortcomings of RBO by correctly handling identical rankings, making it more suitable for practical evaluation scenarios.
By comparing diferent versions of RLS, we aim to capture how robustly the models handle perturbations, ensuring a more reliable assessment of system stability. For brevity, we will refer to the Jaccard version of RLS as @, and the FRBO version of RLS as @ in the subsequent sections and tables. The notation @k indicates a cutof at the first k item of the lists.
We aim to answer the following questions: • RQ1: Does a variation of (an apparently) small factor influence the robustness of Neural Recommender Systems? • RQ2: Do ensemble of models improve the robustness of Neural RecSys? • RQ3: Can we improve performance through ensembling? 3.1. RQ1 Tables 3 and 4 reveal that models initialized with diferent seeds, which means entirely diferent weight initializations, obtain an average Jaccard@10 of 0.505 and an average FRBO@10 of 0.542, both of which are significantly below the ideal value of 1.
When initializing only the item embeddings with the same seed, Tables 5 and 6 show a slight but not significant increase, getting a value of 0.514 for the Jaccard@10, and 0.547 for the FRBO@10. computed between models initialized with the same embedding seeds, for diferent values of (5, 10, and 20). The notation x_y represents a model where the embedding layer is initialized with seed x, while the rest of the model is initialized with seed y.
Seeds 1
Seeds 2
Instead, initializing the rest of the network with the same seed but using diferent embedding seeds, leads to a significant improvement: in Tables 7 and 8 we observe an value of 0.541 for the Jaccard@10 and 0.575 for the FRBO@10. computed between models initialized with same rest of the network seeds for diferent values of (5, 10, and 20). The notation x_y represents a model where the embedding layer is initialized with seed x, while the rest of the model is initialized with seed y.
Seeds 1
Seeds 2 @20
These results suggest that embedding initialization has a limited impact on the final model performance and resulting rankings. We therefore aim to investigate the relationship between two final trained embedding spaces.
To explore the relationship between embeddings from different models, we applied a linear transformation to map the embeddings from one model to another, followed by a visualization using Principal Component Analysis (PCA). This analysis aims to provide insights into how well the embeddings from diferent models align after the transformation.
The experimental procedure involved the following steps: 1. Embedding matrices: and are the embedding matrices of the first and second models, respectively, each with a shape of × . 2. Fitting a Linear Regression Model: We use a linear regression model to estimate a transformation matrix that maps the embeddings from Model 1 to those of Model 2:
ˆ = · where ˆ is the transformed embeddings obtained from Model 1 that should match those of Model 2. 3. Dimensionality Reduction with PCA: To visualize the embeddings, we reduce their dimensionality using Principal Component Analysis (PCA). Both the transformed embeddings ˆ and the original embeddings are projected into a 2D space by retaining the first two principal components:
ˆ PCA = PCA(ˆ ) PCA = PCA( ) 4. Visualization: We use scatter plots to visualize the relationship between the transformed and the original embeddings, for each PCA component. A red dashed line representing the bisector ( = ) is included in both plots to visually assess the alignment of the components.
Fig. 1 reveals a strong alignment between the transformed embeddings and the original embeddings from Model 2, as the points are closely distributed along the bisector. This suggests that the linear transformation was highly efective in mapping the embeddings from Model 1 to the embedding space of Model 2. This finding implies that the embedding layers converge to a similar space, that is diferent from the others apart from a linear transformation. Assuming that the transformation matrix W has full rank, it represents an endomorphism, meaning that W is invertible. This implies that the attention mechanism, which applies linear transformations as described in Section 2.1, can efectively learn to align the embeddings, regardless of the specific embedding space to which the layers converge.
As a result, we argue that the position to which the embedding converges is almost independent of the initialization seed. This means that even when the rest of the network is initialized with diferent seeds, the embeddings tend to converge to similar positions in the embedding space. As a result, changing the seed for the rest of the network has a more significant impact because the representations of the items in the embedding space remain quite similar regardless of the seed, while the rest of the network’s components do not. This highlights that the embedding space is relatively stable across diferent seeds, whereas the rest of the network is more sensitive to the initial seed. 3.2. RQ2 The next step of the analysis is to combine the scores of diferent models, in order to check if the RLS between combined models shows significant changes.
It is evident from Table 9 that using ensembles significantly improves the RLS compared to individual models, indicating better stability.
42 43 44 45
In the context of enhancing the robustness of neural recommender systems, the use of an ensemble of models provides significant benefits. However, it also introduces a computational cost that scales linearly with the number of models in the ensemble. Specifically, the computational cost of training an ensemble is approximately the cost of training a single model multiplied by the number of models, denoted as:
Costensemble = models ×
The same applies to the inference stage. As shown in Tables 9,11, the performance of the system remains relatively stable regardless of the number of models used, while the robustness sees notable improvements when moving from a single model to an ensemble of two models. Given this, a reasonable compromise is to use an ensemble of two models. This choice results in a computational cost of approximately:
Costtraining = 2 × Costsingle_model × Iterations for backpropagation during training, and:
Costinference = 2 × Costsingle_model for inference during deployment. This allows for a significant improvement in robustness by paying around the double in terms of computational cost.
In conclusion, this work addresses the critical issue of ranking variability in recommender systems, which arises due to diferent model initialization seeds. Our findings suggest that ensemble methods, particularly those incorporating shared embeddings, ofer a promising solution to mitigate this variability. By reducing ranking fluctuations, these methods enhance the reliability and consistency of recommendations. However, while our approach significantly reduces variability, some residual variability remains, indicating the need for further research to explore additional techniques or refinements to achieve even greater consistency in recommendations.