Sequential Recommender Systems (SRSs) are widely used to model user behavior over time but they often face a critical challenge: they can fail when faced with perturbations in their training data. While the conventional Rank-Biased Overlap (RBO) measure is widely used, it does not properly address this issue, especially when dealing with finite rankings. To fill this gap, we introduce the Finite Rank-Biased Overlap (FRBO) measure. We study the impact of removing elements at the beginning, in the middle, and at the end of the sequence: the latter removal has a negative impact on performance of up to 60% in NDCG. Surprisingly, removing elements from the beginning or middle of sequences has minimal impact on performance. These results shed light on the crucial role of element positioning within the training data and highlight the urgent need for improved robustness in SRSs. We make available our code implementation1 for FRBO and invite further exploration and adoption by the research community.
Recommender systems are now ubiquitous, crucial for helping users navigate the vast online
information [
In Sequential Recommendation, each user is represented by a sequence of items = (1, 2, ..., , ..., − 1, ) with which they have interacted, where is the sequence length. We investigate this type of removal: Beginning: = (+1, . . . , − 1), Middle: = (1, . . . , ⌊ − 1− , ⌊ − 1+ ⌋..., − 1), End: = (1, . . . , − 1− ), with ≤ 10.
2 ⌋ 2
To evaluate model performance, we use traditional metrics: Precision, Recall, MRR and NDCG.
For stability assessment, we employ the Rank List Sensitivity (RLS) [
RBO(X, Y) = (1 − ) ∑︁ − 1 |[1 : ] ∩ [1 : ]| (1)
In recommendation systems, metrics are often computed using finite-length rankings, NDCG, MRR) adapt well to this format, but RLS does not when using RBO due to a key limitation: it does not converge to one for identical finite-length lists. To address this, we introduce Finite Rank-Biased Overlap FRBO@k, designed to ensure a convergence value of
= (1, ..., ), such that , ∈ , and ∈ N+ Theorem 1. Given a set of items = {1, ..., }, two rankings = (1, ..., ) and | | =1 +∞ =1 ,
,
Finding the minimum and maximum values of RBO is crucial for normalizing it when summing up to the top k items of the ranking. We also need to show that these values are not necessarily limited to 0 and 1. (1, ..., ), such that , ∈ , and ∈ N+, the following holds: Lemma 1. Given a set of items = {1, ..., }, two rankings = (1, ..., ) and = , ⎧⎨0, ︂(
⎩(1 − ) 2 ⌊ 21−⌋− − ℓ ︂) if ≤ ⌊ otherwise 2 ⌋ (2) (3) where RBO(X, Y)@k = (1 − ) ∑︁ − 1 |[1 : ] ∩ [1 : ]| =1 ⌊︁ ⌋︁ and ℓ = 2 Φ( , 1, ⌊ 2 ⌋ + 1) − Φ( , 1, + 1) Lemma 2. Given a set of items = {1, ..., }, two rankings = (1, ..., ) and = (1, ..., ), such that , ∈ and ∈ N+, the following holds: max, RBO@k = 1 − .
We use four diferent datasets: MovieLens (1M and 100K versions) [ 10] and Foursquare (New York City and Tokyo) [11]; we use two diferent architectures to validate the results: SASRec [12] and GRU4Rec [13]. The RecBole library [14] was utilized for conducting all the experiments, encompassing data preprocessing, model configuration, training, and testing. The code required to replicate the experiments is accessible in our GitHub repository1.
The inherent resilience of the models when using alternative starting seeds is shown in the Baseline row in Tab. 1. In general, the deviation is almost always less than 1%. However, the RLS shows a large deviation from the optimal value of 1, indicating quite diferent ranks. The aggregated results suggest that the models achieve a suficient level of performance regardless of the initialization seed, but the generated rankings are significantly afected by it.
The performance and stability of keeping all elements in the training set with a constant initialization seed versus removing 10 elements are compared in Tab. 1. It can be seen that the performance of the model is not significantly afected by removing items from the beginning or the middle of the sequence. Rather, we can see how drastically the metrics are reduced by removing items from the end of the sequence: for example, when SASRec is applied to the MovieLens 1M dataset, the NDCG decreases by more than 50%. In addition, the Jaccard similarity and FRBO values approach 0, indicating that very few items are shared by the generated ranks.
As seen before, removing items at the beginning or in the middle of the sequence has a negligible impact on performance, as confirmed by Fig. 1. However, removing elements from the end of the sequence leads to a noticeable decrease in the metrics, proportional to the number of elements removed. This efect is consistent across both models tested and across all datasets evaluated. In particular, Fig. 1 illustrate how the SASRec model performs on the MovieLens datasets under
these conditions, highlighting in particular a significant decrease in metrics. Interestingly, even removals from the beginning and middle of sequences in ML-1M show a significant decrease in RLS-FRBO, suggesting considerable variation in the rankings generated despite overall stable performance levels, likely influenced by the large user base and interaction volume of the dataset. 10 8
6 4 Elements removed
2
This study investigates the efect of item position within a temporally ordered sequence in
SRSs. First, it introduces Finite RBO, a variant of RBO tailored for finite lists, which has been
shown to normalize within the [
This work was partially supported by projects FAIR (PE0000013) and SERICS (PE00000014) under the MUR National Recovery and Resilience Plan funded by the European Union - NextGenerationEU. This work has also been supported by the NEREO (Neural Reasoning over Open Data) project funded by the Italian Ministry of Education and Research (PRIN) Grant no. 2022AEFHA. [10] F. M. Harper, J. A. Konstan, The movielens datasets: History and context, ACM Trans. Interact. Intell. Syst. 5 (2015). URL: https://doi.org/10.1145/2827872. doi:10.1145/2827872. [11] D. Yang, D. Zhang, V. W. Zheng, Z. Yu, Modeling user activity preference by leveraging user spatial temporal characteristics in lbsns, IEEE Transactions on Systems, Man, and Cybernetics: Systems 45 (2014) 129–142. [12] W.-C. Kang, J. McAuley, Self-attentive sequential recommendation, in: 2018 IEEE International Conference on Data Mining (ICDM), IEEE, 2018, pp. 197–206. [13] B. Hidasi, A. Karatzoglou, L. Baltrunas, D. Tikk, Session-based recommendations with recurrent neural networks, 2016. arXiv:1511.06939. [14] W. X. Zhao, S. Mu, Y. Hou, Z. Lin, Y. Chen, X. Pan, K. Li, Y. Lu, H. Wang, C. Tian, Y. Min, Z. Feng, X. Fan, X. Chen, P. Wang, W. Ji, Y. Li, X. Wang, J.-R. Wen, Recbole: Towards a unified, comprehensive and eficient framework for recommendation algorithms, 2021. arXiv:2011.01731. [15] M. S. Bucarelli, L. Cassano, F. Siciliano, A. Mantrach, F. Silvestri, Leveraging inter-rater agreement for classification in the presence of noisy labels, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 3439–3448.