Recommendation systems have improved the characterization of user preferences by modeling their digital footprints and item content. However, another facet, model behavior, has attracted a great deal of attention in both academic and industry ifelds in recent years due to the increasing awareness of fairness. The shared task, a Rounded Evaluation of Recommender Systems (EvalRS @ CIKM 2022), is introduced to broadly measure multifaceted model predictions for music recommendation. To tackle the problem, we propose the RecFormer architecture with a personalized temporal-aware transformer to model the interactions among user history in a single framework. Specifically, RecFormer adopts a masked language modeling task as the training objective, which enables the model to capture fine-grained track embeddings by reconstructing tracks. Meanwhile, it also integrates a temporal-aware self-attention mechanism into the Transformer architecture so that the model is able to consider time-variant information among diferent users. Moreover, we introduce linearized attention to reduce quadratic computation and memory cost since the limited time is one of the challenges in this task. Extensive experiments and analysis are conducted to demonstrate the efectiveness of our RecFormer compared with the oficial baseline, and we examine the model contribution from the ablation study. Our team, yao0510, won the seventh prize with a total score of 0.1964 in the EvalRS challenge, which illustrates that our model achieved competitive performance. The source code will be publicly available at https://github.com/wywyWang/RecFormer.
In recent years, characterizing users with accurate
interests has evolved due to the use of advanced
recommendation systems (RSs). These RSs have been used to
develop several real-world applications in industry, for
example, Amazon ROSE [
to tackle both standard evaluation metrics and model trates an example of a music recommendation system.
LGOBE Information and Knowledge Management
ages/loop.jpg
22.5 minutes per fold, which is challenging when adopt- in RecFormer.
ing deep learning techniques to have a fine-grained track
representation if the space and computation complexity
is high. 3) Time-variant event. The target users have 2. Related Work
their corresponding habits of listening music songs
during the day; for example, students are likely to listen to Recommendation System. The recent progress of
recmusic after school, while ofice workers are likely to lis- ommendation systems (RSs) has brought great incomes
ten during working hours. For example, Koren [
To address the aforementioned challenges, we propose users’ records into account to predict future preferences.
RecFormer, a novel personalized temporal-aware Trans- However, one of the limitations in this task is the
valiformer for fair music recommendation, which consists of dation strategy, which does not guarantee that the test
personalized user embeddings, a temporal-aware multi- set is from the latest timestamp. This hinders the above
head linearized-attention in a modified Transformer [
In summary, our contributions are as follows: 1) We evaluating multi-dimensional behaviors of models. In propose RecFormer, a novel personalized temporal-aware general, this dataset is able to help researchers achieve Transformer for fair music recommendation by adopting not only quantitative performance but also non-standard masked language modeling tasks as training objectives metrics for fairness. to learn fine-grained track representations. 2) To reduce the computation complexity and model time-variant in- 3. Methodology formation, a temporal-aware linearized-attention is designed by replacing softmax with kernel computation and Figure 2 illustrates an overview of our proposed Recintegrating temporal embeddings into attention scores. Former, some of which is inspired by the recent research Furthermore, we propose a new metric (MRED_DOH) on natural language understanding. Given a user histo reflect the diferent performance of predicting tracks tory (sequence), we first apply random masks to the in each hour, which is also an essential but challenging sequence, and then personalized user embeddings are dimension of fairness. 3) Our RecFormer outperforms generated based on the user metadata and correspondthe oficial baseline at least 116% in terms of the total ing tracks. Afterwards, the RecFormer, which modifies score. Moreover, extensive experiments were further the self-attention mechanism to the proposed temporalconducted to examine the contribution of each module aware linearized-attention in each layer, is introduced
As each user has multiple tracks they have listened to
and the corresponding user metadata, we incorporate
each type of user metadata with each track to model each
persona, which is inspired by [
history, which is hard to consider with traditional CF and
based on the Transformer encoder [
the standard Transformer encoding steps to encode the
of FFN is
.
Temporal-Aware Linearized-Attention (TALA): To
reduce the quadratic memory and computation
complexity from the conventional self-attention mechanism, we
replace the softmax with the kernel computation, which
only requires linear computation [
Since users listen to music tracks following their
personal habits (e.g., at work or on a bus), it is essential to
take time-invariant information into account for
recommending user preferred tracks, e.g., [
After layers of RecFormer to encode multi-hop information, we get the final output for all items of the user sequence. The track classifier is employed to predict (3) (4) (5) sequence with the proposed temporal-aware linearized- the masked tracks as shown in Figure 2. Specifically, we attention (TALA), residual connection and layer normal- apply a feed-forward layer to generate the -th output: ization (Norm), dropout, and feed forward network (FFN) = ( ); ̂ = ( ), (6) as follows: =̃ ( + ()), = ( +̃ ( ) )̃.
where ∈ ℝ× , is the total number of tracks, and (2) is the activation function. Training and Testing. As one of the challenges in this task is the validation strategy, it is expected that the test set may not be sampled at the latest timestamp. To tackle this issue, we applied masked language modeling tasks (MLM) as the training objective to learn a robust track representation. The goal of MLM is to reconstruct the masked tracks by giving a user sequence, which enables the model to learn the relation between tracks.
Following [9], we use the final output with the track classifier to predict the masked tracks, and the loss function is defined as follows:
| | = − ∑ ( ̂ ),
=1 where is the set of user sequences.
In the inference phase, we empirically set the mask in the last timestamp of a sequence to predict the most possible 100 tracks that the user are likely to listen to. In addition, as we cannot fetch the timestamp in the test set, we set the predicted hour as the hour that the user often listens to music tracks to fit into TALA.
a per-group or slice basis (gender balance, artist popularity, user country, song popularity, and user history), and behavioral tests (be less wrong, and latent diversity) [11].
The data are pre-processed by filling NaN values of user gender and country with n and UNKNOWN, respectively.
Proposed MRED_DOH Metric: Since one of the
challenges we aim to address is the time-variant event, we
propose a new metric, MRED_DOH, to evaluate the
difference performance in terms of various listening times
in a day, which reflects another critical but unexplored
dimension of fairness. That is, this MRED_DOH enables
us to investigate if a model bias to predictions that are
(7) from users listening to in specific time slots. In other
words, we operationalize this metric as the smaller the
diference similar to MRED_Gender proposed in Reclist
[
Specifically, we represent the hour when the user most often listens to select the test set as sub-groups of listeners (i.e., there are total 24 sub-groups), which is represented as the user hour. Afterwards, we can evaluate the MRED score using the existing RecList with the user hour to measure the model performance at each hour in a day.
We note that this is a aspect-driven metric, which can be adjusted by monitoring diferent temporal dimensions based on user needs. For example, the hour when the user most often listens can be easily changed to the least active hours or the average active hours. Moreover, it can also be employed in the sequential recommendation systems by changing the user hours to sequential positions.
The dimension was set to 64, the inner dimension of the feed-forward layer was 256, and the number of heads was set to 1. The dropout rate was 0.0, and the max sequence length ( ) was 60 due to the time limit, which kept about 25% tracks on average. The batch size was 100, the learning rate was set to 1e-3, the training epochs 4.2. Overall Performance Comparison were set to 50, and the seeds were tested from 42 to 52. It is noted that the number of predicted tracks is based on Ablation Study. To verify the contribution of each modthe train set. That is, we hypothesize that our RecFormer ule in RecFormer, we conduct ablative experiments by only recommends tracks that have been listened to before. removing personalized user embeddings, temporal-aware All the training and evaluation phases were conducted on computation, and both. From Table 1, we can observe a machine with AMD Ryzen Threadripper 3960X 24-Core that removing any one module in RecFormer results in Processor, Nvidia GeForce RTX 3090, and 252GB RAM. a performance drop in terms of metrics adopted in this
The evaluation metrics include diferent perspectives: task, which testifies to the efective design of RecFormer. standard RSs metrics (HR, and MRR), standard metrics on However, our RecFormer performs the worst in terms of our proposed MRED_DOH, which indicates that our method still fails to meet the fairness in the temporal aspect. In addition, this result also indicates that considering temporal-awareness is not able to address the temporal fairness, which will be investigated in our future research.
We note that several continuous variables (e.g.,
novelty_artist related features) are also included in the
personalized user embeddings with projecting as in [
Oficial Score. Table 2 shows the performance in the formal phase. We also implemented BERT4Rec to compare the performance, which can be viewed as one of our variants. It can be observed that the MRED_DOH performance of BERT4Rec is the best, but the performance of standard RSs metrics fails to meet the requirement (hit-rate > 0.015). One of the reasons is that BERT4Rec is not converged using the same hyper-parameters due to the computational complexity, which is attributed by linear attention in our RecFormer. Therefore, these fairness results cannot be directly compared with the oficial baseline and our RecFormer since BERT4Rec cannot recommend possible tracks. Our framework achieved 0.1964 of the total score, which outperformed the oficial baseline by 116%, while it still has some gaps compared to the first prize. Despite the result, our approach still demonstrates that using MLM as the training objective can achieve competitive performance.