It is a long-standing challenge in modern recommender systems to make recommendations for new users, namely the cold-start problem. Cross-Domain Recommendation (CDR) has been proposed to address this challenge, but current ways to represent users' interests across systems are still severely limited. We introduce Personal Knowledge Graph (PKG) as a domain-invariant interest representation, and propose a novel CDR paradigm named MeKB-Rec. We first link users and entities in a knowledge base to construct a PKG of users' interests, named MeKB. Then we learn a semantic representation of MeKB for the cross-domain recommendation. Beyond most existing systems, our approach builds a semantic mapping across domains using Pretrained Language Models which breaks the requirement for in-domain user behaviors, enabling zero-shot recommendations for new users in a low-resource domain. We experiment MeKB-Rec on well-established public CDR datasets, and demonstrate that the new formulation achieves a new state-of-the-art that significantly improves HR@10 and NDCG@10 metrics over best previous approaches by 24%-91%, with a 105% improvement for HR@10 of zero-shot users with no behavior in the target domain. We deploy MeKB-Rec in WeiXin recommendation scenarios and achieve significant gains in core online metrics. MeKB-Rec is now serving hundreds of millions of users in real-world products.
line Recommender Systems for decades. One promising line of methods to deal with the cold-start problem is
transfers user and item information from other domains to assist recommendation for the target domain [3, 4, 5].
where users partially overlap in two domains, and many users have rich behaviors in the source domain but no or few interactions in the target domain. How to efectively make recommendations for such users is a key challenge to improve modern Recommender Systems.
ferable representations with the help of users that overlap between domains. However, most existing CDR approaches are constrained by a wide-spread hypothesis that embedding representations of users can be sufi
nEvelop-O (Q. Chen)
(MeKB) SylasRole
Aatrox
Role
LPL JDG Team
Team
RNG Dota Game Legends
Leagueof Company Tencent Similar Location Similar
Company Dance
Research
Team Footbal MeKB Construction Global Knowledge Graph Entity Linking EmbedMe Algorithm
for Retrieval EmbedMe
User Behaviors in Source Domain Retrieval Results inTarget Domain MeKB construction step builds a personal knowledge graph (MeKB) representing a user’s interest as a sub-graph of entities. model to map the entity sequence of MeKB for retrieval in the target domain. transferability, as it tends to weaken the diverse semantics behind users’ interactions in the source domain; 2) the data sparsity for overlapped users can severely hinder the learning efectiveness, without help of external
ciently learned from a few examples of overlapped users. The EmbedMe algorithm step employs a pre-trained language © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License et al. [6] to model relations between users and real-world els interest relations between users and entities, named past decades. Two key challenges for the growth of recMeKB. At the core of its advantages, (1) MeKB is seman- ommendation systems are, (1) how to efectively recomtically diverse, in that it leverages external information mend where user-item interactions are limited, namely from large Knowledge Graphs (KGs) to connect and or- the data sparsity problem; and (2) how to handle new ganize discrete user interests in the source domain; (2) users or items that enter the system, namely the cold-start MeKB is transferable, in that it forms a domain-agnostic problem. [13] To address these problems, Cross-Domain graph of user interests that are semantically stable for Recommendation (CDR) is an increasingly popular direccross-domain transfers. tion. The setting of CDR involves two recommendation
To build MeKB, we first perform Entity Linking (EL) domains, namely “source” and “target”, and the key idea is [7, 8] on items’ textual content to identify their entities of to transfer users’ representations from the source domain interest. Then we connect the entities with users through to assist recommendation in the target domain. [14] user-item interactions (e.g. likes or reviews) in the source According to diferent recommendation settings, CDR domain. These steps establish MeKB, a sub-graph for approaches are divided into sub-problems: user (or item) the global KG centered around the user to express their non-overlap, user (or item) partial overlap and user (or interests. MeKB is essentially a general interest storage item) full overlap [2]. This paper studies the scenario that represents user interests as world entities. of user partial overlap, which is an important case that
With MeKB constructed, it remains non-trivial to make occurs frequently in real-world recommender systems. cross-domain recommendations with this rich user rep- In the user partial-overlapped scenario, existing CDR resentation. We notice that Pretrained Language Models approaches typically focus on building feature represen(PLMs) are recognized as good few-shot learners [9, 10]. tations of users that can carry over from the source to Here we propose EmbedMe, a representation learning the target domain. One common approach is Embedding mechanism to map semantic knowledge in MeKB to fit & Mapping. Specifically, a user’s feature in the source recommendation tasks in the target domain. By combin- domain is represented by an embedding , and the user ing MeKB with PLMs, EmbedMe can elevate the semantic feature in the target domain is represented as . During knowledge to significantly ease the learning process in the transfer learning process, the model is trained by CDR, a novel approach to bring world knowledge in PLMs overlapped users to learn a mapping function , which into recommender systems. In particular, EmbedMe first maps the feature into a downstream-friendly reprelfattens MeKB into a weighted sequence of entities, and sentation in the target domain. [15]. The object is to then encodes the sequence with a PLM-powered trans- establish the relationship between the latent space of former encoder to retrieve recommendation results for domains: the target domain.The construction of MeKB and the min ∑ ( ( ; ), ). (1) EmbedMe mapping model together form the MeKB-Rec ∈ system, as illustrated in Figure 1. Embedding-and-mapping approaches aim at transferring
We conduct experiments with MeKB-Rec on two real- representations across domains [16, 17, 18], but the single world recommendation products. We first benchmark on feature embedding become the bottleneck for expressing three settings in the Amazon dataset which is commonly a user’s diverse interest. Besides, learning the embedused for CDR evaluations [11, 12]. Compared with prior ding requires a large amount of examples for overlapped methods, MeKB-Rec achieves the state-of-the-art perfor- users. In real-world CDR problems, the variety of user mance, outperforming other systems by a large margin features and the lack of overlapped users’ interactions on HR@10 and NDCG@10 metrics (varying from 24% can limit the application scope of the Embedding & Mapto 91%). We will publish this processed dataset to facili- ping method. tate future research in the camera-ready. Furthermore, To utilize more user features across domains beyond we conduct ofline and online experiments on WeiXin, a the traditional feature embedding, another approach is popular online platform with billions of users. Besides building shared graphs to represent the relationships improvements in ofline metrics, MeKB-Rec provides sig- among users, items, attributes, and other factors. The nificant DAU and PCTR gains in online A/B testing of constructed graph integrates the information over both WeiXin Live recommendation. MeKB-Rec is now de- source and target domains, and by using Graph Neural ployed into the product to serve hundreds of millions of Networks, the users’ representations are embedded into a users. shared latent space [19, 20, 21, 22]. Although such methods successfully introduce more features in the source 2. Related Work domain and improve CDR performance, they tend to ignore the rich semantics behind users’ discrete interest points, and the insuficient amount of overlapped users’ behavior leads to significant dificulty for optimization.
Entity Linking on Items
Global Knowledge Graph
MeKB-Rec, a two-stage recommendation system, as illustrated in Figure 1. The first stage is building MeKB, a personal knowledge graph that connects users and entities via interest relations, derived from interactions in the source domain. The second stage is EmbedMe, an efective representation learning model that maps MeKB to a downstream embedding for retrieval in the target domain. 3.1. MeKB Construction MeKB is an instance of Personal Knowledge Graphs (PKGs) [23]. It distills user-item interactions into userentity interest relations by mapping items into entities of interest. Diferent with prior works that learns latent interest representations, MeKB is an explicit interest store for a user’s interests in the form of entities.The overall process for MeKB construction is shown in Figure 2. 3.1.1. Knowledge Graph. 26], and have been widely adopted for applications such as Search and Question Answering [27]. There have been eforts to use KGs for recommendation [ 28, 29], but few work used KGs in the CDR setting. Personal
a new concept to model user-entity interactions. Some work adopted PKGs for recommendations [6], though most work focused on news recommendations and few on the CDR problem.
In this paper, we use Wikidata [30] as our KG.The set of all entities in KG is denoted as . Each ∈ is an entity: persons, organizations, locations, works etc, usually with a Wikipedia page. The set of relations between entities is denoted as . Each ∈ is a triple between two entities: (1, , 2) where 1 ∈ and 2 ∈ . Each entity comes with a unique title () for the given language.
3.1.2. Entity Linking on Items.
to connect items with entities of interest in KG. Entity
proposed to represent structural world knowledge [24, 25, the source domain. We first extract a subgraph of the global KG by performing entity linking on items. Linked entities are weighted by Frequency Aggregation on user-item interactions, and an Interest IDF scoring function. recommendation items with KG to build its semantic representation.
scenario. Given KG entities and a recommendation item ∈ , denote the textual content of the item as . An Entity Linking system outputs as set of enare correlated with the item text, denoted tities: { 1, 2, … , }, where ∀∈[1,] ∈ . Output entities
to extract an Alias Table (AT) from Wikipedia, by parsing
Alias Table defines the prior probability of a text mention linking to an entity , estimated with the formulation: ∀∈[1,] ( (|) = (, ) () : (2) (3) , ) where (, )
denotes the number of anchor texts with the surface form
with reference to the entity page denotes the total number of greater than a certain threshold K, that is
As our experiment scenarios span English and Chi- ios, it is beneficial to reduce the weight of entities with in Wikipedia, and () anchor texts with the surface form .
To enable application on a large quantity of items, for the sake of simplicity and eficiency, we directly use the tion, which is considered a good EL benchmark across languages [34]. nese languages, we choose available EL methods in both languages: for English datasets, we first perform named entity recognition (NER) on the textual content of the ing [31, 33] with the constructed alias table to map the name to the entity. For Chinese datasets, we directly match against the item’s textual content against the alias 3.1.3. User-Item Interaction Aggregation.
actions and collecting entities on items, we can already build the first version of MeKB. However, naive frequency-based aggregations assume that entities appearing more frequently better represent users’ interest, which is often inaccurate in actual recommender systems.
world scenes, we devise an aggregation mechanism to adjust entity weights in the personal knowledge graph.
Concretely, the global KG reflects the global knowledge of the world. KG entities have equal rights, and the edges between entities only reflect the objective world relations. However, for the recommendation system, it remains challenging to model which entities play a better role in representing users’ interests. On one hand, counting the number of items that associate with each entity gives a first estimation of the interest’s strength. On the other hand, this frequency-based measurement may not be ideal as it can overestimate entities with higher global frequency. Therefore, for these two factors, we respectively integrate two interaction aggregation methods into the entity weight of MeKB: 1) Frequency Aggregation and 2) Interest IDF Scoring.
1) Frequency Aggregation. We first set a frequency score for each entity in the original personal knowledge graph, according to the number of user-item interactions that are associated with each entity. For example, a user clicks on 1 videos associated with entity Association_football and 2 videos with Manchester_United_F.C.. Then the initial frequency scores of the corresponding entities are set to 1 and 2. Considering that some entities with extra-high frequency may dominate the head distribution, we perform a log smoothing for the frequency () = { ≤
* [1 + log ∗ ] > * 2) Interest IDF Scoring. Considering the deviation of popularity distribution in actual recommendation scenarhigher exposure so that the scores in MeKB are more in line with the actual strength of users’ interests. We are inspired by the concept of IDF [36] from natural language processing, to judge the contribution of an interest entity in recommendation. Intuitively, IDF is correlated with information gain. An IDF-weighted interest score downweights popular interests and reduces exposure bias on the recommendation system. The IDF calculation formula for each entity is as follows. (4) (5) (6)
: (7) =1 ([]) = log { / ∑ ([], []) } ([], []) [] that is where [] is the user and the [] is the entity , and is an indicator function for whether the interest relationship between [] and []
exists. We note that is the cooccurrence frequency of [] and as calculated by the Frequency Aggregation step, ([], []) = ( ) Then we perform IDF on the entity frequency score to obtain the final interest score of MeKB, denoted ([], []) = ([], []) ⋅ ([]) ∑
=1 ([], []) ⋅ ([])
3.2. EmbedMe Algorithm Although MeKB makes user interest semantically valid across domains, there is still an essential dificulty for learning to make accurate recommendations from MeKB to items in the target domain. The critical challenge is that, although MeKB adopts large knowledge graphs to bring in a large amount of auxiliary entity information and interest associations, the scarcity of CDR samples makes it dificult to implement end-to-end training with this amount of auxiliary features. On one hand, MeKB has a holistic view of a user’s interests as a set of entities in the global KG, each of which is semantically meaningful. On the other hand, the overlapped users’ behavior data is not enough to support end-to-end training of recommendation systems for all PKGs in the CDR setting.
MeKB (User) Embedding
MLP
Mean Pooling "̂$/ "%̂/ "̂&/ "̂'/ "̂(/ "̂)/ "̂*/ "̂+/ "̂,/ "̂-/ "%̂/$ "%̂/% "%̂/& ⋯ ⋯ "̂./
Multi-layerTransformer Encoder (Pretrained Language Model) "$ "% "& "' "( ") "* "+ ", "- "%$ "%% "%& ⋯ ⋯ ". [CLS] Machine Learning [SEP] Computer Vision [SEP] Footbal [SEP] LOL [SEP] Premier League [SEP] !! !" !# !0 !1
User’s MeKB Entity Sequence
Item3047 Item342 User Item8253 Item9236 …
Items inTarget Domain
Item Embedding
MLP Item342: Cute and Funny CatVideos Compilation Item3047:The Top 100 Plays of 2022! MLB Highlights … Item8253:AI forum:AIGC progress and challenges Item9236: POP Dance Tutorial (Step-by-step)
To overcome this challenge, we present an algorithm structure. Therefore, we turn a user ’s interest graph to map a user’s personal knowledge graph, MeKB, into a into a sequence of natural language tokens, ordered by dense embedding representation suitable for retrieval in their adjusted score (, ) in descending orthe target domain. We name the algorithm EmbedMe. der. By encoding over the sequential input, the PLM can
To implement EmbedMe, we transform the personal learn to encode cross-interest interactions to better fit knowledge graph into a semantically weighted sequence recommendation tasks. Each entity is represented by representation of entities, and then introduce a Pre- its unambiguous title () (Wikipedia title or name trained Language Model (PLM) as the semantic encoder in domain-specific KGs), and tokenized via WordPiece to map the entity sequence to an embedding. We fine- following Devlin et al. [39]. Entities are separated by a tune this semantic encoder together with the item repre- special [ ] token. For eficiency, the entity sequence is sentation in the target domain via a dual-encoder archi- truncated to a maximum of tokens ( = 256 ). We add tecture. a mean-pooling layer over PLM’s sequence output on all tokens, followed by a Multi-layer Perceptron (MLP). The 3.2.1. Pretrained Language Model MLP transforms the output to an embedding dimension of ( = 256 ), which is the user’s MeKB Embedding.
cessful thanks to the help of Pretrained Language Models 3.2.3. End-to-end Training. (PLMs) such as BERT and GPT [37, 38], leading to significant improvements in Search, Advertisement and Assis- For end-to-end training of MeKB Embedding with the tant [39]. PLMs are known for their few-shot learning specific CDR task, we employ a dual-encoder framework ability [10], which makes them potentially suitable for that aligns MeKB Embedding and the learned item’s emCDR tasks with limited training examples. bedding in the target domain, into the same representa
Our core idea is applying PLMs on a textual represen- tion space. The overall training architecture is illustrated tation of MeKB to model user interest. The advantage in Figure 3. of the PLM is that it can absorb a wide range of corpus Item representation. Since item representation is for robust semantic understanding, but there is still gap not the focus of the work, we choose to represent items between the pretrained semantic representation and the simply by its ID embedding, which is modeled through specific recommendation system. Therefore, we devise a weight matrix with dimension [ , ] , where is the an end-to-end fine-tuning method of the PLM for the number of items in the target domain, and is the same CDR-specific scenario to extract task-oriented user em- embedding dimension. bedding from MeKB. Measurement. The user and item embedding are optimized with dot-product similarity. We define the 3.2.2. MeKB Sequence Encoder. score between a user and item as their embeddings’ dot-product similarity:
forms MeKB to an output embedding of the user, named MeKB Embedding. Since MeKB is inherently a graph structure, it becomes challenging for PLMs to exploit the (, ) = () (), (8) where is the MeKB sequence encoder, and is the item encoder, projecting user and item representations into -dimensional vectors respectively.
Optimization. The dual encoder is trained on positive interactions in the target domain, i.e. records in the target domain that have a non-empty MeKB input. In our experiments, MeKB can be constructed with user interactions in both source and target domains, which yields more available examples than traditional settings where only users with at least a number of actions in both domains are selected for training. [12]
For each positive example, we optimize for full softmax loss [40, 27] over all items with respect to the measurement: (, ) = − ((, )) ∑ =0 ((, )) where is the number of items in the target domain.
pare the recommendation performance of the proposed
4.1. Dataset
mains in Amazon: (1) Books, (2) Movies and TV, and (3) CDs and Vinyl, abbreviated as Books, Movies and
Movies, Amazon Books → Music, and Amazon Movies →
and place all their records into the test set, to verify our efectiveness on cold-start users.
extracted from WeiXin recommendation scenarios. The dataset comes from real user-item interaction on the
mains: Articles, Channels (short videos), and Live (live broadcasts). We set up two cross-domain recommendation scenarios that are WeiXin Articles → Videos and WeiXin Videos → Live, and construct CDR datasets with millions of records. The detailed parameters of the Amazon and WeiXin datasets used in the experiments are shown in Table 3. The Overlap Ratio is the fraction of overlapped users of two domains over the number of users in the current domain, i.e. ( , ) = | ∩ |/| |.
(9) datasets. 4.2. Experiment Settings In the experiments, we randomly divide each dataset into training, validation and test sets according to the ratio of 8:1:1. We also put 20% overlapped users in test set of target domain for cold-start evaluation. In every experiment, we repeat the training and testing process five times and take the average as the final result. In terms of evaluation methods, we adopt Hit Rate @K and NDCG @K as the evaluation criteria for the recommendation systems. We choose K=10 and sample 1 positive item and 999 negative items to compute the metrics for all
For the implementation of MeKB-Rec, we distribute a total of 1024 examples into 8 NVIDIA A100 GPUs to accelerate training. We adopt Lamb optimizer [41] with a weight decay of 5e-4, and set its initial learning rate to 5e
4.3. Baseline We compare our proposed MeKB-Rec method with the following state-of-the-art recommendation methods, among which the first two are general recommendation methods, while others are specifically for CDR: (1) YoutubeDNN [42] is a popular and eficient DSSM model for recommendation retrieval tasks, not specifically designed for CDR.
(2) CMF: Collective Matrix Factorization [43] adopts a Matrix Factorization (MF) method to simultaneously factorize the rating matrices across the source and target domains, where user representations are shared between two domains.
(3) CoNet: Collaborative Cross Networks [5] replaces matrix factorization with neural networks in CDR problem, and builds a cross-mapping to connect hidden layers in two base networks.
(4) EMCDR: Embedding-and-Mapping framework for
items. A mapping function with linear and non-linear
space for overlapping users in both domains.
(5) DCDCSR: a Deep framework for both Cross
posed to resolve the two challenges. The method is based on Matrix MF models and a fully connected Deep Neural
(6) DDTCDR: Deep Dual Transfer Cross Domain Recommendation [3] uses a pretrained autoencoder for user and item representation. Users’ within-domain preferences and cross-domain preferences are jointly modeled.
WeiXin
Videos → Live 0.0816 0.1855 +113.92% +102.89% +112.29% +127.33% Note that the auto-encoders are pre-trained on recom- user activity, and find that MeKB-Rec is particularly efmendation data without incorporating world semantic fective for zero-shot and few-shot users. These analyses knowledge. are in Appendix B.
(7) SSCDR: Semi-Supervised framework for CrossDomain Recommendation [45] is proposed to utilize both 4.4.2. WeiXin Dataset overlapping users and source-domain items to train the overlapping function. 4.4. Results
4.4.1. Amazon Dataset
On Amazon Datasets, MeKB-Rec achieves uniform im- knowledge graph, MeKB, as a medium for expressing provements on all settings, a 24%–91% gain over the cross-domain interests. As the method strikes CDR persecond-best methods. The advantage in the Books → formance to a new level, we analyze why it is efective Movies scenario is smaller than that in the other two sce- from two standpoints: narios. We hypothesize that CDR settings with a larger 1) Domain-agnostic interest representation of MeKB. domain gap or a lower user overlap ratio may observe MeKB uses global entities to represent user interests. Enmore advantages by using our method. tities are domain-agnostic elements that are semantically
We further conduct additional ablation studies to vali- stable, therefore can naturally serve as units of interests date the efectiveness on PLMs, and show that the pre- that are valid across domains, say from Movies to Music, trained parameters play a key role on the efectiveness of or from Articles to Videos.
MeKB-Rec. We conduct a frequency-binned analysis on 2) Few-shot knowledge transfer of PLMs. Traditional CDR methods sufer from limited training examples of
tween MeKB-Rec and the established YoutubeDNN baseline. [42] Other methods are not compared due to performance reasons at the scale of this dataset. As shown in Table 2, on both CDR settings tested, MeKB-Rec performs significantly better than YoutubeDNN, and the margin of advantage is larger than that of the Amazon dataset. 4.5. Analysis on MeKB-Rec’s Efectiveness ric. The error bar stands for the 95% confidence interval. Overall, the experiment group with MeKB-Rec achieves a significant gain of DAU +0.531% ( < 0.01 ) for non-active users, suggesting strong improvements over cold-start (zero-shot and few-shot) users. This pool of users accounts for a significant proportion and is critical for the growth of the recommendation scenario. The overall UCTR of all users is improved by +0.240% ( < 0.01 ), indicating that users are generally more in favor of the recommended content compared to the baseline.
As online experiments show significant gains, MeKBRec has been deployed to the production system for 100% of non-active users in this scenario.
In this paper, we propose a novel knowledge-graph-based overlapping users. As shown in subsection B.2, they fail method, MeKB-Rec, that significantly improves Crossto generalize to users with no or few activities in the Domain Recommendation (CDR) performance and retarget domain. MeKB-Rec on the other hand, can benefit duces the requirement for overlapped users’ information. world knowledge learned from PLMs to understand and MeKB-Rec is a two-phase method, firstly constructing transfer the semantics to better serve recommendations MeKB, a user’s personal knowledge graph, and secondly in the target domain. The EmbedMe training procedure mapping MeKB entities into a semantic embedding space is essentially a few-shot fine-tuning step of the original for recommendations in the target domain, namely the PLM to fit the target recommendation task. As PLMs EmbedMe algorithm. MeKB-Rec achieves state-of-theare good few-shot learners, this approach is especially art recommendation results on multiple CDR scenarios efective. In subsection B.1 we show that by removing in Amazon and WeiXin. The method also achieved sigthe pretrained parameters, the performance drops by nificant gains in online experiments in WeiXin Live rec25%–40%. ommendation, and is deployed in the product serving hundreds of millions of users. 4.6. Online Experiments Specifically, MeKB-Rec has achieved benefits in the following aspects: (1) obtained substantial advantages on We conduct A/B experiments in the WeiXin online rec- large-scale CDR datasets: 38%–67% relative improvement ommendation system serving hundreds of millions of of HR@10 on Amazon and 112%–114% gain of HR@10 on users. We report results in a major Live-stream recom- WeiXin datasets; (2) utilized the information contained mendation scenario in the platform, where we deploy in the global KG, to supplement the feature represenMeKB-Rec to leverage user behaviors in the short video tation for user interests that carry across domains; (3) domain to improve the recommendation for live streams, used PTMs to eficiently learn on insuficient training especially for cold-start users. examples. We conclude that injecting global semantics
The control group is the current recommendation sys- via KG and PTM largely eases learning when the data tem in production. The experiment group is the same is sparse. This assumption is validated by MeKB-Rec’s system with MeKB-Rec as an additional retrieval step noticeable wins on zero-shot and few-shot user groups for recommendation candidates. Experiments are run (respectively 105% and 66% HR@10 gains on Amazon) for 9 days, during which the item tower of MeKB-Rec is insuficient for traditional methods to learn a reliable continuously trained on an hourly basis, to fit recently- user representation. popular live broadcasters. We focus on two metrics: (1) Regarding limitations, we note that the quality of the DAU of non-active users in Live, and (2) Overall UCTR construction of MeKB is afected by the accuracy of the for all users, which is the unique click-through rate at entity linking method. Specifically, the ambiguity of the user level. entity names or the insuficient coverage of entities will
The experimental result is shown in Figure 4. The X- cause the representation deviation of MeKB. Future work axis is the day of the experiment, and Y-axis is the relative may exploit more comprehensive entity linking models to percentage improvement of the experiment group com- improve MeKB’s expressiveness and MeKB-Rec’s overall pared to the control. The metric shown in the DAU chart results. is accumulative while the UCTR chart shows a daily met- The work of this paper is an important attempt at knowledge-graph-based methods in Cross-Domain Rec- [10] T. Gao, A. Fisch, D. Chen, Making pre-trained ommendations. Future work may focus on: (1) combin- language models better few-shot learners, in: ing knowledge graphs with Large Language Models to Proceedings of the 59th Annual Meeting of the further improve recommendations; (2) leveraging more Association for Computational Linguistics and sophisticated content understanding methods to improve the 11th International Joint Conference on Natthe construction of MeKB; (3) better representation learn- ural Language Processing (Volume 1: Long Paing mechanisms to improve Embedme; and (4) explor- pers), Association for Computational Linguising more usage for personal knowledge graphs such as tics, Online, 2021, pp. 3816–3830. URL: https:// MeKB. aclanthology.org/2021.acl-long.295. doi:10.18653/ v1/2021.acl-long.295. [11] J. Ni, J. Li, J. McAuley, Justifying recommendations References using distantly-labeled reviews and fine-grained aspects, in: Proceedings of the 2019 conference on [1] A. I. Schein, A. Popescul, L. H. Ungar, D. M. Pen- empirical methods in natural language processing nock, Methods and metrics for cold-start recom- and the 9th international joint conference on natmendations, in: Proceedings of the 25th annual ural language processing (EMNLP-IJCNLP), 2019, international ACM SIGIR conference on Research pp. 188–197. and development in information retrieval, 2002, pp. [12] J. Cao, X. Lin, X. Cong, J. Ya, T. Liu, B. Wang, Dis253–260. encdr: Learning disentangled representations for [2] T. Zang, Y. Zhu, H. Liu, R. Zhang, J. Yu, A sur- cross-domain recommendation, in: Proceedings vey on cross-domain recommendation: taxonomies, of the 45th International ACM SIGIR Conference methods, and future directions, arXiv preprint on Research and Development in Information RearXiv:2108.03357 (2021). trieval, 2022, pp. 267–277. [3] P. Li, A. Tuzhilin, Ddtcdr: Deep dual transfer cross [13] Y. Zhu, J. Lin, S. He, B. Wang, Z. Guan, H. Liu, domain recommendation, in: Proceedings of the D. Cai, Addressing the item cold-start problem 13th International Conference on Web Search and by attribute-driven active learning, IEEE TransacData Mining, 2020, pp. 331–339. tions on Knowledge and Data Engineering 32 (2019) [4] M. Liu, J. Li, G. Li, P. Pan, Cross domain recommen- 631–644.
dation via bi-directional transfer graph collabora- [14] A. Da’u, N. Salim, Recommendation system based tive filtering networks, in: Proceedings of the 29th on deep learning methods: a systematic review and ACM International Conference on Information & new directions, Artificial Intelligence Review 53 Knowledge Management, 2020, pp. 885–894. (2020) 2709–2748. [5] G. Hu, Y. Zhang, Q. Yang, Conet: Collaborative [15] T. Man, H. Shen, X. Jin, X. Cheng, Cross-domain cross networks for cross-domain recommendation, recommendation: An embedding and mapping apin: Proceedings of the 27th ACM international con- proach., in: IJCAI, volume 17, 2017, pp. 2464–2470. ference on information and knowledge manage- [16] Y. Zhang, Y. Liu, P. Han, C. Miao, L. Cui, B. Li, ment, 2018, pp. 667–676. H. Tang, Learning personalized itemset mapping [6] Y. Yang, J. Lin, X. Zhang, M. Wang, Pkg: A per- for cross-domain recommendation., in: IJCAI, 2020, sonal knowledge graph for recommendation, in: pp. 2561–2567.
Proceedings of the 45th International ACM SIGIR [17] P. Li, A. Tuzhilin, Dual metric learning for efective Conference on Research and Development in Infor- and eficient cross-domain recommendations, IEEE mation Retrieval, 2022, pp. 3334–3338. Transactions on Knowledge and Data Engineering [7] X. Ling, S. Singh, D. S. Weld, Design challenges (2021).
for entity linking, Transactions of the Associa- [18] Y. Zhu, K. Ge, F. Zhuang, R. Xie, D. Xi, X. Zhang, tion for Computational Linguistics 3 (2015) 315–328. L. Lin, Q. He, Transfer-meta framework for crossURL: https://www.aclweb.org/anthology/Q15-1023. domain recommendation to cold-start users, in: doi:10.1162/tacl_a_00141. Proceedings of the 44th International ACM SIGIR [8] W. Zhang, W. Hua, K. Stratos, Entqa: Entity Conference on Research and Development in Inforlinking as question answering, arXiv preprint mation Retrieval, 2021, pp. 1813–1817. arXiv:2110.02369 (2021). [19] X. Wang, X. He, L. Nie, T.-S. Chua, Item silk road: [9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- Recommending items from information domains plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- to social users, in: Proceedings of the 40th Intertry, A. Askell, et al., Language models are few-shot national ACM SIGIR conference on Research and learners, Advances in neural information process- Development in Information Retrieval, 2017, pp. ing systems 33 (2020) 1877–1901. 185–194. [20] Q. Cui, T. Wei, Y. Zhang, Q. Zhang, Herograph: A [32] T. Févry, N. FitzGerald, L. B. Soares, T. Kwiatkowski, heterogeneous graph framework for multi-target Empirical evaluation of pretraining strategies for cross-domain recommendation., in: ORSUM@ Rec- supervised entity linking, in: Automated KnowlSys, 2020. edge Base Construction, 2020. [21] J. Li, Z. Peng, S. Wang, X. Xu, P. S. Yu, Z. Hao, Het- [33] Z. Xu, Z. Shan, Y. Li, B. Hu, B. Qin, Hansel: A chierogeneous graph embedding for cross-domain rec- nese few-shot and zero-shot entity linking benchommendation through adversarial learning, in: In- mark, arXiv preprint arXiv:2207.13005 (2022). ternational Conference on Database Systems for Ad- [34] J. A. Botha, Z. Shan, D. Gillick, Entity Linking in 100 vanced Applications, Springer, 2020, pp. 507–522. Languages, in: Proceedings of the 2020 Conference [22] K. Xu, Y. Xie, L. Chen, Z. Zheng, Expanding re- on Empirical Methods in Natural Language Prolationship for cross domain recommendation, in: cessing (EMNLP), Association for Computational Proceedings of the 30th ACM International Confer- Linguistics, Online, 2020, pp. 7833–7845. URL: https: ence on Information & Knowledge Management, //aclanthology.org/2020.emnlp-main.630. doi:10. 2021, pp. 2251–2260. 18653/v1/2020.emnlp-main.630. [23] K. Balog, T. Kenter, Personal knowledge graphs: [35] M. Honnibal, I. Montani, S. Van Landeghem, A research agenda, in: Proceedings of the 2019 A. Boyd, spaCy: Industrial-strength Natural LanACM SIGIR International Conference on Theory of guage Processing in Python (2020). doi:10.5281/ Information Retrieval, 2019, pp. 217–220. zenodo.1212303. [24] D. Vrandečić, M. Krötzsch, Wikidata: a free col- [36] S. Qaiser, R. Ali, Text mining: use of tf-idf to examlaborative knowledgebase, Communications of the ine the relevance of words to documents, InternaACM 57 (2014) 78–85. tional Journal of Computer Applications 181 (2018) [25] X. Chen, S. Jia, Y. Xiang, A review: Knowledge 25–29.
reasoning over knowledge graph, Expert Systems [37] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, with Applications 141 (2020) 112948. Bert: Pre-training of deep bidirectional transform[26] C. Zhang, C. Ré, A. Sadeghian, Z. Shan, J. Shin, ers for language understanding, arXiv preprint F. Wang, S. Wu, Feature engineering for knowledge arXiv:1810.04805 (2018). base construction, IEEE Data Eng Bull (2014). URL: [38] L. Floridi, M. Chiriatti, Gpt-3: Its nature, scope, https://arxiv.org/abs/1407.6439. limits, and consequences, Minds and Machines 30 [27] T. Févry, L. Baldini Soares, N. FitzGerald, E. Choi, (2020) 681–694.
T. Kwiatkowski, Entities as experts: Sparse [39] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: memory access with entity supervision, in: Pre-training of deep bidirectional transformers for Proceedings of the 2020 Conference on Empirical language understanding, in: Proceedings of the Methods in Natural Language Processing (EMNLP), 2019 Conference of the North American Chapter of Association for Computational Linguistics, On- the Association for Computational Linguistics: Huline, 2020, pp. 4937–4951. URL: https://www. man Language Technologies, Association for Comaclweb.org/anthology/2020.emnlp-main.400. putational Linguistics, Minneapolis, Minnesota, doi:10.18653/v1/2020.emnlp-main.400. 2019, pp. 4171–4186. URL: https://www.aclweb.org/ [28] X. Wang, D. Wang, C. Xu, X. He, Y. Cao, T.-S. Chua, anthology/N19-1423. doi:10.18653/v1/N19-1423.
Explainable reasoning over knowledge graphs for [40] J. Ling, N. FitzGerald, Z. Shan, L. B. Soares, T. Févry, recommendation, in: Proceedings of the AAAI con- D. Weiss, T. Kwiatkowski, Learning cross-context ference on artificial intelligence, volume 33, 2019, entity representations from text, arXiv preprint pp. 5329–5336. arXiv:2001.03765 (2020). URL: https://arxiv.org/abs/ [29] X. Wang, T. Huang, D. Wang, Y. Yuan, Z. Liu, X. He, 2001.03765.
T.-S. Chua, Learning intents behind interactions [41] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhowith knowledge graph for recommendation, in: janapalli, X. Song, J. Demmel, K. Keutzer, C.-J. Proceedings of the Web Conference 2021, 2021, pp. Hsieh, Large batch optimization for deep learn878–887. ing: Training bert in 76 minutes, arXiv preprint [30] D. Vrandečić, Wikidata: A new platform for collab- arXiv:1904.00962 (2019).
orative data collection, in: Proceedings of the 21st [42] P. Covington, J. Adams, E. Sargin, Deep neural netinternational conference on world wide web, 2012, works for youtube recommendations, in: Proceedpp. 1063–1064. ings of the 10th ACM conference on recommender [31] N. De Cao, G. Izacard, S. Riedel, F. Petroni, Au- systems, 2016, pp. 191–198.
toregressive entity retrieval, in: International Con- [43] A. P. Singh, G. J. Gordon, Relational learning via ference on Learning Representations, 2021. URL: collective matrix factorization, in: Proceedings of https://openreview.net/forum?id=5k8F6UU39V. the 14th ACM SIGKDD international conference
Amazon Datasets. The Amazon dataset is a widely used public recommendation dataset. We selected three popular domains in Amazon: (1) Books, (2) Movies and TV, and (3) CDs and Vinyl, abbreviated as Books, Movies and Music. In the CDR problem, we use these three domains of data to construct three cross-domain scenarios.
Following Cao et al. [12], Liu et al. [4], two domains are combined into one CDR scenario. We combine the domains into three CDR scenarios: Amazon Books → Movies, Amazon Books → Music, and Amazon Movies → Music.
The degree of the semantic gap between domains in these scenarios is diferent, which can verify the robustness of the methods tested. Positive interactions are defined as a review score of 4 or above. Following prior work [12], we filter out users and items based on the frequency of their interactions. Specifically, we filter out users with less than five positive interactions and filter out items with <50 positive interactions as the focus of this work is not on cold-start items. We randomly select 20% of the overlapped users and place all their records into the test set, to verify our efectiveness on cold-start users.
WeiXin Datasets. We build large-scale CDR datasets on Knowledge discovery and data mining, 2008, pp. extracted from WeiXin recommendation scenarios. The 650–658. dataset comes from real user-item interaction on the [44] F. Zhu, Y. Wang, C. Chen, G. Liu, M. Orgun, J. Wu, A WeiXin platform, involving three recommendation dodeep framework for cross-domain and cross-system mains: Articles, Channels (short videos), and Live (live recommendations, arXiv preprint arXiv:2009.06215 broadcasts). We set up two cross-domain recommen(2020). dation scenarios that are WeiXin Articles → Videos and [45] S. Kang, J. Hwang, D. Lee, H. Yu, Semi-supervised WeiXin Videos → Live, and construct CDR datasets with learning for cross-domain recommendation to cold- millions of records. To construct this dataset, we consider start users, in: Proceedings of the 28th ACM In- reading an article, fully watching a video, and watching ternational Conference on Information and Knowl- a live stream for ≥ 30 seconds as positive behaviors. We edge Management, 2019, pp. 1563–1572. randomly sample all positives within two months for training and the week after for testing. Similarly, we filter out items with <50 positives. When constructing the A. Dataset Details data set, we anonymized the user information, and only kept the user’s anonymous id and sampling behavior samples required for the experiment. Compared with the Amazon dataset, the WeiXin dataset covers a much larger scale of users and items. The detailed parameters of the Amazon and WeiXin datasets used in the experiments are shown in Table 3. The Overlap Ratio is the fraction of overlapped users of two domains over the number of users in the current domain, i.e. ( , ) = | ∩ |/| |.
Dataset-specific KG augmentation. Although Wikidata is a general KG with rich world information, complementing it with domain-specific knowledge can be helpful for the recommendation. We further incorporate dataset-specific entities into the KG to be considered in building MeKB, to increase item coverage and enrich user representations. For the Amazon dataset, we include three types of entities available in the dataset: brands, categories, and products, each represented by its textual name. They are considered as additional entities with a lower weight (set to 0.5, 0.3 and 0.1 respectively, where original Wikidata entities are assigned a weight of 1.0) Entity Linking on these entities is trivial, as each item in the dataset already have these entities denoted. For the WeiXin dataset, we include an additional keyword dictionary as domain-specific entities, such as hashtags in videos. Custom entities are associated with items via YouTubeDNN CMF CoNet EMCDR DCDCSR DDTCDR SSCDR MeKB-Rec Improved keyword matching. not cover the semantic information of all entities in the
Following prior work [12, 45], examples in both source global KG. The introduction of the pre-trained language and target domains can be used in CDR evaluation. There- model enables the semantic understanding of entities fore we combine interactions in both domains with equal in the global KG by pre-training on massive external weights to construct a user’s MeKB, though in practice corpora. one can choose to only use the source domain to con- This hypothesis is confirmed in an ablation study struct a pure out-of-domain MeKB for specific tasks. We shown in Table 4. We observe a significant decline in only use examples in the training set to construct MeKB recommendation performance in MeKB-Rec, after dropto make sure there is no data leakage on validation or ping PLM parameters obtained during pretraining. The test examples. “w/o PLM” setting uses the same model architecture but replaces the PLM parameters with random initialization.
This indicates that the world semantic knowledge obB. Additional Experiments tained during pretraining is critical for modeling users’ interest in our model architecture. We suggest that fuB.1. Impact of Pretrained Language ture work may try to further pretrain the PLM on the
Models corpus from the recommendation scenarios to improve As mentioned above, MeKB has a strong representa- the adaptation efect to the recommendation system. tion capability of users’ interests, but this capability also brings the challenge of feature mapping for recommen- B.2. Frequency-binned Analysis on User dation. Our proposed MeKB-Rec alleviates the problems Activity of insuficient training records with the help of a pretrained language model. In essence, MeKB embodies a To understand how well our method performs for users large amount of world knowledge contained in the global with a limited amount of training data, we split users KG to build a complete expression of users’ interests. in the Movies test set based on their frequency seen in The training records in recommendation scenarios can- training data in the target domain, and report HR@10 numbers in each bucket. As shown in Table 5, MeKBRec achieves uniform improvement in all user groups compared to all baselines. Particularly, MeKB-Rec shows a significant advancement for the zero-shot user group (appearing zero times in the training set) and few-shot users (seen less than ten times in training) compared to other methods, with 105% and 66% HR@10 gains respectively. MeKB-Rec performs less well on high-activity users with more than 10 training examples, as the baseline methods usually have a dedicated ID embedding that ifts these users better after seeing many examples. As expected, MeKB-Rec is better for zero-shot and few-shot users which is a critical challenge in CDR that we aim to address.