We present the approach to the RecSys Challenge 2017, which ranked 7th. The goal of the competition was to prepare job recommendations for the users of the social network for business Xing.com. Our algorithm consists of two di erent models: Content-based User Proling and Local Collective Embeddings. The rst content-based model contains many hand-tuned parameters and data insights, so it performs fairly well on the task of the challenge despite its simplicity. The second model is based on Matrix Factorization and may be applicable to a wide range of cold-start recommendation tasks. The combination of these two models have shown the best performance on local validation.
Unlike last year's competition [
The task of o ine phase was standard cold-start recommendation problem while the online stage was focused on the push noti cations and was designed by the following scheme: { Every 24 hours teams needed to send a submission to the organizers; { For each job posting, a team needed to provide a list of 250 users; { Each user could receive 1 push per day.
The recommendations were delivered to the users through the following channels: activity stream in the mobile applications, jobs marketplace, emails, and recruiter tools. Some challenges that the participating teams needed to solve: { Balance between user interests and the interests of recruiters { Balance between relevance and revenue { Item cold-start problem { Smart targeting of push recommendations 2
In the challenge, the semi-synthetic dataset from XING was provided: arti cial users and events were added, text tokens from job postings titles and descriptions were replaced by numeric IDs and random noise was added in order to anonymize the data.
The organizers have provided datasets: { Interactions | interactions that the user performed on the job posting items (clicked, bookmarked, replied, recruiter interest or deleted) as well as details about items shown to users by the existing recommender (impressions). { Users | details about those users who appear in the above datasets: job roles, career level, discipline, industry, location, experience, and education. { Items | job postings details that were/will be recommended to the users: title, career level, discipline, industry, location, employment type, tags, created time and ag if item was active during the test.
On the online phase datasets contained 1M users, 853K items, 88.7M impressions, 4.2M interactions (84% clicked, 8.5% deleted, 4.3% bookmarked, 2.3% replied, 0.8% recruiter interest). Every 24 hours the organizers provided a new portion of target users and items for prediction. 3
Let T be a list of target items. For each target item i 2 T a recommender selects users to whom that item is pushed as a recommendation. Leaderboard score is computed as follows:
S =
X s(i; r(i)): Here, recommendations r(i) speci es the list of users who will receive the given item as a push recommendation.
Let Ur = r(i) be a list of recommended users. The score function s(i; Ur) sums up the success rates of the users Ur and the item-based success rate: s(i; Ur) = itemSuccess(i; Ur) + X userSuccess(i; u); u2Ur where userSucess scores a user-item pair according to the user interactions: userSuccess(i; u) = (u; r) premiumBoost(u); where is event type weight and premiumBoost is a boost coe cient for premium user (2) (3) (4) (5) (6) (7) (u; r) = 8 10; if delete only; > > ><1; if clicked; >5; if bookmarked or replied; > >:20; if recruiter interest; premiumBoost(u) = (2; if the user is premium;
1; otherwise: itemSuccess(i; Ur) = (w(i); if 9u 2 Ur : userSuccess(i; u) > 0
0; otherwise; where w is a boost coe cient for a paid item.
w(i) = (50; if the item is paid;
25; otherwise:
Evaluation measure shows the business value of di erent types of events from di erent types of users and di erent types of vacancies.
For local validation in the rst phase we used the last week of interactions. In the second phase, we used userSuccess score as a local score and recommendations were built only for those users and items that were present in the all currently available data from the daily chunks. 4
In our solution, we used two di erent approaches: Content-Based User Pro ling
and Local Collective Embeddings [
Content-Based User Pro ling
Our rst content-based [
Let U be a set of target users and I be a set of target items. For each pair (uk; ik) we check whether it has matches between User Job Title and Item Job Title (or tags) to reduce the number of the observed pairs. Then we apply following rankers to make top-N recommendations.
{ User Title ranker | rstly, we calculate 3 types of inverse document frequency for each token as (8) (9) (10) (11) (12) where juj is a number of tokens in the user's title. The nal ranker score for a pair (uk; ik) is calculated as a sum of token scores in intersection between User Title and Item Title (tags).
score(u; i) =
X scoret t { User Interest ranker | this ranker is quite similar to User Title ranker described above, but instead of scoring tokens from User Job Title, we score tokens from items target user interacted with. This scoring includes Item Job Title and Item Job Tags with weights based on the interaction type. { Industry, Discipline and Career Level rankers | those rankers also use information from both user pro les and user interactions. The Pro le Career Level ranker is based on the Career Level di erence between user and item: CLD(u; i) = juCL iCLj where U Ft stands for User Title, IFt stands for Item Title, and T Ft stands for Item Tags. For each user u we calculate the token weight as: Ft = log(
#unique tokens #token occurrences
) scoret = 20
U Ft IFt T Ft p u
j j wCL(u; i) = 8>1:2; if CLD(u; i) 1 <
0:7; if CLD(u; i) = 3 >:0:5; if CLD(u; i) 4 score(u; i) = wCL score(u; i) As the features Industry and Discipline are categorical, we multiply the pair score by w > 1 having exact match, and by w < 1 otherwise. Interactions rankers are slightly di erent from the pro le ones, and the main purpose of those rankers is to modify the score according to the ratio of parameter matches and the total user interactions. { User Behavior ranker | di erent users have di erent behavior patterns.
That ranker is based on user interaction history. If the user has all positive interactions, we add a constant to his score, if the user has all negative interactions, we subtract a constant from his score. Also, we multiply user score by a w < 1, which is based on the positive/negative interactions ratio. { Premium Ranker is a simple ranker that multiplies the score by 1:1 for a premium user and by 1:05 for a premium item. Weights were rst set manually based on our expertise and then they were optimized on the local validation set using a parameter grid.
Final predictions were made by applying all rankers to user-item pairs, thresholding, and taking top-100 users for each item. 4.2
Local Collective Embeddings
Local Collective Embeddings (LCE) [
Assume that we have: { a set of m properties stored in a matrix Xs 2 Rn m, where a row corresponds to an item and a column to an item property; { a set of u users stored in a matrix Xu 2 Rn u, where a cell (i; j) indicates whether the user j has shown interest to item i.
At the test time, we are given a new item q with description qs 2 R1 m, and our goal is to predict qu 2 R1 u, i.e., to score how likely is the user to show interest to the new item.
De ne LCE optimization problem:
To regularize this matrix factorization model we use manifold assumption: if two data points xi and xj , in any view, are close in the intrinsic geometry of the distribution, then their representations in the low-dimensional space should also be close to each other. Let's construct a graph with n nodes where each node represents a data point. For each point, we nd the p nearest neighbors and we connect the corresponding nodes in the graph.
This results in a matrix A which can later be used to measure the local closeness of two points xi and xj . Using the above de ned weight matrix A we may measure the smoothness of the low dimensional representation as: S = where D is a diagonal matrix which entries are the row sums of A (or column, as A is symmetric), i.e., Dii = P Aij ; L = D A is called the Laplacian matrix of the graph [10] and T r() is the trace operator.
This leads to the following optimization problem: where L is the Laplacian matrix of the graph, and is a hyper-parameter which controls the extent to which locality is enforced.
Learning algorithm have the following Expectation Maximization (EM)[
W
W
[ XsHsT + (1 [ W HsHsT + (1
)XuHuT + )W HuHuT +
AW ] DW + e109 u l a v e v i t c e jb108 O 75 100 125
Number of iterations
objective alpha beta gamma 150 175 200 25 50 100 500 1000
4000 200
k 200
k After parameter tuning we had local score of 649 for Content-Based User Pro ling and 405 for LCE. We generated 500 recommended items for each user from both models. Then we got the nal score for user-item pair as a weighted sum of scores of two models:
scorefinal = 0:8 scoreLCE + scorec0o:1n5tent based:
The coe cient choice for blending was based on the distribution of scores. The
content-based algorithm produced scores from 0 to 20K (approximately). LCE
scores were in 0-1 range. Therefore, the content-based model score was raised to
power p < 1. Then the exact values of the parameters were selected on the local
validation set. This ne-tuning scheme of two rankers by their linear combination
is similar to the scheme from [
In our approach, we used two di erent models: Content-based User Pro ling and Local Collective Embeddings. In this Challenge, the content-based approach have shown better results on local validation than the Local Collective Embeddings. A possible reason for the fact that the more sophisticated LCE algorithm has lost in quality to the simpler content-based method is that LCE was not customized enough for the task of competition. In particular, it was trained on binary data and did not take into account the di erent types of features (numeric, categorical, binary). However, the composition of the two algorithms had a better score.
The underlying rules leading to the observed results are as follows: within
the content-based approach it is very important to understand how the score for
each prediction is formed. In the Matrix Factorization approach [
The second phase of the competition can be considered as an A/B test. Obviously, it is hard to carry out for organizers (it is necessary to split into groups correctly, collect solutions from participants, etc.), but it may stimulate participants since they can compete in real-time and observe how the algorithm shows itself comparing to other algorithms. Daily feedback in the online phase was too rare to use it for automatic parameter tuning or multi-armed bandits approach. In the o ine stage, it was also important not to use features which we would not be able to use in the online stage.
The main scienti c contribution of the paper answers the question of how to use Local Collective Embeddings with user's side features. Originally, LCE was not designed to include user features.