Online academic repositories help millions of researchers discover relevant articles, a domain in which there are many potential signals of relevance, including text, citation links, and how recently an article was published. In this paper we present a case study of prodictionizing learning to rank for large scale recommendation, which utilises these diverse feature sets to increase user engagement. We first introduce itemto-item collaborative filtering (CF), then how these recommendations are rescored with a LtR model. We then describe offline and online evaluation, which are essential for productionizing any recommender. The online results show that learning to rank significantly increased user engagement with the recommender. Finally we show through post-hoc analysis that the original CF solution tended to promote older articles with lower traffic. However, by learning from subjective user interactions with the recommender system, our relevance model reversed those trends.
The rate of scientific discovery is ever increasing, with new methods, theory and
practice being published each day. This growth poses a challenge for researchers,
who need to stay up to date. Online academic catalogues, such as
ScienceDirect,1 give users access to large amounts of peer reviewed scientific publications.
However, the experience of using such catalogues can be characterised as a
combination of information overload [
The specific use case we aim to address is to help users of ScienceDirect
find additional relevant articles that go beyond explicitly encoded relationships
such as authorship or publication venue. In contrast to the personalised research
article recommendations in platforms such as Mendeley [
In this paper, we describe how we built an initial system using item-based collaborative filtering (IBCF). Once this system is in production, we collected data on which recommendations users preferred, and then trained a Learning to Rank (LtR) model to re-rank the recommendations using a range of article features and similarity metrics. We focus on the evaluation methodology and how the recommendations surfaced by LtR differed from the pure collaborative filtering system. These give insight into what the LtR system achieves that could not be done with collaborative filtering (CF) alone.
Recommender Systems (RS) have become key tools in a researcher’s
content discovery toolkit, as they not only allow them to more efficiently navigate
large and ever-growing catalogues, but also discover content that they would not
have seen otherwise. Previous work has resulted in a distinction between
methods used for personalised and non-personalised approaches. This can be seen
in the use of implicit feedback and CF for personalised recommendations [
Through this research, we ultimately found that to train and evaluate a better model of article relevance, it is necessary to go beyond trying to predict what users will browse next. While the implicit feedback from article browsing is valuable for CF, it is limited by the very same information shortage problem that the recommender system attempts to mitigate. In addition to developing the model, we present some post-hoc analysis that sheds light on some of the biases in the CF results that are reversed by applying LtR. Our contributions can be summarised by the following points: 1. Offline evaluation should be matched to the online challenge: By comparing two offline evaluation scenarios with an online experiment, we show that higher accuracy at predicting browsing behaviour does not correspond to having the highest engagement from users.
and the citation network: By training a number of LtR models we show that relevance of a document to a user is a combination of textual similarity, recency, popularity, usage similarity, and proximity in the citation network.
Traditional bibliometric features have limited impact. 3. The ranking model increases diversity: On average, the ranking model increases the number of distinct journals in each list of related items.
more traffic in the past year: Although other collaborative filtering
systems have reported a popularity bias [
A wide variety of techniques have been used to generate recommendations for
academic publications. [
LtR has traditionally been used in information retrieval (IR) systems such as
search engines [
To aid research into LtR, a number of frameworks have been developed which
allow for the test and training of models. Microsoft released LEROT [
The recommendations generation has two distinct phases. First we generate the recommendation candidates using CF and then re-rank them with a LtR model to get the recommendations list. We use different data sets within these two distinct phases. 3.1
The main data set for CF contains implicit feedback from users as they browse ScienceDirect. This usage data set is in the format of <sessionID, articleID, accessTime>. We apply IBCF on this data set to find candidate related articles based on co-usage patterns. High traffic users are removed, as these can represent public access machines in institutions. Additionally, we remove traffic that was elicited by the recommender system, in order to remove a positive feedback loop. 2 https://www.mendeley.com/suggest 3 https://www.citeulike.com 3.2
LtR requires labelled training data that represents user preferences in relation to the recommendation list. We computed relevance labels by aggregating clicks and impressions from the live CF recommender on ScienceDirect. When users visit a ScienceDirect article page, a set of related-article recommendations are displayed (i.e. impressions) and they may click on one or more of the recommendations to view or download the article. We ignored impression data for page loads where none of the recommendations were clicked, or where all of the recommendations were clicked. For each query article, we aggregated the recommended articles across all user sessions. For simplicity we treat relevance as a binary label: 1 if the recommended article was clicked at least once, otherwise 0. 3.3
Each article has a large amount of data and meta-data associated with it. This includes titles, authors, abstract, references, and various metrics on article, journal impact and citation information. These additional data sets are used to generate features for the LtR models. 4
4.1
At the core of IBCF is the method proposed by [
cosine(d; d0) =
jSd \ Sd0 j pjSdj jSd0 j (1)
Using cosine similarity to score neighbours ignores the statistical confidence in the correlation between usage patters. For example, many documents were not viewed in very many sessions, so a similar document d0 may have only one or two sessions in common with the focal document d, but nonetheless rank highly in terms of cosine(d; d0) because jSd0 j is also small. Therefore, we scale cosine similarity with a significance weighting computed from the number of articles in common: score(d; d0) = min(1; jSd \ Sd0 j ) q cosine(d; d0) (2)
If the documents have fewer than q sessions in common, then its contribution to the CF score is scaled down. This means that preference is given to recommendations that are generated from high co-occurrence neighbours, who are more likely related to the focal document d. An alternative would be to discard pairs with low co-occurrence, but to keep catalogue coverage high we chose to keep them and reduce their scores instead. 4.2
Once the set of recommendations (Rd) have been generated for each document (d 2 D) it is re-scored using a pre-trained LtR model. The premise of LtR is to rank items higher which users are more likely to engage with through observing their past action. Training takes the form of n labelled query documents, qi(i = 1; :::; n) and their associated recommended documents with feature vectors with relevance judgements xi = fx(ji)gjm=(i1) , where n is the number of recommendations in the query.
For this work we focus on methods which have been implemented in the
LtR package RankLib [
RankNet [
CF score: A measure of co-usage of the query document and the candidate recommendation (Equation 2).
Popularity: Popularity (number of views) of a document potentially indicates its quality or its future engagement.
Citations: Two documents (the query document and the recommendation) share references then this could indicate a quality recommendation, and likewise if two articles are both cited by the same article. To quantify this, we compute two measures, the first being the Jaccard index between the citation neighbourhoods, cite_sim(Cd; Cd0 ) = jCd \ Cd0 j jCd [ Cd0 j where the neighbourhood Cd is the set of articles that either cite document d or are cited by document d, plus document d itself. The second measure is the total number of citations a document has received.
Journal Metrics: Impact factors are potential predictors of the quality of the research, although a weakness is the huge variation in article impact within a journal. We added to the feature set several impact metrics of the journal where the recommended article was published .
Temporal: We represent age as the number of years since the cover date.
Topics: All articles published are tagged with a scientific taxonomy, indicating which topics and subjects they cover. We calculate binary similarity between the sets of topics associated with each document .
Text: We included the similarity of the recommended document text to the query document text, where text is represented as an n-gram tf-idf vector of a document’s title and abstract. (3) Training We used query-recommendation pairs with relevance labels inferred from recommender logs, as described in Section 3.2. We held out 20% of the query articles from the training data as a validation set, and trained on the remaining 80%. The validation set was used for hyper-parameter tuning and feature selection.
For hyper-parameter tuning, LambdaMART and MART require choosing the learning rate, the number of trees, maximum leaves per tree, and minimum training examples per leaf. With LambdaMART, we used 20 leaves per tree, with at least 200 examples per leaf and a learning rate of 0:1. The number of trees ( 250) was determined by early stopping, again based on the validation set. The final model used was LambdaMART, optimising for NDCG@3.
We pruned features through backwards elimination: removing the least important feature (highest NDCG@3 when removed), and then repeating the process as long as NDCG@3 increased on the validation set. In Section 6.3 we compare the importance of the different types of features. 4.3
The candidate recommendations are ranked using the LtR model or, in the initial
version of the system, by their CF scores. Before selecting the top portion of the
list to display to the user, we apply dithering [
In the recommendation click prediction task, each test case consists of a query article and its recommended articles that were displayed to users. Some of the recommendations were clicked (labelled 1) and some were not clicked (labelled 0). The recommender is evaluated on its ability to rank the clicked items higher than the non-clicked, using NDCG@k. This is the same ranking task that the LtR model was trained on, but the test data came from a time interval after the training and validation data. Its limitation is that it only evaluates performance on the recommendations that were displayed by the incumbent recommender system, so it cannot judge articles that a new ranker will introduce into the top k.
In the session prediction task, each test case consists of a sequence of articles that were browsed in the same user session, excluding browsed articles elicited by the recommender system. The top k recommendations for the first item are compared to what the user actually browsed next. The recommender is evaluated on its ability to rank the browsed items highly (NDCG@k). We have found this evaluation procedure useful for tuning CF candidate selection, where it correctly predicted which CF variant had higher click through rate (CTR).
For both types of evaluation, we split the data set on a time boundary ( ), train the model on data generated before , and then test the model to see if it predicts the actions after . This means before we use the clicks in sessions (browsing between articles) to train the CF model, and then clicks on the already deployed CF recommender is used to train the LtR model. We use actions after as the ground truth, where sessions browsed are used for sessions prediction and click on the deployed recommender for click predictions. Although they were collected in the same time window, the test sets from the two evaluation tasks use mutually exclusive subsets of the logs, e.g. the first only uses recommender clicks, whereas the second excludes recommender clicks.
We calculated a random baseline for each evaluation task. For session prediction, the baseline is a random permutation of the candidate recommendation list for each query article. For recommendation click prediction, the baseline is a random permutation of the displayed recommendations for each query article. 6 6.1
LambdaMART performed best in terms of NDCG@3 on a time-split test set of recommender clicks and impressions (Table 1).
Having chosen a modelling approach for LtR, we tested whether LambdaMART’s performance gains transferred to the session-prediction task. As shown in Figure 1, the ranking using LambdaMART was an improvement over CF score on the click prediction task, but not on the session prediction task. The LambdaMART and CF score rankings outperformed the random baseline in both tasks (Figure 1). Nonetheless, we decided to proceed with online evaluation, because the click prediction task is based on user behaviour in the recommendation context, and we therefore thought it would be a better proxy for online performance. We discuss the discrepancy between the two offline evaluation tasks in more detail in Section 7. 6.2
Our best candidate algorithms from offline evaluation are compared against the current production model via A/B testing. Prior to the work on LtR, we had 0.820 0.815 all features 0.810 3@0.805 G DC0.800 N0.795 0.790 0.785 e r o c s F C y iltr a u p o P t x e T e g A s n o iitt a C c i p o
T Type of feature removed s c itr e m l a n r u o J A/B tested a number of iterative improvements on the CF stage, for example by varying the duration of log data ingested to generate the recommendations. A/B tests performed included increasing the time period of input data for CF candidate selection which resulted in a CTR change of +9:6%, and Re-rank using LambdaMART mode which increase CTR by +9:1%.
For LtR, we carried out an online A/B experiment comparing LtR (using the LambdaMART model in Figure 1) to our best CF algorithm. Each user is randomly allocated to one of two treatments, either receiving recommendations from the incumbent CF system or from the LtR variant. The LtR variant resulted in a statistically significant 9:1% increase in CTR. This result agreed with the offline performance of LtR on the click prediction task, but it disagreed with their performance on the session prediction task (Figure 1). 6.3
Figure 2 shows the effect of removing each category of features, as introduced in Section 3. Using backwards elimination (Section 4.2), we obtained a slight increase in the validation metric (NDCG@3). Using fewer features also reduced training time and simplified the feature extraction pipeline. The journal metrics were all removed, and we retained one or two features from each of the other categories. The evaluation steps described above used this tuned LambdaMART model with a reduced feature set.
As one can see in Figure 2, CF score was the most important feature, in that removing it resulted in the greatest drop in NDCG@3 on the held-out validation set. The feature importance for our ranking model should not be interpreted as what makes an article “relevant” in general, because the training data were collected through a specific user interface that displayed some article metadata and not others, and furthermore the order of the articles was determined by CF score. Instead, the impact of features within the model guided us as to which could be safely discarded (e.g. journal impact metrics). )e 0.2 r sco 0.1 ,eg 0.0 (as 0.1 0.2
Method cf ltr (a) Mean age of items at a(b) Correlation between age(c) Correlation between popgiven rank. and rank ularity and rank Item Age One of the main motivations behind research article discovery tools is to help people stay up to date with recent developments in their fields. Therefore we examined the mean age of recommended documents in each rank position, where age is defined as the current year minus the publication year of the recommend article. As shown in Figure 3a, recommendations ranked by CF scores are biased against newly published articles, with higher-ranking articles tending to be older. When ranked by the LtR model, there is a bias towards younger articles instead.
The bias towards older articles, when ranked by CF, is especially pronounced where the query article is relatively cold, in other words it had less usage data available for collaborative filtering (Figure 3b). Figure 3b bins the query items based on their popularity within the CF input data, then for each query item it computes the Spearman’s rank correlation ( s) between the recommendation age and rank (ordered either by LtR or CF score), and finally takes the mean of s within each bin. A positive value of s indicates that older recommended articles tend to be ranked higher.
However, for recommendations ranked by LtR this effect of bias towards
older articles has been reduced (lower values of s in every bin), resulting in
warm articles having recommendations that favour newer publications.
Popularity In addition to favouring older articles, our CF method also tends
to give higher scores to less popular articles (Figure 3b), where popularity is
measured as the number of unique users viewing the item in the 12 months of
input data for CF (this is assessed using the same method as in Section 6.4).
This is the opposite pattern to the popularity bias often found in UBCF [
Diversity One of the challenges of RS is to highlight content from across a
catalogue. Diversity in recommendations has been shown to increase users’
interactions and perception of fairness [
To quantify diversity within our recommendations we look at the number of recommendations that come from different journals. With journals identified as distinct ISSN numbers.
As one can see in Table 2, LtR caused a slight increase diversity, in terms of the number of different journals in the top three recommendations for each query article. To further confirm the significance of change in diversity we perform a chi-square test ( 2) of the contingency table showed a significant dependency between the ranker and number of distinct journals (p < 10 16). 7
Designing offline evaluation raises the question of what the recommender system is designed to do. The session-prediction task assumes the goal of recommendation is to predict human browsing behaviour. Although it was a good proxy for online performance of the candidate selection step, the session-prediction task did not agree with online results when it came to choosing a ranking method.
One explanation for why CF outperformed the LtR model on the session prediction task is that users’ browsing behaviour outside of the recommender system is at odds with what they actually find useful as recommendations. Users might struggle to find the most relevant articles, which is the very problem of information shortage that motivated us to create the recommender. For example, a recently published article may be less well known within its field, and yet when it is presented in a recommendations list it attracts more attention than older articles.
The post-hoc analysis demonstrated some trends in the types of items recommended by IBCF, which LtR counteracted. Cold start and data sparsity are well-known challenges in collaborative filtering. Even among warm items with usage data, those with fewer users are expected to have less accurate recommendations. We found that the less popular the query article, the more the CF system tended to promote old or unpopular articles as recommendations. Based on which articles users clicked, the LtR system learned to promote articles that were published more recently or had more activity in the past year.
Based on these results, we should be able to improve the model by including as a feature the popularity of the query article (not just the recommended article). That would allow the tree-ensemble model to learn different relevance functions depending on how warm the source article is, for example giving less importance to CF score for items with sparser usage data. In theory, the model could also counteract the unpopularity-bias for colder articles without adding a popularity bias to the warmer articles’ recommendations. 8
In this work we have demonstrated the benefits of LtR for improving upon a production CF system, which is especially important in a domain such as research articles where there are vast catalogues of recommendable items, each of which has high quality structured metadata. We showed how changes to the production system were ultimately decided on the basis of online evaluation, but given the number of models and parameter choices, we also needed an offline evaluation that was a good proxy for online performance.
The winning LambdaMART model combines co-usage, semantic similarity, shared citations, popularity, and recency. The original IBCF solution tended to promote older articles with lower traffic, but by learning from subjective user interactions with the recommender system, our relevance model reversed those trends and therefore overcame some of the limitations of CF.