Learning-to-Rank in research paper CBF recommendation: Leveraging irrelevant papers

Abstract

Suggesting relevant literature to researchers has become an active area of study, typically relying on content-based ltering (CBF) over the rich textual features available. Given the high dimensionality and the sparsity of the training samples inherent to this domain, the focus has so far been on heuristic-based methods. In this paper, we argue for the model-based approach and propose a learning-to-rank method that leverages publicly available publications' metadata to produce an e ective prediction model. The proposed method is systematically evaluated on a scholarly paper recommendation dataset and compared against state-of-the-art model-based approaches as well as current, domain-speci c heuristic methods. The results show that our approach clearly outperforms state-of-the-art research paper recommendations utilizing only publicly available meta-data.

Keywords

• Research paper recommendation
• Learning-to-Rank
• Contentbased Recommendation
• Model-based user pro le

1. INTRODUCTION

Scholars and researchers are confronted with an overwhelming number of newly published research papers in their domain of expertise. Although advantageous in restricting the domain, keyword-based search tools typically available in digital libraries o er a limited help to researchers in locating the relevant content. As a result, researchers need to manually search within unspeci c search results to identify paper(s) of interest. This is the situation where recommendaer systems have great potential, and indeed plenty of works adopted di erent techniques to tackle this problem. A recent extensive survey in this domain [ 3 ] identi ed content-based ltering (CBF) as the predominant approach for research paper recommendation because of the rich textual features available. For learning user pro le, almost exclusively the focus was on relevance feedback approaches, building on the assumption that papers appearing in user's preference list have an equal (or a presumed extent) share in the underlying user taste. Thus, user pro les are constructed as aggregation of relevant papers' keywords. Based on the classi cation suggested by Adomavicius et al. in [ 1 ], these approaches are referred to as heuristic-based. In contrast, model-based approaches depend on a learning method to t the underlying user model (pro le). This enables constructing a better modeling of researcher-keywords relation in user pro les. But they require a large body of training data which is not intuitively available in this domain. As a result, little work on applying model-based approaches exists for this problem.

In this paper, we employ pairwise learning-to-rank [ 4 ] as a model-based technique for learning user pro le. We incorporate both relevant and irrelevant \peer" papers -papers published in relevant papers' conferences- to formulate pairwise preferences and enrich the training set. Our main contributions include:

We investigate and customize learning-to-rank for CBF research paper recommendation.

We incorporate only a small set of data, restricted to publicly available metadata of papers. This makes our approach suitable for a much larger domain than previous approaches which require papers' full-text. We perform an initial, yet systematic study on a realworld datatset in which we show that our approach clearly outperforms existing heuristic- and model-based algorithms.

The rest of this paper is organized as following: the second section provides an overview of existing related work. In section 3 we present our approach and in section 4 we demonstrate experimental setup and results. Finally, we conclude in section 5 by summarizing our ndings and situate this work within our future plan. 2.

RELATED WORK

A rich amount of related work tackled the problem of research paper recommendation. collaborative ltering (CF) approaches [ 8, 13, 14 ] showed a successful application of model-based methods incorporating knowledge from other \similar" users. However, we restrict our search to contentbased scenarios considering only information from the active user. In this domain, the main focus in learning user pro le has been on heuristic-based approaches with a wide adoption of relevance feedback and cosine similarity [ 3 ]. Papers are recommended which are most similar to one or more of previously published or liked papers. In [ 10 ], De Nart et al. used extracted terms (keyphrases) from user's liked papers in constructing user pro le. The pro le has a graph representation, and the focus here was on the keyphrases extraction method and the graph structure. The approach of Lee et al. [ 6 ] proposed a memory based CBF, where users' papers are clustered based on their similarity, and candidate papers are ranked based on the distance from user's clusters. Sugiyama et al. in [ 11, 12 ] applied a relevance feedback approach utilizing all terms from the fulltext of the researcher's publications in addition to terms from the citing and the referenced papers in order to build pro les. All of these works are heuristic-based, where weights in user pro le are set by aggregating individual keywords' scores of relevant papers. On the contrary, model-based approaches depend on machine learning techniques to learn user a nity towards keywords, promising a more representative user pro le. In a previous work [ 2 ], we showed the superiority of a modelbased method over relevance feedback methods for CBF research paper recommendations. We applied multivariate linear regression to learn researchers' pro les from their previous publications. Yet, the work was tailored to researchers with previous publications and didn't consider irrelevant papers. In [ 9 ], Minkov et al. presented a collaborative ranking approach for events recommendation. They compared it with a content-based baseline that applies pairwise learningto-rank on pairs of relevant and irrelevant events. In our work, we follow similar approach in applying learning-torank on pairs of relevant an irrelevant papers. However, we push it further and investigate the quality of these pairs and their e ect on the model performance.

3. PROPOSED APPROACH

This work targets users who have previously interacted with scienti c papers and identi ed some as papers of interest (relevant papers). Having a set of relevant papers for a user, the recommendation process can start and a machine learning method is applied to t a user pro le (model). The learned model is used to rank a set of candidate papers and recommend the top ranked papers to the user. Our approach is to employ the pairwise learning-to-rank technique in building the user pro le. We chose this method because of its desirable properties: It was proven to be successful in solving ranking tasks in similar problem domains like online advertising [ 7 ]. It also shows a good performance on problems with sparse data. The main idea of pairwise learningto-rank is to build pairs of preferences out of the training set. Each pair consists of a positive and a negative instance. Afterwards, the pairs are fed as training instances to a learning algorithm, which in turn learns the desirable model. In the underlying problem, papers marked as interesting by users are the positive instances. However, the negative instances or the irrelevant papers are usually not explicitly provided by the users. This makes pairwise learning-to-rank not directly applicable on this setup. In our contribution, we seek implicit information about the irrelevant papers. For this, we start from the following hypothesis: when users identify relevant papers, they, to some extent, implicitly rate other papers published at the same conference (we call them peer papers) as irrelevant1. Based on this hypothesis, we utilize peer papers as irrelevant papers as follows: for each user, we build pairs of preferences out of relevant and peer papers. Such pairs are called pairwise preferences or for simplicity pairs, we will use these terms interchangeably along the paper. Afterward, we feed these pairs as training examples to a learning algorithm in order to t the user's model. This model is used later to rank candidate papers and recommend top ranked ones to the user. Before delving deeper in the method details, we rst introduce some notation. The function peer(:) is de ned over the interest set Pirnt of a user r. It delivers for a paper p 2 Pirnt the set of p's peer papers. In practice, this can be retrieved via digital libraries like DBLP registry2. For the paper modeling, we adopt a vector space model representation. Having the domain related keywords extracted from paper's title, abstract and keyword list as features, each paper p is a vector: p = hsp;v1 ; :::; sp;vjV j i, with vi 2 V is a domain-related vocabulary and sp;vi is a score re ecting the importance of vi in p. We adopt the TF-IDF score as the weighting scheme. Based on this representation, the similarity between two papers is calculated by the cosine similarity between the papers' vectors. 3.1

Method Steps

An overview of the proposed approach is depicted in Figure 1. For the experimental setup only, we split user's r interest set Pirnt into training and test sets Ptrrain; Ptrest respectively. However, this step is dropped out in the nonexperimental recommendation scenario and the rst step receives, in this case, the complete interest set Pirnt.

User’s 

Interest list 1 ) % 0 8  ( g n ii n a r t ) % 0 2  (t s e T

Peer papers  augmenting 1

Peer papers  augmenting 2

Forming pairwise  preferences 5

Ranking & 

Evaluation 4

Model learning 3

Preferences  validation

User’s Model  (profile)

1. Peer papers augmenting: in this step, the peer papers are retrieved for all relevant papers. Retrieved peer papers serve as potential negative classes and are important for empowering the learning algorithm to construct a better understanding of user's taste. 2. Forming pairwise preferences: here we apply the concept of pairwise learning from learning-to-rank. The training set in this step is reformulated as a set of pairs P, where each pair consists of two components: a relevant paper and an irrelevant paper. That is, each relevant paper p 2 Ptrrain is paired with all papers from peer(p):

P = f(p; p0)j8p 2 Ptrrain ^ 8p0 2 peer(p)g 1Later, we introduce a validation process that checks the correctness of this hypothesis for each pair. 2http://dblp.uni-trier.de A pair (p; p0) 2 P depicts a preference in user's taste and implies that p has a higher relevance to user r than p0. 3. Preferences validation: In the rst step, we introduced the peer papers as negative classes based on the hypothesis mentioned earlier in this section. Yet, this can't be adopted as a ground truth due to: (a) it is not explicitly a rmed by users that they are not interested in peer papers; and (b) some peer papers might be of interest to the user but might have been overlooked. Having this in mind, not all pairwise preferences formulated in the previous step have the same level of correctness. Therefore, this step examines pairwise preferences and makes sure to pass valid ones to model learning. We propose two di erent mechanisms to accomplish this validation: pruning based validation and weighting based validation. We explain these techniques in the next section. 4. Model learning: In this step, we apply a pairwise learningto-rank method (Ranking SVM [ 5 ]) to train a user model w^r. Using validated pairwise preference from the previous step, we seek w^r that minimizes the objective function:

1 w^r = arg max 2 jjwrjj2 + C:L(wr)

wr With C 0 is a penalty parameter and L(wr) is the pairwise hinge loss function:

L(wr) =

X (p;p0)2P max(0; 1 wrT (p p0))2 (*) 5. Ranking & Evaluation: Given the user's model as a result of the previous step, here we apply the prediction on candidate papers. For the experimental setup, this is the the test set, which is constructed out of relevant papers Ptrest (the positive instances), in addition to their peer papers as irrelevant papers (the negative instances). 3.2

Preferences Validation Methods

As pairwise learning-to-rank expects pairs that show contrast between negative and positive classes, pairs with \wrongly assigned" peers pose a potential noise to the learning process. After all, the validity of a pairwise preference (p; p0) depends on the correctness of considering its peer paper p0 irrelevant. The pair's relevant paper p forms the ground truth and hence, it can be considered as the reference point to decide whether p0 is irrelevant or not. For each pair (p; p0) 2 P we measure the similarity between p and p0, and adopt two methods to validate the pair based on this similarity: Weighting Based Validation (WBV). This strategy is based on giving pairwise preferences di erent weights based on the dissimilarity between the pairs components. This boosts the importance for pairs with dissimilar components and assures that the more similar the pair's components are, the less important the pair for model learning is. Therefore, we weight the importance of each pair according to the distance (1-similarity) between the relevant paper and the peer paper. Then, we rede ne the loss function from (*) to consider pairs' weights as following:

max(0; 1 wrT (1 similarity(p; p0))(p p0))2 L(wr) =

X (p;p0)2P Pruning Based Validation (PBV). Here we lter out invalid pairwise preferences. Validity is judged based on the dissimilarity between the pair's components. If they prove to be similar, then we don't consider p0 as an irrelevant paper and consequently, the pair (p; p0) is not eligible for model learning. A similarity threshold is applied and a pair (p; p0) is pruned if similarity(p; p0) > . In our experiments, we empirically test a range of values for and discuss the corresponding e ect on the model. 4. 4.1

EXPERIMENTS

Dataset & Setup

We evaluated the proposed approach on the Scholarly publication recommendation dataset from [ 12 ], including the extensions applied in our previous work [ 2 ]: Papers are identi ed and enriched with meta-data from the DBLP register, namely titles, abstracts, keywords and the publishing conference. The dataset contains 69,762 candidate papers, as well as the lists of relevant papers for 48 researchers. The number of relevant papers ranges from 8 to 208 with an average of 71 papers. After augmenting peer papers, we got a skewed distribution as the ratio of relevant papers to peer paper ranges from 0.45% to 3% with an average of 1.2%. We performed o ine experiments with 5-folds cross validation following the steps outlined in Figure 1. For each researcher we randomly split the interest list into training and test sets; then, we learn researchers' models as described in section 3; nally, we evaluate the learned models on the test set. The test set consists of: (a) positive instances, the test relevant papers (20% of the researchers interest list) and (b) negative instances, the peer papers of the positive instances. This applies for all of our experiments, except for experiments on the pruning based validation method (PBV). In PBV, we lter out those pairs which components have a similarity higher than from the training set. Therefore, we apply the same rule on the test set and we lter out peer papers based on their similarity to the corresponding relevant paper. For example, given a similarity threshold and a relevant paper p from the test set, a peer paper p0 2 peer(p) is added as an irrelevant paper to the test set if and only if similarity(p; p0) . 4.2

Metrics

We measured the following metrics to determine the performance for top k ranking and also overall classi cation. We show the averages over all researchers for each metric: Mean Reciprocal Rank (MRR): evaluates the position of the rst relevant paper in the ranked result.

Normalized Discounted Cumulative Gain (nDCG): nDCG@k indicates how good the top k results of the ranked list are. We look at nDCG for k 2 f5; 10g AUC and Recall: used to study the behavior of validation strategies PBV, WBV and the baseline algorithms: Logistic Regression and SVM. 4.3

Results & Discussion

In total, we performed three di erent experiments. The rst experiment (with the results shown in Table 1) shows a superior performance for our weighting based validation method (WBV) over the state-of-the-art heuristic-based work (Sugiyama [ 12 ]) and model-based (PubRec [ 2 ]) approach. The experiments were performed using the same features and datasets present in these works and show a clear lead over all metrics.

MRR WBV 0.728 PubRec 0.717 Sugiyama[ 12 ] via [ 2 ] 0.577

The second experiment compares the performance of our approach over other, baseline classi cation algorithms like SVM and logistic regression to provide a more general understanding of its capabilities. As shown in Figure 2, logistic regression showed a weak performance on all metrics, particularly on Recall. It didn't succeed in identifying relevant papers even when it is fed with a balanced training set. However, SVM showed a better ability to recognize the relevant papers with a better recall value, but produced a lot of false positives and this is clear from its lower MRR and nDCG values. In contrast, all variants of our method showed a superior performance in all metrics. Finally, we compare between the suggested pair validation techniques WBV and PBV, including tuning the latter by varying the similarity threshold from 1 (where no pairs are ltered, this case represents the CBF approach of [ 9 ]), down to 4 10 4 (where a lot of \noisy" pairs are pruned from the training set). WBV showed in general a very good performance, beating PBV for higher values of on all metrics except recall. There, PBV gives a slightly better recall even without ltering any pairs (when = 1). This refers to the fact that weighting the pairs in WBV causes the model to miss some relevant papers, while PBV made models more capable of recognizing the relevant papers by eliminating the noisy pairs from the training set. When decreasing , PBV shows very good scores, but these results need additional investigation before leading to a clear conclusion. As mentioned earlier in this section, reducing also leads to a smaller number of irrelevant papers in the test set. This reduces the underlying bias in the test set which has an (additional) positive impact on the metrics, even though there is still a clear bias (the relevant/peer ratio is on average 11.2%) present at the lowest values.

CONCLUSION

In this paper, we investigated the application of learningto-rank in research paper recommendation. We proposed a novel approach that leverages irrelevant papers to produce more accurate user models. O ine experiments showed that our method outperforms state-of-the-art CBF research paper recommendations utilizing only publicly available metadata. Our future steps will focus on further understanding the e ect of the similarity threshold in pruning based validation (PBV) on the model quality and study the suitability of pairwise learning-to-rank algorithms other than Ranking SVM for this problem. 50:8 @0:7 G0:6 C :5

0 nD0:4 0:3 1 :9 ll 00:78 ca0::6 e 0:5

04 R : 00::32 0 00:7 0.00050.0010.005 0.010.05 0.1 0.5 1 0.00050.0010.005 0.010.05 0.1 0.5 1

PBV SVM

LR WBV 0.00050.0010.005 0.010.05 0.1 0.5 1

References

[1] G. Adomavicius , Z. Huang , and A. Tuzhilin . Personalization and Recommender Systems . 2014 .

[2] A. Alzoghbi , V. A. A. Ayala , P. M. Fischer , and G. Lausen . Pubrec: Recommending publications based on publicly available meta-data . In LWA , 2015 .

[3] J. Beel , B. Gipp , S. Langer , and C. Breitinger . Research-paper recommender systems: a literature survey . IJDL , 2015 .

[4] L. Hang . A short introduction to learning to rank . IEICE TRANSACTIONS on Information and Systems , 2011 .

[5] R. Herbrich , T. Graepel , and K. Obermayer . Large Margin Rank Boundaries for Ordinal Regression . 2000 .

[6] J. Lee , K. Lee , J. G. Kim , and S. Kim . Personalized academic paper recommendation system . In SRS , 2015 .

[7] C. Li , Y. Lu , Q. Mei , D. Wang , and S. Pandey . Click-through prediction for advertising in twitter timeline . In KDD , 2015 .

[8] S. M. McNee and et al. On the recommending of citations for research papers . In CSCW , 2002 .

[9] E. Minkov , B. Charrow , J. Ledlie , S. Teller , and T. Jaakkola . Collaborative future event recommendation . CIKM , 2010 .

[10] D. D. Nart and C. Tasso . A personalized concept-driven recommender system for scienti c libraries . Procedia Computer Science , 2014 .

[11] K. Sugiyama and M.-Y. Kan . Scholarly paper recommendation via user's recent research interests . In JCDL , 2010 .

[12] K. Sugiyama and M.-Y. Kan . Exploiting potential citation papers in scholarly paper recommendation . In JCDL , 2013 .

[13] A. Vellino . A comparison between usage-based and citation-based methods for recommending scholarly research articles . In ASIS&T , 2010 .

[14] C. Wang and D. M. Blei . Collaborative topic modeling for recommending scienti c articles . In KDD , 2011 .