In this paper, we build on the idea of Citation Proximity Analysis (CPA), originally introduced in [1], by developing a step by step scalable approach for building CPA-based recommender systems. As part of this approach, we introduce three new proximity functions, extending the basic assumption of co-citation analysis (stating that the more often two articles are co-cited in a document, the more likely they are related) to take the distance between the co-cited documents into account. Asking the question of whether CPA can outperform co-citation analysis in recommender systems, we have built a CPA based recommender system from a corpus of 368,385 full-texts articles and conducted a user survey to perform an initial evaluation. Two of our three proximity functions used within CPA outperform co-citations on our evaluation dataset.
The number of scholarly articles is increasing exponentially every year, according
to Gipp and Beel [
West et al. [
In the past, many academic recommender systems unless developed by
publishers or corporations that negotiated access to scienti c literature, faced
several limitations. For instance, limitations of machine access to the full texts of
papers and sometimes even the citation information. Consequently, full-text
features have been so far relatively unexplored. Most of the recommender systems
Recommender systems suggest relevant and useful information to match the need
of its users. These systems are popular in both commerce and academia. Over the
years, various metrics and approaches such as Collaborative Filtering (CF) [
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. In hac habitas e platea dictumst. In leo ante, venenatis eu, volutpat ut, imperdiet auctor, enim. Aenean scelerisque metus eget sem. Nul a sed lacus. Mauris tempus diam. Curabitur accumsan felis in erat. Integer porta. Praesent scelerisque. Quisque pretium rutrum ligula. Praesent scelerisque. Duis sem velit, ultrices et, fermentum auctor, rhoncus ut, ligula. Curabitur risus urna, placerat et, luctus pulvinar, auctor vel, orci. Praesent semper, neque vel condimentum hendrerit, lectus elit pretium ligula, nec consequat nisl velit at dui. Nam mas a turpis, nonummy et, consectetuer id, placerat ac, ante. Sed at turpis vitae velit euismod aliquet. Maecenas justo. Donec ut urna. Aenean luctus vulputate turpis.
Pel entesque condimentum felis a sem. Etiam pharetra lacus sed velit imperdiet bibendum. Cras ac enim vel dui vestibulum suscipit. Aenean turpis ipsum, rhoncus vitae, posuere vitae, euismod sed, ligula. Cras gravida. Sed elementum, felis quis port itor sol icitudin, augue
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum interdum
a augue accumsan suscipit.
Phasel us id erat neque. Aliquam eu pretium enim.
Maecenas semper pel entesque
mi ac rhoncus. Donec
[
mat is elit. Fusce faucibus mauris ut arcu consequat ultrices.
Donec eu lectus ultrices justo
luctus placerat. Phasel us scelerisque
sapien non blandit lobortis ipsum
Lorem
ipsum
dolor
sit
[
Document C Document A Document B .
.
Analysis (CPA). Length
of
solid
underlined Red text
signi
[
erent citation based concepts have b een implemented and evaluated for scholarly recommendation,
either using full-texts or with meta-data. While is the state-of-the-art approach for recommending items in commerce, it is CF also prone to some limitations such as the cold start problem, i.e. the need to have go o d coverage of ratings. In the domain of scholarly pap ers, it is typically di cult to obtain ratings. Agarwal et
al. [
They also showed that \Conclusion" section weight more than other section for computing e ectiveness of the pap er.
al. [
Another p opular approach is
citation based approach, such as, Co-Citation
which was prop osed by Small [
This approach is a well-established in scholarly recommendation system by now.
West et
al. [
al. [
Co-Citation than Paper Level Co-Citation. Gipp et al. [
Researchers have tried and tested citation-based approaches which are proving in some situations, such as for expert users, more powerful than purely text based approaches. Consequently, it is worthwhile to further explore the role of citation proximity in recommender systems based on co-citations. 3
Method The design of our CPA-based recommender system consists of ve modules (Extracting Citation Information, Citation Information Normalisation, Sparse Matrix, Citation Proximity Analysis, Recommendation) depicted in Figure 2. We start by extracting citation information, including the positions of citation anchors in the body of the full-texts. We then normalise the extracted reference strings (typically found at the end of each paper) trying to detect and merge those referring to the same canonical document. The output of this process is a sparse square matrix where rows and columns correspond to unique references found in the full-texts of research papers in the original collection. These unique references form the set of recommendable items. Each cell of the matrix contains all the co-occurrences of the corresponding references in any of the research papers in the original collection, including their character positions. The information stored in each cell is passed to the CPA component which applies a proximity function to produce a CPA value.
To produce recommendations for a given paper reference, it is necessary to look up a corresponding row (or column), calculate CPA values for each nonempty cell and select n references with the highest CPA scores as the recommendations. Next sections describe the process in more detail. 3.1
Extracting citation information
The aim of this component is to:
{ extract reference strings, typically appearing at the end of research papers,
{ identify and parse the reference structure, such as article title, authors,
publication year or DOI, of each reference and
{ detect and extract the character o set of each citation occurrence (citance)
on the body of the research paper.
Vestibulum ante ipsum primis in fauc
us orci luctus et ultrices posuere a cu
cubil a [
Mo del dia gram of the pro cess carried out for recommending research articles The input of this comp onent is the full text of a research pap er and the output is
a tuple: (referenceId, title, authors [], characterO sets [], yearPublished, sourceId) (1) er is of an each array the is where referenceId is an
identi of the referenced article, authors reference strin g, title is the
title of author name sets is an array of o sets of citances in the b o dy of given reference, yearPublished the year of publication unique identi er of the full-text research pap er from which strings, the full chara
ctext and this sourceID reference Citation information
normalisation citation normalisation comp onent takes as an input, a dataset of the tuples sparse de
ned matrix.
in As the
Equation 1,
deduplicates deduplication is not the key them fo cus naive deduplication metho d that targets precision at and
represents of the this pap er, we
use exp ense of recall.
Firstly, uments we removed the sp ecial characters and spaces from the title of each and group ed records with the same title, publication date and at do cleast one matching author and singled out only one record from the group. By doing this, we ended up only unique documents in the dataset.
After normalisation, we represent the output as a Citation-Positions matrix. Each cell Vi;j in this matrix contains character o set information of all the citances where a normalised reference i co-occurs with a normalised reference j in a given source document. This is all that is needed as input for the CPA module which then calculates the CPA value based on a given proximity function. 3.3
Proximity functions The CPA proximity function takes as an input, a set of character o set distances and produces a single value. The higher the value the higher the relevance. In a research paper, a reference can be cited multiple times leading to a set of pair distances for the co-cited pair. Additionally, references can be co-cited in multiple source documents. The intuition behind the proximity metric is that higher number of co-citations as well as closer proximity should lead to an increased relevance.
In this work, we have de ned and experimented with the following proximity measures: MinProx, SumProx and MeanProx described below. Our baseline cocitation method, which does not use any proximity information, can be in this framework de ned as: cocitbaabseline = jDocja2Doc^b2Doc; (2)
This means that the number of co-citations is de ned by the number of source documents where references a and b co-occur. 3.3.1 MinProx: Uses only the distance of the closest co-occurrence in the denominator. During distance computation, one of the hypothesis is the distance between co-cited documents are never Zero or One. For example, the extreme case of citations being cited together will be like [X; Y ] and this will always have a separator character between them. If reference \X" has character o set 102 and reference \Y" has character o set 104 then the distance will be 104-102=2.
As, in the following proximity functions (3, 4, 5), logarithm is applied for smoothing large distances. The nominator is equal to the baseline co-citation measure.
proxaMbin =
jDocja2Doc^b2Doc log(minfd1ab; :::; danbg) where dab denotes the rst distance between the co-cited references a and b and 1 danb denotes the last distance between them. 3.3.2 SumProx: Uses the sum of the logs of all the co-cited distances in the denominator.
(3) proxaSbum = jDocja2Doc^b2Doc
Pn i=1 log(diab) where di is the ith distance between the co-cited documents a and b 3.3.3 MeanProx: Uses the log of the mean of all the co-cited distances in the denominator.
proxaMbean =
jDocja2Doc^b2Doc log(meanfd1ab; :::; danbg) (4) (5) where dn is the last distance between the co-cited documents a and b. 4
Experiments
To evaluate CPA against the co-citations baseline, we have developed and \trained"
a recommender system, as described in Section 3, on a sample collection
scienti c documents from CORE [
Experimental system
We used GeneRation Of BIbliographic Data (GROBID) [
As expected, GROBID could not successfully process all the documents (due to PDFs that were scans, badly encoded PDF les or citation extraction failing on valid PDFs). We were con dent of 368,385 documents which yielded citations information along with their positions and used these in our experiment. The resulting set consisted of 6,609,147 references. This means we have obtained on average 18 references per document. Figure 3(a) shows the probability distribution of the number of citation mentions (citances) in a document while Figure 3(b) shows the probability distribution of the number of references in a document.
For an e cient implementation of the subsequent components, i.e. normalisation and CPA calculation, we made use of the MapReduce paradigm and implemented the solution using the data ow language Apache Pig. The normalization step took approximately 2 minutes with 100 parallel processes and 1 https://core.ac.uk/services#datasets 0.030 t n e cum0.025 o d r e p se0.020 ffrcenR0.015 e e o n o it u ib0.010 itrsd y lit ib0.005 a b o r
P0.0000 20 # of extracted citation mentions 80 40 60 100 20 4#0of References 60 80 100 resulted in a citation-positions sparse matrix containing 142,157,561 co-cited pairs. Figure 4 shows the distribution plots of relevance score produced by each proximity functions, we have de ned along the baseline standard's results. 4.2
Evaluation dataset
Evaluating recommender system is generally a challenging task due to the variety
of criteria and goals recommendation methods can be developed with, such as
recency, serendipity, relevance, etc. Consequently, some evaluation datasets can
work better with one metric or task while other's may not [
For the evaluation, we randomly selected 6 sample documents from the dataset from the area of \Computer Science" (specially \Data mining" and \Information Retrieval") with which the annotators were familiar with. Ten annotators (survey participants) from computer science department (working on \Data mining" and \Information Retrieval") were asked to provide binary relevance judgments on each recommendation o ered by each evaluated system. As, we had 4 evaluated metrics, 5 recommendations for each sample document and ten participants, this yields 6 5 4 10 = 1; 200 individual relevance judgments. 4.3
Results We have calculated precision at 3 di erent precision levels as shown in Table 1. Our experimental results indicate that proximity information helps in producing (a) proximities using minimum co-(b) proximities using summation of cited Distance metric co-cited Distances metric (c) proximities using mean of co-(d) proximities using frequency of cited Distances metric co-citation metric better recommendations than the baseline co-citation approach. More speci cally, out of the three proximity functions two, SumP rox and M eanP rox, outperform the Baseline. The improvement over the tested dataset over the baseline for P@5 (from 0:27 to 0:34) corresponds to a more than 25% improvement.
To assess the subjectivity of the task, we have also calculated inter-rater reliability statistic to weight the agreement between the contributors. To do so, we have used Fleiss's as follows: =
P 1
Pe
Pe where, Pe denotes the observed agreement and P denotes the probability of chance agreement. Hence, (1 P ) is the degree of agreement which is obtainable by chance and P Pe gives the degree of agreement which is actually obtained. For all the sample data and its recommendations, we have observed = 0:25 suggesting a fair agreement. (6)
Discussion and future work
There are several things we would like to address in our future work as this is an
initial practical observation of the CPA concept. Firstly, our current proximity
functions are based on working with absolute character o sets, i.e. our distance
measure is a character distance of the citances. In the future, we would like
to extend our work by experimenting with distance measures that re ect the
lexico-syntactic structure of language. For example, we could use information on
whether two papers have been co-cited in the same sentence clause, sentence,
paragraph, section, table, etc. Additionally, we would also like to compute the
CPI values as conceptualised by Beel and Gipp in [
Secondly, our current method treats all co-citations equal. However, it would be interesting to explore how the concept of \authority" might be applied in this problem. For example, if one paper is co-cited (or even compared/contrasted) with another highly signi cant work (e.g. famous authors, policy document or a high value determined by any particular scientometric method) or if the cocitation is present in a highly signi cant work, this information should in uence the strength of this co-citation evidence and be e ectively used in the recommender.
Thirdly, more work is needed to understand the impact of document length on co-citation analysis approaches as long documents, like theses or books, produce signi cantly higher numbers of co-citations than shorter documents (short, long papers, demo), hence they have a higher impact on the results of the recommender.
Finally, we would also like to use machine learning algorithms using position of the citation as one of the features to improve the weighing process. However, to do so, we will be needing big dataset with the ground truth of recommendation results for training the system so collecting such dataset will be one of the major hurdles. 6
Conclusion In this paper, we performed an experiment to convert the concept of CPA into practice and benchmark this approach against co-citation Analysis. We introduced three di erent proximity functions used within the CPA method and developed a highly scalable CPA implementation that runs on a cluster using the MapReduce paradigm. Our initial results suggest that CPA can provide better performance in recommender systems than the co-citation method. More speci cally, two of our proximity functions outperformed the baseline co-citation approach on our dataset, the SumProx function by a margin of more than 25% for precision@5. However, a larger evaluation dataset is needed to con rm these results.