Analyses and applications based on bibliographic references are of ever increasing importance. However, reference linking methods described in the literature are only able to link around half of the references in papers. To improve the quality of reference linking in large scholarly data sets, we propose a blockingbased reference linking approach that utilizes a rich set of reference fields (title, author, journal, year, etc.) and is independent of a target collection of paper records to be linked to. We evaluate our approach on a corpus of 300,000 references. Relative to the original data, we achieve a 90% increase in papers linked through references, a five-fold increase in bibliographic coupling, and a nine-fold increase in in-text citations covered. The newly established links are of high quality (85% F1). We conclude that our proposed approach demonstrates a way towards better quality scholarly data.
Scholarly data is becoming increasingly important and with it its quality and coverage. Connections between publications in the form of literature references are of particular importance, as they are used as a basis for various analyses, decision making, and applications. Some examples are research output quantification [ 1], trend detection [2], summarization [3], and recommendation [4, 5].
However, reference linking methods1 described in the literature are only able to link around half of the references contained in the original papers to the cited publications [6, 7]. This lack in coverage is especially afecting references to non-English publications [ 8], which are in general underrepresented in scholarly data [9, 10, 11, 12] along with publications in the humanities [13, 14].
We see the reason for this lack in linked references in two key shortcomings of current methods. First, references are linked using simple string similarity measures that are often relying only on publications’ title and author information (which is not always contained in references; see Figure 1). Second, references are exclusively linked to a target collection of CEUR Workshop Proceedings paper records—usually a large metadata set like DBLP2 or OpenAlex3, or a set of IDs like DOIs or PMIDs. This means references to literature which is not contained in the target collection, as well as to non-source items [15], cannot be linked (see “?” markers in Figure 2).
Linking references can be seen as a task of entity resolution (ER) [16], which is concerned with identifying entities referring to the same object within or between large data sets. Because the task requires a one-to-one comparison between each of the involved entities, it is inherently of quadratic complexity. To make approaches scalable, entities are assigned into groups of likely matching candidates prior to comparison, a technique called blocking [17]. While blocking
3See https://openalex.org/. based approaches are used in the domain of scholarly data to, for example, identify duplicate paper records [18, 19, 6] (where information such as abstracts are used) and authors [20], they are not utilized for bibliographic references.
We therefore address both of the aforementioned problems with current reference linking
approaches, (
We showcase the feasibility and benefits of our approach, implementing a pre-processing,
blocking, and matching pipeline and evaluating it on a corpus containing 300,000 references.
We show that relative to the original data, our approach gives us a 90% increase in papers
linked to the target collection, a five-fold increase in bibliographically coupled [21] papers
(see marker “(
In summary, we make the following contributions.
• We propose a blocking-based approach for matching bibliographic references that is independent of a target collection of paper records. • We perform a large-scale evaluation showing that our approach results in a manifold increase in high quality reference links.
• We make our data and code publicly available.5
Blocking-based approaches have been used in the domain of scholarly data, though to the best
of our knowledge not for bibliographic references. We therefore report on (
Simonini et al. [18] develop BLAST (Blocking with Loosely-Aware Schema Techniques) which adapts Locality-Sensitive Hashing. Among data sets from other domains, they also evaluate their approach for the task of linking 2,600 DBLP paper records to the ACM6 and Google Scholar.7 Sefid [ 19] proposes several models to match paper records utilizing the papers’ title, header, and citation information. The models are evaluated in three scenarios matching 1,000 paper records from CiteSeer [22] to IEEE, DBLP, and Web of Science. Lastly, Färber et al. [20] detect duplicates among 243 million author records in the Microsoft Academic Knowledge Graph [ 23] and evaluate their approach using ORCiD IDs.
4With the “coverage” of in-text citation markers we refer to markers associated with linked references, relative to markers belonging to unlinked references.
5See https://github.com/IllDepence/ulite2022. 6See https://dl.acm.org/.
7See https://scholar.google.com/.
Lo et al. [6] introduced the data set S2ORC, which contains 9.6 million open access papers and has recently seen extensive use in area of scholarly document processing. The authors link references to papers within their data set using a heuristic similarity measure based on n-grams and the Jaccard similarity, which only uses the paper title. Using this method, 26 million out of 50 million references (52%) are successfully linked. The authors report that the low number is “due to large numbers of papers (mostly in the field of physics) for which the bibliography entries are formatted without paper titles.” Saier et al. [7] introduce unarXive, a data set created from papers’ LATEX sources containing over 1 million publications. Bibliographic references in the data set are linked to the Microsoft Academic Graph [ 24, 25]. The linking procedure is based on string similarity of papers’ titles and author information. With this procedure 17 million out of 40 million references (42%) are successfully linked. Lastly, CiteSeer [26, 22] in another large data set containing paper records. Similar to S2ORC, references are linked to paper records within the data set itself. In the case of CiteSeer the linking is performed through a heuristic assignment based on title and author information. We are not aware of information on the percentage of references that are successfully linked in CiteSeer .
Our approach consists of the following three steps: (
Pre-processing References as they appear in papers are hard to match for several reasons, such as the variety of citation styles, variants of author names, venue abbreviations, sparsity of information, and typing errors [27] (see Figure 1). To mitigate these issues, we pre-process references in three steps: first, we apply GROBID’s [ 28] reference string parsing module,89 then we expand journal and conference abbreviations, and lastly all strings are lowercased and Unicode normalized. For the abbreviation expansion we use a mapping for 47.6k journal titles provided by JabRef10 and 2.6k conference titles crawled from various web sources. Following [30] we select seven reference fields for the blocking step: title, author, year, volume, journal, booktitle, and pages.
Blocking Following [31], we build our blocking pipeline from components for (
Token blocking is chosen for the block building step because it is schema-agnostic and therefore robust against the varying level of information contained in or missing from bibliographic references. In this step, references are assigned to blocks based on all tokens (i.e., words) contained in the identified and normalized reference fields. As a result, references at this point are associated with multiple blocks, which leads to a high level of redundancy.
Block purging [32] removes oversized blocks based on a comparison cardinality metric, which we determine heuristically and set it to 0.01. Intuitively, the removed blocks originate from common tokens, meaning that matched reference strings within them are highly likely to also share smaller blocks. Purging therefore reduces the number of overall comparisons with minimal efect on the final result quality.
Meta-blocking [33], our comparison cleaning step, reduces unnecessary comparisons within blocks by generating a weighted graph of entities (references in our case) based on their shared blocks, removing edges based on a pruning scheme, and lastly creating a new block collection based on the reduced graph. For both the weighting and the pruning of edges several schemes exist. In Section 4 we describe how we determined the most suitable combination of schemes for our use case. Here, we briefly mention the schemes involved. Available graph weighting schemes include the Common Blocks Scheme (CBS), the Enhanced Common Blocks Scheme (ECBS), the Aggregate Reciprocal Comparisons Scheme (ARCS), and the Jaccard Scheme (JS). For graph pruning, we consider Cardinality Node Pruning (CNP), which relies on cardinality to select the top edges for each node, as well as Weight Edge Pruning (WEP), which removes edges based on their assigned weight.
Matching To determine which references within a block refer to the same publications, we utilize a weighted average of Jaccard similarities across our seven reference fields. Based on [ 34] as well as preliminary experiments, we set the weights for title, author, journal, booktitle, year, volume, and pages to 8, 6, 5, 5, 3, 3, and 2 respectively, and set the threshold for a match to 0.405.
We use a large corpus of scholarly publications to perform two types of evaluations. (
Data For our evaluation we use the data set unarXive [7]. We chose this data set over similar
data sets such as S2ORC [6], because it not only contains paper’s full text with annotated
in-text citation markers, but also a dedicated database of all raw references in plain text. From
unarXive we sample the 300,000 most recent references to conduct our evaluation. The 300,000
references originate from 9,917 papers from the disciplines of physics (
Large-Scale Evaluation Our large-scale evaluation is performed in two steps. First, we determine the most suitable configuration of graph weighting and pruning scheme for our meta-blocking step, then we apply our pipeline to the evaluation corpus and determine the number of additionally linked entities.
To chose a graph weighting and pruning scheme, we use the 13,976 references in our corpus which are are already linked to the target collection as ground truth. Following [33], we select ifve combinations of schemes to evaluate. The combinations are evaluated using the metrics pair completeness (PC), which expresses the ratio of detected matches with respect to all true matches, pair quality (PQ), which estimates the portion of true matches within all executed comparisons in the block collection, and reduction ratio (RR), which measures the number of unnecessary comparisons that are saved through blocking. Table 1 shows the results of our evaluation. We achieve the best results using ECBS weighting and CNP pruning. Accordingly, we apply our pipeline with this configuration on the full evaluation corpus of 300k references, where our approach performs 496,051 comparisons after blocking and identifies 71,826 matches.
As shown earlier in Figure 2, we can use the matches identified by our pipeline to create two
types of new links. First, new links to the target collection, and second, links between references
created through bibliographic coupling. New links to the target collection are established
whenever a reference with no existing link is matched to a reference with an existing link (see
marker “(
To improve the quality of reference linking in large scholarly data sets, we proposed a blockingbased reference linking approach that is independent of a target collection of paper records. In a large-scale evaluation, we first determined the most suitable meta-blocking scheme for our particular application case. Subsequently applying our approach to a corpus of 300,000 references, we saw a manifold increase in linked papers, references, and in-text citation markers. The newly established links are of high precision and have a high recall, which we confirmed through a manual evaluation on a sample of our results. This demonstrates the benefits and quality of our approach.
Key limitations of the work presented are (
11For further details see https://github.com/IllDepence/ulite2022/tree/master/5_manual_evaluation.
In the future we want to address these points by expanding our work through using more advanced blocking methods such as progressive blocking [35, 36], using larger evaluation corpora such as the whole unarXive data set, including data from more diverse disciplines such as the humanities, and evaluating the time performance of our approach. Because references in our evaluation corpus are linked to in-text citation markers, we furthermore plan to explore application scenarios utilizing the paper full texts.
Tarek Saier: Conceptualization, Data curation (support), Formal analysis, Investigation (support), Methodology (support), Software (final evaluation), Visualization, Supervision, Writing – original draft (lead), Writing – review & editing. Meng Luan: Data curation, Formal analysis, Investigation, Methodology, Software, Writing – original draft (support). Michael Färber: Supervision, Writing – review & editing. [11] H. F. Moed, V. Markusova, M. Akoev, Trends in Russian research output indexed in Scopus and Web of Science, Scientometrics 116 (2018) 1153–1180. [12] O. Moskaleva, M. Akoev, Non-English language publications in Citation Indexes - quantity and quality, in: Proceedings 17th International Conference on Scientometrics & Informetrics, volume 1, Edizioni Efesto, Italy, 2019, pp. 35–46. [13] G. Colavizza, M. Romanello, Citation Mining of Humanities Journals: The Progress to
Date and the Challenges Ahead, Journal of European Periodical Studies 4 (2019) 36–53. [14] C. Kellsey, J. E. Knievel, Global English in the humanities? A longitudinal citation study of foreign-language use by humanities scholars, College & Research Libraries 65 (2004) 194–204. [15] P.-S. Chi, Which role do non-source items play in the social sciences? A case study in political science in Germany, Scientometrics 101 (2014) 1195–1213. doi:1 0 . 1 0 0 7 / s 1 1 1 9 2 0 1 4 - 1 4 3 3 - 1 . [16] V. Christophides, V. Efthymiou, K. Stefanidis, Entity Resolution in the Web of Data, Synthesis Lectures on the Semantic Web: Theory and Technology 5 (2015) 1–122. doi:1 0 . 2 2 0 0 / S 0 0 6 5 5 E D 1 V 0 1 Y 2 0 1 5 0 7 W B E 0 1 3 . [17] G. Papadakis, D. Skoutas, E. Thanos, T. Palpanas, Blocking and Filtering Techniques for Entity Resolution: A Survey, ACM Computing Surveys 53 (2020) 31:1–31:42. doi:1 0 . 1 1 4 5 / 3 3 7 7 4 5 5 . [18] G. Simonini, S. Bergamaschi, H. V. Jagadish, Blast: A loosely schema-aware meta-blocking approach for entity resolution, Proc. VLDB Endow. 9 (2016) 1173–1184. doi:1 0 . 1 4 7 7 8 / 2 9 9 4 5 0 9 . 2 9 9 4 5 3 3 . [19] A. Sefid, Record Linkage Between CiteSeerX and Scholarly Big Datasets, Master’s thesis,
The Pennsylvania State University, 2019. [20] M. Färber, L. Ao, The Microsoft Academic Knowledge Graph Enhanced: Author Name Disambiguation, Publication Classification, and Embeddings, Quantitative Science Studies 3 (2022) 51–98. doi:1 0 . 1 1 6 2 / q s s _ a _ 0 0 1 8 3 . [21] K. W. Boyack, R. Klavans, Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately?, Journal of the American Society for Information Science and Technology 61 (2010) 2389–2404. doi:1 0 . 1 0 0 2 / a s i . 2 1 4 1 9 . [22] J. Wu, K. Kim, C. L. Giles, CiteSeerX: 20 Years of Service to Scholarly Big Data, in: Proceedings of the Conference on Artificial Intelligence for Data Discovery and Reuse, AIDR ’19, 2019. doi:1 0 . 1 1 4 5 / 3 3 5 9 1 1 5 . 3 3 5 9 1 1 9 . [23] M. Färber, The Microsoft Academic Knowledge Graph: A Linked Data Source with 8 Billion Triples of Scholarly Data, in: Proceedings of the 18th International Semantic Web Conference, ISWC’19, 2019, pp. 113–129. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 0 3 0 - 3 0 7 9 6 - 7 _ 8 . [24] A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-J. P. Hsu, K. Wang, An Overview of Microsoft Academic Service (MAS) and Applications, in: Proceedings of the 24th International Conference on World Wide Web, WWW ’15 Companion, ACM, 2015, pp. 243–246. doi:1 0 . 1 1 4 5 / 2 7 4 0 9 0 8 . 2 7 4 2 8 3 9 . [25] K. Wang, Z. Shen, C. Huang, C.-H. Wu, D. Eide, Y. Dong, J. Qian, A. Kanakia, A. Chen, R. Rogahn, A Review of Microsoft Academic Services for Science of Science Studies, Frontiers in Big Data 2 (2019) 45. doi:1 0 . 3 3 8 9 / f d a t a . 2 0 1 9 . 0 0 0 4 5 .