In recent years, a large number of Scholarly Knowledge Graphs (SKGs) have been introduced in the literature. The communities behind these graphs strive to gather, clean, and integrate scholarly metadata from various sources to produce clean and easy-to-process knowledge graphs. In this context, a very important task of the respective cleaning and integration workflows is deduplication. In this paper, we briefly describe and evaluate the accuracy of the deduplication algorithm used for the OpenAIRE Research Graph. Our experiments show that the algorithm has an adequate performance producing a small number of false positives and an even smaller number of false negatives.
In recent years, large amounts of scholarly data have become openly available due to the
increased popularity of the Open Science [
Several SKGs have been produced in recent years either from the academic community (e.g.,
the OpenAIRE Research Graph [
In this work, we conduct a preliminary evaluation of the efectiveness of the deduplication
process currently used for the creation and update of the OpenAIRE Research Graph [
One of the most popular approaches for scientific knowledge representation is that of
Scientific/Scholarly Knowledge Graphs (SKGs), many of which have been developed as
industrydriven initiatives, such as the Web of Science (WoS) [
The OpenAIRE infrastructure1 is an initiative and Legal Entity whose purpose is to facilitate,
foster and support Open Science in Europe. Among others, OpenAIRE supports the technical
services that facilitate and monitor Open Science publishing trends. To this end, the OpenAIRE
service infrastructure consists of metadata aggregation services and information inference
services whose purpose is to populate the OpenAIRE Research Graph [
The graph’s data model is depicted in Figure 1 and its main entities are described below: • Research Products represent the outcomes of research activities. • Organizations correspond to companies or research institutions involved in projects, responsible for operating data sources or consisting the afiliations of Product creators. • Funders (e.g. EC, Wellcome Trust) are agencies responsible for a list of Funding Streams. • Funding Streams represent investments (funding actions) from Funders (e.g. FP7 or H2020). • Projects are research projects funded by a Funding Stream of a Funder.
• Data Sources are the resources used to collect metadata for the graph objects.
On top of the graph, OpenAIRE ofers various services, such as a search and exploration portal
and a number of dashboards (the Research Community Dashboard [
The entire deduplication process used to materialize the final version of the OpenAIRE Research
Graph is managed by the gDup framework [
• Duplicates grouping: the final operation that creates representative objects and persistent identifiers for the newly created records.
Of course, the similarity function used to compare pairs of entities should be able to capture record equivalence and should be flexible and configurable for every diferent entity type. In gDup, the similarity function was defined by a weighted sum of the similarity scores between entity attributes, while a set of conditions that implement early exits in the comparison have been defined. To make this task smarter, gDup was extended to a new framework, called fDup, which introduces a decision tree mechanism which enables early exits and diferent similarity match strategies based on intermediate results of the comparisons between entity attributes. The mechanism considers the Levenshtein distance (normalized to obtain a value between 0 very diferent - and 1 - identical) of the entity titles to determine if two entities are equivalent or not. A threshold depending on the number of common IDs is applied to the similarity score: in case the entities have common IDs, the threshold on the score is lower than the other case (0.9 vs. 0.99). This means that a higher similarity score of the title is needed if two entities do not share IDs. In the last case, a further comparison on the title version (i.e. numbers in the title string) and the author lists is performed to guarantee the correct result. If the entities have diferent versions in the title and diferent sizes of author lists, the early exit tells that there is no need to compute the Levenshtein distance as the two entities are considered to be diferent. This specific deduplication configuration is currently used as the OpenAIRE’s deduplication algorithm and it will be referred in the following as fDup-2021.
In this section we elaborate on our assessment process to evaluate the accuracy of the fDup-2021 algorithm in identifying DOIs that correspond to equivalent objects (i.e., closely related entities). Our experiments can be divided into two groups: first we compare fDup-2021’s output to sets of known DOI aliases (Section 3.1) and, then, we further investigate those fDup-2021’s equivalent objects that do not correspond to known aliases (Section 3.2).
To perform a preliminary analysis on the quality of the output of fDup-2021, in our first experiment, we leveraged information from doi.org’s REST API2 regarding DOI aliases. Reporting DOI aliases is the default mechanism for registrants of DOIs to report duplicate DOIs3. Since not 2In particular, we gathered data from the HS_ALIAS field provided by the API.
3DOI aliases: https://www.crossref.org/documentation/reports/conflict-report/#00243 (accessed Dec 20th 2021) all duplicates are reported by the respective registrants, DOI aliases cannot be used to quantify the false positives that DOI deduplication algorithms produce. However, any DOIs that have been reported as aliases are guaranteed to refer to equivalent objects, hence they can be used as a ground truth to quantify false negatives and this is how we leveraged them in this experiment.
Since gathering the aliases for all distinct DOIs in the OpenAIRE Research Graph (>120M) is a time-consuming process (especially, if the implemented process makes responsible usage of the API respecting request limits), we decided to restrict our analysis only to those DOIs that are reported to have at least one equivalent DOI according to the fDup-2021 algorithm (a more complete evaluation is planned for an extension of the current work). Our snapshot of the graph (produced on October 26th, 2021) contained 112 216 333 distinct deduplicated entries (i.e., distinct OpenAIRE IDs) in total, 5 885 861 of which contained at least two equivalent DOIs. Using doi.org’s REST API we gathered all the aliases of the respective distinct DOIs (14 427 982 in total) and generated the corresponding groups of DOI aliases. It should be noted that 6 185 of the DOIs of the graph were problematic, i.e., unresolvable at the time of data gathering.4
We, then, compared these sets of aliases with the sets of equivalent entries provided by fDup-2021. During this comparison, we ignored all unresolvable DOIs (i.e., the analysis was performed using the rest). A summarisation of the results is presented in Table 1.
In particular, we found that a lot of fDup-2021’s deduplicated entries (32 476) were completely compliant with the list of known aliases (i.e., were confirmed true positives). Also 1 100 of the entries could be considered as false negatives, since they did not contain even one known alias. However, the vast majority of the deduplicated entries were containing groups of known aliases (hence, implying missing aliases or false positives). Finally, only a negligible number of the deduplicated entries contained only unresolvable DOIs.
It is evident that fDup-2021 produces a very small number of confirmed false negatives (they account for less than 0.02% of the examined entries). In addition, it seems that fDup-2021 identifies a very large number of equivalent DOIs which are not reported as aliases in doi.org. In the next section, we attempt to determine whether this can be mainly attributed to a large number of false positives, or if a huge number of equivalent DOIs are not reported as aliases.
In this experiment, we further investigated fDup-2021’s deduplicated entries that involve sets of DOIs that have not been reported as aliases (line 3 in Table 1). Our main objective was to get 4It should be noted that although OpenAIRE aggregates data from multiple resources (as discussed in Section 2.2), it is not responsible to guarantee that all collected DOIs are resolvable. insights about the scale of false positives in fDup-2021’s output. The only way to fulfil this objective is to have expert judgements on the sets of equivalent objects that the deduplication algorithm produces. The experts use DOI-related metadata and the corresponding content (e.g., the manuscript in case of publications) and provide judgements regarding the correctness of the algorithm output (i.e., if the reported DOIs correspond to equivalent objects or not). We followed this approach assigning the respective task to 4 experts (computer engineers, two of them PhDs). However, since the manual inspection is time consuming and the data to be examined is immense (more than 5.8M entries), we opted to assign a sample of 300 randomly selected entries per expert, resulting in a dataset of 1 200 entries. Each expert was given the task of assigning each set of equivalent DOIs with one of 8 predetermined class labels (Table 2).
Each of the classes has particular semantics, explained in the ‘Interpretation’ column. These semantics determine whether the objects in the respective group are equivalent or not (‘Judgement’ column). The dataset that has been generated by the aforementioned process, was made openly available on Zenodo5 under CC-BY license.
Figure 3a illustrates the proportion of deduplicated entries that have been annotated with each of the classes, while Figure 3b summarises the proportion of true and false positives; due to the existence of the AMBIGUOUS class, there were also a lot of entries for which it was not possible to provide a judgement (denoted by ‘N/A’ in Figure 3). It is evident that the majority of deduplicated entries (64.9%) produced by fDup-2021 are correct; most of them contain diferent versions of the same object and extensions of older works. The false positives, on the other hand, correspond to a significantly smaller percentage ( 23%).
Our main findings can be summarized as follows: deduplication algorthims are useful and bring significant added-value; this is highlighted by the fact that manually curated collections (like the DOI aliases) fail to report a large number of true positive equivalent objects. Specifically, for more than 5.8M sets of equivalent objects (according to fDup-2021) there is no reported DOI alias. Furthermore, fDup-2021 has adequate results, producing a lot of useful true positives; however, there is room for improvements since the proportion of false positives is relatively large. This is FALSE 23,0%
TRUE 64,9% VERSIONS 57,2% (a) Deduplicated entries annotated with the classes of Table 2. (b) True and false positives. expected since fDup-2021 is fairly inclusive (favoring false positives instead of false negatives). Some common errors were related to the fact that the algorithm is oblivious of the author lists, grouping together articles of diferent authors having the same title. Another common mistake was that it could not distinguish between main articles and their supplementary materials.
It is worth noting that this work contains a preliminary analysis on this subject. Our analysis has important limitations. For instance, for eficiency reasons, we used only deduplicated entries with at least two equivalent objects for our analysis. To alleviate this issue, more time is required to collect the DOI alias info from doi.org’s REST API; we plan it as an extension of the current work along with extending our ground truth that currently consists of 1 200 expert judgements.
In this work, we conducted a preliminary assessment on the efectiveness of fDup-2021, the deduplication process used for the creation and update of the OpenAIRE Research Graph. The main contributions of our work were the following: we explain why DOI deduplication algorithms are important; we introduce a ground truth dataset that can be used for the assessment of deduplication processes for Scholarly Knowledge Graphs (SKGs) and leveraged it to perform a first assessment of fDup-2021, providing insights on its major weaknesses. In the future we plan to perform more thorough experiments to confirm the results of the current study and we aim to design an improved instance of the gDup framework that alleviates all identified issues.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101017452.