Wikidata is a rapidly growing user-edited open knowledge graph that provides easy access to structured data. Since Wikidata allows contradictory information, references are crucial for supporting statements and tracking the source of information. Consequently, investigating the use, types, and scientific value of references within Wikidata is essential. In this paper, we will first conduct a heuristic evaluation of Wikidata references using a sampling method. Subsequently, we will focus on a specific category of references, Digital Object Identifiers (DOIs), known for citing scientific publications. Our sampled Wikidata statements analysis indicates widespread adoption of the DOI system within Wikidata. To assess the quality of scholarly resources referenced in Wikidata, we used percentile metrics derived from the OpenAlex platform. Additionally, h-index indicators from OpenAlex were employed to evaluate the credibility of these sources and determine whether the Wikidata citations originated from reputable sources or publishers. Our findings show that papers in the social and physical sciences tend 4th International Workshop on Scientific Knowledge: Representation, Discovery, and Assessment, 12 November 2024 - Baltimore, ∗Corresponding author. †These authors contributed equally.
Wikidata is an entity-oriented database designed to represent items related to various topics, concepts, and objects, along with detailed claims and qualifiers describing these entities. Similar to Wikipedia pages, the Wikidata Knowledge Graph relies on references to support the claims and statements it entails. These references should point to the source of the provided statement, with statements supported by and linked to at least one source according to internal Wikidata guidelines.
Since its rapid expansion from 42.3 million items in 2017 to over 113 million items in 2024 [
Several researchers and practitioners have investigated diferent features and characteristics of Wikidata, with much of the work focusing on the quality of the sources. Adequate, relevant, and trustworthy references are increasingly important for improving the reputation of Wikimedia projects. In contrast, missing or inappropriate references can afect reliability and hinder the reuse of data.
In this paper, we first examine the quality and structure of external references within Wikidata, with a particular emphasis on their scientific character and background. Next, we focus on a specific category of references: scientific publications identified by Digital Object Identifiers (DOIs) and examine how
CEUR
ceur-ws.org these DOIs are utilized within Wikidata. We compare the performance and impact of these referenced papers in Wikidata with their performance in the broader scientific community, using data from the OpenAlex dataset.
In Wikidata’s data structure, a claim combines a property with at least one value and optional qualifiers to provide information about items. When this property-value pair is enriched with additional information (such as references or ranks), it becomes a statement. A claim without a qualifier is called a snak, representing a basic triple consisting of an item and a property-value pair. References or other qualifiers, which can be appended to statements, are connected to the value, allowing for multiple references for a single statement. Wikidata accommodates contradicting statements to reflect controversial, uncertain, or debatable information, requiring references to support the entire statement.
Albeit, there have been numerous studies on Wikidata and its references, no study has solely examined
the scientific background of sources in more detail with a special focus on the used properties and
identifiers. The research conducted by [
The authors in [
Another study [
A recent Wikidata analysis [
Another paper [
Over the past few years, the number of Wikidata items classified as scholarly articles has significantly increased. As of September 2024, Wikidata contains over 44 million scholarly articles, reflecting an increase of more than 6 million in the past two years. All of these articles are instances of a single entity, Q13442814, in Wikidata.
We analyzed the properties and their frequencies within reference nodes. As of September 2022, we identified more than 5,267 distinct reference properties, encompassing a total of 335,960,448 records. The top 10 reference properties represent 89.2% of the total population and are illustrated in Figure 1. The two top most used reference properties (P248 and P813) are used more than 80M times. Those properties can be universally used and reference to either external or internal reference values. There are 65,151,203 reference URLs in Wikidata which point towards external references.
PubMed ID with 29,644,517 and PubMed Central ID with 5,123,024 reference counts show that there are many references linked with those identifiers. One possible reason these IDs are more prevalent in Wikidata is the presence of a bot that links the IDs to Wikidata items. This may also explains the large number of data properties in three categories: ’stated in’, ’retrieved’, and ’reference URL’, as shown in 1.
The most popular identifiers for scientific references are PubMedID and PMCID, which indicates that in the fields of biomedical and life sciences, Wikidata entries link to more scientific sources than in other areas. The distribution of the top 10 Wikidata reference properties for scientific identifiers excluding PubMed ID and PMC ID is depicted in Figure 2.
where the is the position of a paper’s citation count within its field, publication year, and open access status, and the is the number of papers within the same grouping. Next, we translated the percentile ranking from OpenAlex to Wikidata by applying the percentile rank calculated for each paper in OpenAlex to the number of citations recorded in Wikidata. This allowed us to place the Wikidata citation counts within the same percentile framework used in OpenAlex. Using a common percentile ranking system enables a more accurate comparison of paper performance, as it avoids the creation of two independent ranking systems, which could lead to inconsistencies in interpretation.
In Figure 4, we can see that especially in the social sciences, the box for Wikidata is much higher, and the whisker ends before the box of the OpenAlex part ends, indicating that the interquartile range of the percentiles for Wikidata is higher. We can also see that the box for Wikidata in physical sciences is higher than the box for OpenAlex, but the whiskers are of the same length, suggesting a higher median but similar variability. The diferences in the other two categories are not as pronounced; the boxes in life sciences and health sciences are slightly bigger for the Wikidata one.
In Figure 5, we wanted to see how the performance according to the OpenAlex percentile ranking changes when we separate them into the diferent journal categories we defined. The most apparent diference is that the percentile ranking of Wikidata references from emerging journals is much higher than that of the OpenAlex references, as we can see that the boxes are nearly at opposite ends of the scale. Also, in the mid-tier category, the percentile rankings in Wikidata are much higher than in OpenAlex. We can also see that the percentile rankings in the well-regarded category are higher for Wikidata than in OpenAlex. In the top-tier category, they are nearly the same.
We observed that, especially in the social sciences, papers cited in Wikidata generally exhibit higher citation counts compared to those in OpenAlex. This trend is somewhat visible across other domains as well, suggesting that Wikidata may include a wider range of journals in its citations. Additionally, while top-tier journals continue to receive a substantial number of citations on both platforms, Wikidata shows a more varied distribution of citations across diferent journal tiers. This could indicate a more inclusive referencing approach within Wikidata, potentially encompassing a broader spectrum of scientific contributions. However, it’s also possible that this broader distribution reflects eforts by authors to increase the visibility of their work by citing it in Wikidata.
Our study examines the quality of scientific papers cited within Wikidata, a rapidly expanding open knowledge graph that supports diverse contributions. By September 2022, we identified over 5,000 distinct reference properties across 335,960,448 records. While Wikidata has seen significant growth in scholarly references, most references rely on a few dominant properties, particularly PubMed ID and PMC ID in the biomedical sciences. This highlights a strong reliance on external, credible sources but suggests potential gaps in citation diversity across other fields.
Even though the reference system in Wikidata has improved over the past few years, this alone does not provide insight into the quality of the cited sources. To evaluate the scholarly merit of these references, we used OpenAlex metrics, such as the H-index, to assess a journal’s reputation. Our analysis reveals that papers cited in Wikidata, particularly in the social sciences, often show stronger performance metrics compared to those in OpenAlex. This trend suggests a greater diversity in the journal sources referenced within Wikidata.
However, it is crucial to acknowledge several factors that may influence these findings. One limitation is the use of the h-index as a ranking metric, which heavily depends on citation counts and may not fully capture journal quality or impact. Alternative metrics, such as the SCImago Journal Rank (SJR), could potentially ofer a more nuanced evaluation. Additionally, our analysis is constrained by the absence of certain sources in OpenAlex that are present in Wikidata, impacting the comparability of the datasets.
The diferences in citation distributions between Wikidata and OpenAlex may also arise from the nature of contributions to Wikidata, including potential self-citations by authors seeking to increase their work’s visibility.
Despite these challenges, our findings suggest that Wikidata has a broader citation distribution. However, further research is needed to understand the underlying reasons for these diferences and their implications for scholarly communication. Limitations in our methodology, such as the reliance on the H-index and the potential omission of influential sources, should be considered when interpreting our results and generalizing them to broader contexts.
We would like to express our gratitude to Marco Marsoner for his valuable contributions to this work. We acknowledge support from the Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility and Technology (BMK) via the ICT of the Future Program - FFG No 887554.