Given a Wikidata claim, we explore automated methods for locating references that support that claim. Our goal is to assist human editors in referencing claims, and thus increase the ratio of referenced claims in Wikidata. As an initial approach, we mine links from the references section of English Wikipedia articles, download and index their content, and use standard relevance-based measures to nd supporting documents. We consider various forms of search phrasings, as well as different scopes of search. We evaluate our methods in terms of the coverage of reference documents collected from Wikipedia. We also develop a gold standard of sample items for evaluating the relevance of suggestions. Our results in general reveal that the coverage of Wikipedia reference documents for claims is quite low, but where a reference document is available, we can often suggest it within the rst few results.
Wikidata [
As per Wikipedia, Wikidata does not aim to be a primary source of knowledge, but rather a secondary source of knowledge: statements in Wikidata should Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). be interpreted as claims held true according to a speci c, external and authoritative source.1 Thus it is important that statements be independently veri able, meaning that parties other than the editor that added the statement should be able to verify the validity of that statement. Some statements are considered self-veri able. The cases listed by Wikidata editor guides include:2 { Common human knowledge: statements that are obvious to most and can be considered self-evident; for example, that Paris is an instance of city, that Paris is the capital of France, that city is a subclass of urban area, etc. { The value is an external source: statements that point to an external source, such as identi ers associated with the item in external catalogues. { The value refers to an external source: statements that point to an Wikidata item that itself can verify the statement, such as an album stating its artist, a book stating its author, etc.
In the case of statements not falling into one of these three categories, the onus is on the editor that adds (modi es or restores) a statement to establish veri ability by adding a reference for the statement based on an authoritative source. Authoritative sources include books, publications, news media, laws, other popular media, reputable websites, etc. Questionable sources, sponsored sources, self-published sources, etc., may be rejected as non-authoritative sources.
At the time of writing (August 2020), Wikidata describes 1.124 billion
statements about 88 million items and has over 23 thousand active users. Of these
statements, 771 million are referenced to external sources (68.56%), 68
million are referenced to Wikipedia (6.02%), leaving 286 million without reference
(25.42%).3 Of these items, 71 million (80.34%) have at least one referenced
statement. While collecting 771 million referenced statements is an impressive
achievement, more can be done to improve the coverage of references [
In summary, we see a need for research on methods to (semi-)automatically nd authoritative references for statements in Wikidata. Herein we describe our work on an initial such method based on searching over reference documents scraped from Wikipedia. This approach seems initially quite natural: Wikipedia
4 As an anecdotal example of the latter, we refer to Siri reporting the death of Stan Lee, apparently based on an invalid statement added to Wikidata: https://io9. gizmodo.com/siri-erroneously-told-people-stan-lee-was-dead-1827322243. articles are linked with Wikidata items; Wikipedia references follow similar principles of veri ability and authority as for Wikidata; Wikipedia is an older project and thus one might expect more extensive reference lists to have developed over time; a considerable number of Wikidata statements already reference a Wikipedia article that should itself cite an authoritative source; the factual nature of Wikipedia means that one could expect overlap in terms of the claims made about the same entities/items on both sites; Wikidata guides suggest to search Wikipedia for sources; etc. Our results, however, show that Wikipedia's references are quite limited in terms of coverage for Wikidata claims. 2
A number of works have analysed referencing in Wikipedia. In a study of the
quality of Wikipedia articles, Warncke-Wang et al. [
With respect to references on Wikidata, WikiCite is a Wikimedia initiative
to develop and expand the citation data available through Wikidata.5 As part of
the WikiCite initiative, Nielsen et al. [
Regarding datasets, Delpeuch [
suggestions claim Wikidata API wiki urls
articles
results search ref urls
content
with external identi ers (e.g., DOIs). Chou et al. [
We propose to scrape external reference URLs from English Wikipedia, and to download and index their content. Thereafter, given a Wikidata claim for which an editor requires suggestions of potential references, using the labels and aliases of the items involved, we will convert the claim to a search using English terms and apply the search over the inverted indexes of the content of the external documents, using standard relevance-measures to prioritise documents. Finally, to assist the editor, we will return not only the document itself, but also a snippet of text from the document that contains the relevant keywords.
We present the high-level architecture in Figure 1. The API provides an interface that accepts a claim from Wikidata (along with associated metadata) and returns suggestions of potential references. In order to provide these references, a Scraper collects and parses the URLs of external references from articles on English Wikipedia. These URLs are passed to a Crawler that downloads the URLs and saves their content into an Index. The API can then formulate a search for the claim over the Index, which returns relevant documents as results that are returned as suggestions. We consider the option of both an o ine and online mode. In the o ine mode, the Scraper and Crawler process all of (English) Wikipedia, generating the Index over the full corpus that can be searched at runtime. We also consider an online/lazy mode, where the Scraper rather accepts a list of relevant Wikipedia article URLs from the API, which are passed to the Crawler, which in turn populates the Index at runtime before the search is performed. The o ine mode has the bene t of less latency, but a priori it is not clear that performing such a crawl of all external references is feasible; also the Index would require periodic updates. The online mode is easier to keep up-to-date, where the Index rather acts as a cache, but is associated with slower runtime responses as the Crawler operates while the editor is waiting for suggestions.
Our initial goal is to study the feasibility of this overall approach, and to establish baseline methods and datasets for further research. Within this proposal a number of initial research questions arise: RQ1 O ine vs. online indexing : Is it feasible to scrape, download and index the content of all of English Wikipedia's external references o ine? Or would it be better to scrape, download and index the content online/lazily for the external references of the Wikipedia articles relevant to the claim at hand? RQ2 Coverage: How many external references can we source from the article corresponding to each Wikidata item? Can we build a corpus with good coverage of Wikidata items in general? RQ3 Search phrasing : How best should we phrase the search? Should we use only primary labels, or also aliases? What connectives should we use? RQ4 Relevance: Are traditional IR measures su cient to generate good suggestions? Should we search only in references for Wikipedia articles corresponding to the item(s) involved in the claim, or across the entire corpus? RQ5 Suggestion Quality : How often can we generate good suggestions of references for claims? Are the rankings of suggestions suitable? Can we also suggest relevant text snippets from with the documents to support the claim? In this initial work our goal is to gain insights regarding these research questions, rather than seeking de nitive answers. 4
We rst explore the o ine approach. We start with a dump of Wikidata, from
which we extracted the mapping to Wikipedia articles. These articles were then
retrieved from a 2018 HTML corpus of Wikipedia [
1 archive.org bbc.co.uk 2 doi.org nytimes.com 3 nih.gov archive.org 4 nytimes.com billboard.com 5 bbc.co.uk newspapers.com 6 webcitation.org thegazette.co.uk 7 allmusic.com sports-reference.com 8 youtube.com reuters.com 9 theguardian.com baseball-reference.com 10 archive.is bbc.com successfully downloaded and indexed in Nutch. We let Nutch decide the order of the URLs to be accessed; no con guration was made in this matter.8 Though not all URLs were processed at this point, progress in the crawler had slowed to a halt. An issue we did not anticipate was that of redirects: Nutch does not provide a clean mechanism to retrace redirects, though from the logs it is possible to retrace the URLs accessed and, in most cases, recover the original URLs. This was important to match the original Wikipedia articles and external URLs with the redirected content location of the indexed document. In total, we could recover links for 2,058,896 documents (83%) from their original Wikipedia article.
In Table 1 we present the top 10 domains for raw URLs extracted from Wikipedia and indexed (redirected) URLs. We see that the indexed URLs tend to refer to media sources. Notably (for example) doi.org is primarily a redirection service, and hence we do not see this domain appearing in the indexed URLs, which follow the redirects. Regarding Coverage, we managed to associate 3,899,953 (Q-identi ed) items with at least one indexed external reference. Of these, 1,136,477 items (29.1%) had more than one reference indexed. Complete sample: Given the incompleteness of the crawl for the full reference corpus, we decided to also develop a complete crawl for a subset of Wikidata items. Based on some initial samples, which were largely composed of items without English Wikipedia articles, we decided to split our sample into ve groups based on the Q identi ers: A: Q1{Q10000; B: Q10001{Q100000; C: Q100001{ Q1000000; D: Q1000001{Q10000000; E: Q10000001{Q100000000. This sampling is based on the idea that Wikidata ids were de ned chronologically, and that the most important entities (countries, major cities, recent presidents, etc.) would fall into the earlier groups, with later groups being populated by successively more obscure items. From each group we sample 1,000 items and then apply the same process as before; in this case we run the download to completion. 8 By default, Nutch partitions URLs by host and then randomly selects URLs within each partition. We assume an inverted index of content of potential external references and now turn to the question of how to search with the documents. We assume that the API receives a claim as exempli ed in Table 3, with the item IDs/terms, labels and aliases in English. Note that 1982 refers to a date value, where we use the lexical form as the label. There is no clear individual way to construct the search. Using just the labels may run the risk of missing some potentially relevant documents with alias terms. On the other hand using alias terms may introduce noise and return irrelevant documents. We experiment with four options: 1. construct a query for any of the three labels; 2. construct a query for any of the three labels or any property alias; 3. construct a query for any of the three labels or any alias; 4. construct a query for at least one label or alias for each of the three elements. We provide examples of the searches for each of the four options in Table 4. While it may perhaps seem quite broad to use the or connective, initial experience suggested that using and, particularly on property labels (without aliases), meant that few documents were returned as the search was too speci c. Furthermore, Solr uses the BM25F relevance metric (based on TF{IDF), which will rank documents with more occurrences of more terms more highly.
We consider searching only over the references of the article associated with the subject item (similar to the online option) to boost relevance,9 and searching over all documents collected for the o ine corpus to boost recall.
As a further feature, Solr allows for returning a snippet of each document determined to be a highly relevant part of the document for the search. The typical application of this feature is for building results lists, where the user can preview the most relevant part of the text, which also ts our use-case of letting editors preview snippets of text from di erent documents that might support a given claim. We illustrate this feature in Figure 2 for an example claim. Held-out evaluation: As an initial test of the di erent search options, for our set of 5000 items, we can use the 163 URLs that appear on a Wikidata claim 9 Another alternative would be to further include documents for the value item. We discarded this option in order to simplify experiments, observing that the value of a claim is often much more general than the subject item; for example, considering the claim that Neil Young was born in Canada, it would not make sense to search within the external references for Canada. Option 1: "university of chile" or "inception" or "1842" Option 2: "university of chile" or "inception" or "date founded" or ... or "1842" Option 3: "university of chile" or "la u de chile" or ...
or "inception" or "date founded" or ... or "1842" Option 4: ("university of chile" or "la u de chile" or ...) and ("inception" or "date founded" or ...) and "1842" and appear in our index. We take the Wikidata claim that they appear on, and measure the recall of the 163 URLs in the top 3 suggestions for each option searching with the external references of the article associated with the subject item. We also consider a baseline that selects 3 random external references for the article of the subject item. The results are shown in Table 5, where we see that the best results are o ered by search option 1, which retrieves the known external reference as a top-3 suggestion in 72% of the cases. It is important to note that any result returned may be correct as we only know a subset of the correct references, so the recall should be interpreted as a lower bound. Gold standard evaluation: Given the aforementioned limitations of the held-out experiments, we opted to manually label a subset of claims, where we choose 5 items from each of the ve groups A{E, which we then labelled. The labelling indicates which external reference in the Wikipedia article associated with the chosen (subject) item supports which claim on that item. We rst tried a random sampling of 5 items from each group but labelling became infeasible as items with hundreds of associated external references and claims were found, where manually pairing them o was considered too complex; furthermore, in the later groups, some items had only one reference associated. Instead we choose to sample items with a number of associated references close to the mean for that group. We show some statistics for the gold standard in Table 6, where we indicate the average number of claims in Wikidata per item, the average number of references indexed from the corresponding Wikipedia articles per item, and average percentage of claims per item supported by at least one reference from the corresponding Wikipedia article. The All column considers the statistics across all groups. It is worth noting that given the low numbers of references for groups C{E, the results for searches become somewhat trivial; for this reason we will include random baselines. The searches of our gold standard are then formed by the claims for which at least one supporting reference is found. A 48 23
B 18 7
C 17 4
D 13 2
E 8 3
All 21 7 37%
In Figure 3 we present the normalised Discounted Cumulative Gain (nDCG) metric for the di erent search options with respect to the di erent groups. We also include the random baseline for comparison. Intuitively speaking, a score of 1 indicates the best possible ordering possible, ranking all supporting references above all non-supporting references. The results are divided by group. We see that for groups A and B, the search methods perform much better than the random baseline. The best results are given for group E, but this is largely due to the trivial nature of the task when given few references, as noted by the high performance of the random baseline. In general there is not much di erence of note between the di erent search options, though we can perhaps indicate that Option 1 performs (slightly) best and Option 4 performs worst.
Given that the nDCG measure is somewhat di cult to interpret, in Figure 4 we present the Any@k measure: noting that in order to establish veri ability, in general one reference is su cient, we look at the percentage of claims/searches for which at least one suggestion in the top-k was relevant. We believe that this gives a more direct measure of how the reference suggestions perform in practice. We see that considering the top-3 results, Options 1{3 succeed in nding supporting references in close to 88% of cases, increasing to 90% for top-4 results. Snippets: For the 25 gold standard items, we manually evaluated the text snippets that Solr selects to indicate why a document is relevant, where we found that only 9% of these snippets were su cient to support a claim by themselves, although they were often useful to help understand more about the content of the webpage without visiting it. In particular, we found that reference documents often support claims in a more implicit way, requiring a more general understanding of di erent parts of the text, rather than just one part. Global results: Finally we used our 25 gold standards to run searches over the full corpus of 2.5 million references using the search options previously outlined. The results were largely negative: the best results were obtained using Option 1, which yielded Any@5 values of 19%. Manually revising the results, we found that most of the documents returned by Solr were irrelevant to the topic at hand, due to the broader corpus being used. It may, however, be possible to better ne-tune the queries to return better results. 6
We now brie y summarise our insights regarding research questions RQ1{5.
RQ1 O ine vs. online indexing : Online indexing was slow, with references for
well-known entities taking up to 20 minutes to download and index. However,
achieving a complete corpus by o ine indexing is very time consuming.
RQ2 Coverage: Similar to the results of Piscopo et al. [
RQ4 Relevance: BM25F gave good results when searching for claims within the references of the corresponding Wikipedia article, but poor results for the given search phrasing options when considering the full corpus.
RQ5 Suggestion Quality : When a claim has a supporting reference in the corresponding Wikipedia article for the subject item, the proposed method will nd at least one such supporting reference in the top-5 results around 90% of the time; however, the generated snippets rarely su ce to support the claim, meaning the editor will often have to visit and revise the documents.
When claims are supported by references in the corresponding Wikipedia article, traditional Information Retrieval methods appear su cient to give good recommendations. The more general issue we encountered in this initial research is that few Wikidata claims have relevant references in the corresponding Wikipedia article. This suggests two possible future directions: { O ine: Given that some Wikidata items do not have an associated Wikipedia article, that many Wikipedia articles have few references, etc., it would be interesting to develop a broader corpus with more documents from the Web, perhaps from the Common Crawl. In order to ensure that the documents are authoritative, this corpus might only include content from web-sites with a threshold number of references detected in Wikipedia. A challenge will be to ensure the relevance of search results, where the connection between the Wikidata items and the indexed documents would be lost; however, this challenge could be addressed with more advanced relevance measures based on the elds of the documents, comparing the similarity of each document's content to relevant Wikipedia articles, amongst other such techniques. { Online: We have found that our online option is too slow due to the need to crawl references at runtime. Another option similar to the online option { in terms of obviating the need for a local index of documents { would be to use the existing infrastructure of major search engines to search the Web at runtime, ltering for sites that are considered authoritative. A major bene t of such an approach is that the (costly) retrieval, indexing and refreshing of content could be delegated to the search engine. The downside of such an approach would be the issues of respecting rate-limits for the search API, plus the inability to pre-process the content for the speci c task.
In summary, a method for automatically suggesting references for Wikidata
claims would help human editors to be more productive, and would help to
make better use of their (often volunteered) time. As a result, the coverage of
references on Wikidata would increase, and its quality as a secondary source of
knowledge would improve. While this paper does not provide a de nitive
solution, we have gained some important insights into the strengths and limitations
of basing suggestions on Wikipedia's references. We further provide online
material to facilitate future research, including the retrieved content of a large subset
of documents found in the reference sections of English Wikipedia.
Material online. Available on Zenodo [
Acknowledgements. This work was funded by Fondecyt Grant No. 1181896 and ANID Millennium Science Initiative Program ICN17 002.