Wikipedia authors cite external references to support claims made in their articles in order to increase their validity. A large number of claims, however, do not have supporting citations, putting them in question. In this paper, we describe a study in which we attempt to retrieve relevant citations for claims using a variety of information retrieval algorithms. These algorithms are inspired by bibliometric and altmetric insights that exploit readership data from Mendeley's community and rerank results using a Bradfordising approach. The results of the small scale study indicate that both of these approaches can improve upon basic keyword-based search, typically used in digital libraries, in order to return relevant documents for unsupported claims.
Wikipedia has shown to be of extreme value not only for the general public
but also in the academic world. Research has shown an increasing number of
scholarly publications citing Wikipedia and that academic institutions are one
of Wikipedia’s major consumers [
Similar to scholarly literature, Wikipedia authors are encouraged to cite reliable and verifiable1 information sources that back up their claims and therefore strengthen the information quality of articles. This practice has been widely adopted, resulting in a rich set of Wikipedia articles with numerous references.
Experts, however, might still find two serious concerns regarding information quality when it comes to Wikipedia claims and their associated references. On the one hand, despite the large number of already existing references, it is still commonplace for co-authors or readers to encounter unsupported claims in Wikipedia articles. These claims needing corresponding evidence are marked with a {{citation needed}} tag2. It is then up to Wikipedia contributors to find the corresponding references that support such claims.
In order to increase the number of cited claims in Wikipedia, it would be useful to develop a tool that can automatically suggest articles that back them up. Wikipedia contributors could then check if the articles indeed back up the claims and choose to insert relevant ones into Wikipedia pages.
In this paper, (i) we investigate the distribution of citation sources in Wikipedia
to better understand what authors are currently using as verifiable evidence and
the extent to which citations are missing, and (ii) we explore if findings in the
field of bibliometrics can be exploited in developing a system that can
automatically retrieve articles that back up claims. The second study is particularly
focussed on comparing the well established technique of Bradfordising [
3 Wikipedia English Official Offline Edition (version 20130805) [Xprt] - http:// academictorrents.com/details/30ac2ef27829b1b5a7d0644097f55f335ca5241b
claims that require citations. The number of missing citations in Wikipedia articles indicates the need for a tool that can help people to retrieve articles that back up claims. 3
In this study, we were interested in applying some insights from bibliometrics
and altmetrics to inform the design of a tool that can help retrieve articles
that support natural language claims. The behaviour of three algorithms was
investigated. The first is Bradfordising, a technique that has been shown to
improve the ranking of research article results in search engines [
All algorithms were investigated using the popular search engine Lucene6. The article metadata (e.g. title, authors, year of publication, abstract) of Mendeley’s 100 million research articles were indexed. For each claim, the text of the claim plus the title of the Wikipedia page were entered into the search engine and the results were reanked based on either Bradfordising, readership or a combination of Bradfording and readership.
The standard Bradfordising approach was followed, applying it to the first 100 results. That is, the first 100 results were reranked so that the articles from the most frequent publication venue, appearing in the first 100 results, were ranked above the articles appearing in less frequent publication venues, from the first 100 results.
In order to exploit Mendeley’s readership information, a new query handler was written in Lucene, that extended the basic keyword-based search with a weighted boost. The weighted boost is based on a logarithmic function of the number of readers that an article has, as follows; score log10(number_of _readers + 1)
The final score given to each result was based on its original keyword-based score plus the boosting given by the logarithmic function. As a result, articles that had more readers should have higher ranked positions.
Finally, the third algorithm combines both approaches, first applying the readership bias and then Bradfordising.
As a case study, a small scale gold standard data set was manually created by selecting 10 Wikipedia pages with claims citing scholarly articles in them. A claim was randomly chosen from each Wikipedia page and associated with the scholarly article that it cited. It is recognised that more than one scholarly article may support the claim made. This approach, despite this limitation, is common practice in the information retrieval community, as it provides enough information to fairly compare different algorithms and can be fully automated for large scale testing. The task of the algorithms is, given a claim, to retrieve articles that can be used to back it up. We employed two baseline systems: (i) Google Scholar and (ii) Mendeley’s catalogue search. Both cover a broad range of research disciplines and are two of the world’s largest research collection repositories. These baselines were compared to versions of Mendeley’s search engine enhanced separately using Bradfordising and readership biases, as described in the previous section.
Five of the 10 selected claims are provided as examples (Table 2). These claims are made using natural language sentences that paraphrase and/or summarise findings from research articles. The 10 claims cross multiple disciplines of research, just as Google Scholar and Mendeley’s collections do.
Based on the claim and the title of the Wikipedia article page, a query was constructed. The query contained all the words that appeared in the claim with the Wikipedia article page’s title concatenated to it. This query was used to evaluate each of the approaches: (i) Google Scholar (Google Sch.), (ii) Basic Mendeley Keyword Search (Men), (iii) Mendeley + Readers (Men+R), (iv) Mendeley + Bradfordising (Men+B), and (v) Mendeley + Readers + Bradfordising (Men+R+B). The first 100 results lists from each tool were gathered and the position of the cited article in each list was recorded. The closer the article’s position to the start of the results list, the better the approach performs. 5
Mendeley’s basic keyword search outperformed Google Scholar in retrieving the correct target document (i.e. the document actually cited in the Wikipedia article) in the first 100 results in 8 cases. Google Scholar failed to retrieve the target documents in the first 100 results for all queries. When considering only the top 10 results returned, 2 of the sample queries retrieved the correct target citations in the top 10 results using Mendeley’s basic keyword search.
Three algorithms were tested based on bibliometrics and altmetrics insights. The use of readership counts ranked the target articles higher, on average, than the use of Bradfordising and the combination of readership and Bradfordising, resulting in 5, 4, and 3 top 10 hits respectively. When considering the number of cases in which an algorithm ranked the target document highest, Mendeley + Readership provided the highest ranked results in 5 of the cases. Mendeley’s basic keyword search ranked the target article higher in 2 cases compared to Mendeley + Readership + Bradfordising’s single case. The results suggest the combination of the 2 algorithmic enhancements, readership boosting and Bradfordising, appear to produce worse results than using either of these algorithms alone.
There were 2 cases when all algorithms tested failed to retrieve the target article in the top 100 results. In both of these cases, the claims did not contain the keywords present in the metadata of the articles. 6
There is evidence that scholarly articles are increasingly citing Wikipedia. One
study showed that Wikipedia had been cited 3,679 times within a reference data
set taken from the Web of Science (WoS) and Elsevier’s Scopus databases [
When it comes to the influence of readership, incorporating the readership
count or popularity into a information retrieval system has been studied by many
research groups. Researchers proposed that the readership count can be seen as
an indicator for the quality of the retrieved articles and the to rerank the results
accordingly. They found that among multiple quality metrics, the popularity
contributed significantly to the improvement of the results [
None of the algorithms tested managed to retrieve the target article in their
top 100 results in 2 of the tests. In considering the 2 queries, it appears that they
do not share enough keywords in common with the target article’s metadata.
This points to the need for an alternative representation of articles beyond
metadata and possibly an alternative representation of the query itself. Including the
full text could possibly prove beneficial in such scenarios as suggested by [
The presented work was developed within the EEXCESS project funded by the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement number 600601. The Know-Center GmbH is funded within the Austrian COMET Program Competence Centers for Excellent Technologies of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency (FFG).