Searching and exploring online information is fundamental for our society. However, it is common to ifnd inaccurate information on the Internet, that can quickly spread and be hard to identify. Fortunately, today, many fact-checking sources verify online information to provide online users with a means to recognize its truthfulness. These sources use diferent languages and scoring systems, which makes fact validation challenging and time-consuming. To address this issue, we propose a new release of ClaimsKG, a knowledge graph of about 59,580 claims, which covers 13 diferent fact-checking sources and provides a structured way to retrieve verified online claims. ClaimsKG is built using a pipeline that makes use of entity linking and disambiguation tools to fetch entities from DBpedia and an ad-hoc scoring normalization system. ClaimsKG is used as a showcase to provide the public with interesting and verified information about events of our times.
Fact-checking is the task of assessing the veracity of claims made by the public. Currently,
there is worldwide concern over the spread of fake news and how it afects social, political,
and economic well-being. In recent times, we have seen misinformation spreading faster than
truth [
Several studies have utilized machine learning to identify fake news, and one such approach
is the Deep Triple Network (DTN) [
This section describes the latest release of ClaimsKG (available at https://doi.org/10.7802/2469) and reports its statistics. ClaimsKG contains 59, 580 claims harvested from 13 popular factchecking sources. The websites are selected based on the International Fact-Checking Network’s (IFCN)6 signatories list and considered only sources referred by the fact-checking community as highly reputable. The list of covered sources is mentioned in Table 1. As part of the new release, we have crawled new claims from the fact-checking sites that were included in the previous release and added Factograph, Fatabyanno, Eufactcheck, Vishvasnews and Polygraph. This release does not contain claims from Factscan.ca since its website is no longer online. However, past harvested data from Factscan.ca will still be available in previous ClaimsKG releases. Harvested sources contain claims in various languages like English, French, Russian, Urdu, Hindi, Punjabi, Assamese, Tamil, Malayalam, Gujarati, Telegu, Marathi, Odia, and Bengali, thus making it interesting for a broad audience. The time frame for collected claims ranges from 1996 to August 2022. Since these sources were launched at diferent points in time, the start year for the earliest claims of each website is diferent. This allows the study of a multitude of entities through fact-checked claims that are contained by the sources and provides the users with the possibility to study events over a long period. In the latest version, the earliest claim from the year 1996 belongs to the website Snopes. This claim describes “A hostess named Deborah Gail Stone working the America Sings attraction was crushed to death by a rotating wall.”, labeled as “True”. Towards the end of the pipeline (see Section 4) and after each run we generate statistics, both global and per source, to monitor the health of the extracted data and also to keep track of the recent changes in the websites. Table 1 provides information on the total number of claims collected from each of these sources, the total number of entities mentioned in these claims and their reviews, and the veracity label of claims obtained from each website. One can find further information such as the percentage of claims that contain date published and author names, the number of entities per review, the number of entities per claim, and the total number of keywords for each fact-checking source on ClaimsKG website (see Table 2). For other information such as the latest data dump, SPARQL query endpoint, usage instructions, and source code, refer to Table 2. The latest release of ClaimsKG supersedes the previous versions as it contains all the claims from the previous versions together with additional claims as well as improved entity annotations.
In this section, we discuss the processing pipeline of ClaimsKG and also discuss the updates for the newest release. The pipeline consists of two major building blocks, namely the Extractor and Generator. The steps of the pipeline are summarized in Figure 1.
In the Extractor module, we perform web scraping of the identified sources. The web-crawling process is diferent for each source since it is tailor-made and adapted to the structure of each website. For this release, we added five new sources which needed new ad-hoc sub-modules. This module collects the information in a JSON and consolidates it as a CSV file. The data consists of the claim text, its truth value or original rating, the claim body, a link to the claim review from the fact-checking website, the references cited in the claim reviews, the author of the claim, the author of the review, the date of publication of the claim, the date of the review if available, the title of the review article, and a set of topic keywords extracted from the source websites if available. This module’s main pipeline and working mechanism to harvest data have remained similar to the previous ClaimsKG-generating pipeline but we have made changes according to newly added sources. This module does not perform any data processing so that the time for harvesting the source websites is not overloaded with additional computation.
The output of the Extractor serves as input to the Generator. The Generator performs: i) entity annotation and linking ii) rating normalization, and iii) lifting and serialization.
In this phase, we perform Named Entity Recognition and Disambiguation (NERD) of the claims and their reviews. Diferently from the previous pipeline, the entity annotation task is performed within the Generator so that it can be applied to the whole downloaded corpus of fact-checked claims at once. This allows a better separation of various modules of the pipeline, making it easy to maintain. One major update in this module is the use of Python Entity Fishing Client (PEFC)7 instead of TagMe8. The reason for this choice is twofold: (i) TagMe code is legacy and we wanted to move to a more recent and easily deployable service, (ii) PEFC supports multilingual claims in ClaimsKG and performs entity recognition and disambiguation against Wikidata in 11 diferent languages (English, French, German, Italian, Spanish, Arabic, Mandarin, Russian, Japanese, Portuguese, and Farsi). PERC uses GROBID-NER9, a Named-Entity Recogniser based on GROBID for recognizing the Entities. GROBID-NER has 27 Named Entity classes and is specifically dedicated to supporting the resolution and disambiguation of entities against Knowledge Bases. We use SPARQL queries through the Entity Fishing API to fetch DBpedia entities. We run a local version of the NERDClient, using the latest available dump of Wikipedia and Wikidata on Feb 1, 2022, as reported in the guideline10.
We observed that fact-checking source websites have diferent rating systems with non-uniform labels. For example, Politifact uses labels like “Pants on Fire” while AFP Factcheck has values like “Misleading”, and “Satire”. To make the rating uniform, we provide a normalized rating score for all claims in the dataset, alongside the original ratings. We classified sources into four categories TRUE, FALSE, MIXTURE, OTHER respectively indicated within ClaimsKG with rating values 3, 1, 2, -1, and only labeled a claim as TRUE or FALSE if it was completely true or completely false and did not have any ambiguity in their ratings. MIXTURE is assigned to claims which hold a degree of both truth and falsehood, such as “half-true”, “Truth! But Postponed!”, or “misleading”. OTHER is for claims that do not fit into the TRUE/FALSE or MIXTURE categories and has rating names like “Pending Investigation” or “photo out of context”, among several others. The entire approach to rating normalization is similar to what was performed in the previous releases. In the newest version of the generating pipeline, this module has been extended to normalize the ratings of the newly added sources.
The data model of the KG is available on the ClaimsKG website mentioned in Table 2. We used the Rdflib 11 python library to create the model and an abstract RDF graph to then serialize it in 7https://github.com/kermitt2/entity-fishing 8https://sobigdata.d4science.org/web/tagme/tagme-help 9https://github.com/kermitt2/grobid-ner 10https://nerd.readthedocs.io/en/latest/build.html 11Rdflib: https://rdflib.readthedocs.io/en/stable/ one of the supported formats (TTL,n3, XML,nt, pretty-xml,trix, trig, and nquads). We generate unique URI identifiers as UUIDs that are based on a one-way hash of key attributes for each instance. We present an exemplary claim in Figure 2.
The publication of the latest release of ClaimsKG facilitates the uncovering of several relations, patterns, and trends between the entities, claims, and their sources. The SPARQL query endpoint12 allows the fetching of information about specific entities and also supports the execution of federated queries from external knowledge bases like DBpedia. Here we present a few of the use cases as an example of what could be achieved using ClaimsKG.
The query in Figure 3(a) finds all claims mentioning Coronavirus. For each claim, it returns the text, the date, the rating, and the review URL by the fact-checking website. We can further drill down to finding only false claims about Coronavirus for a particular year by adding the “ratingValue=1” filter, as mentioned in Figure 3(b). This query returns, among many claims, the following one “Eating bananas is a preventative against the COVID-19 coronavirus disease.” along with its date published (2020-03-22) and link to the original fact check webpage https://www.snopes.com/fact-check/bananas-coronavirus/. The result of this query shows how ClaimsKG can bring claims from diferent fact-checking sources about one topic in one place, which makes it easier and time-saving for the exploration of facts for a user. Searching for the same result manually would have been extremely laborious or unfeasible. A user would have to visit each website and search for claims related to a particular entity or topic (in this case Coronavirus), which might or might not be allowed for the source website, and would have to manually look for the claims belonging to a particular veracity label.
The query in Figure 4(a) is an example of fetching trends or important events in history. The query fetches all claims regarding the LGBT community and their corresponding year. The result of this query shows a sudden spike in the year 2018 from 2017 with a number of claims that rises from 21 to 75 (see Figure 4(b)). This spike can be attributed to the event of legalizing same-sex weddings by the Australian Parliament in Dec 2017, resulting in a sudden rise in claims regarding this topic. One could look for interesting entities like “Black lives matter”, “Global Warming” and the “Great Recession” which would fetch us claims regarding these topics. 12SPARQL query endpoint: https://data.gesis.org/claimskg/sparql These queries could be particularly useful for social scientists, who are interested in studying specific phenomena at specific points in time and can find out which information (both true and false) was spreading online.
This example demonstrates the interesting ability of ClaimsKG to fetch information from external databases with the help of SPARQL queries. For example, the query in Figure 5 fetches false claims regarding all Presidents of the United States who belonged to the Democratic Party. As the reader can see, the information about which president belonged to the Democratic Party is given by DBpedia and used to explore ClaimsKG. The query outputs the claim text, the review URL, the name of the President as a DBpedia resource and the claim’s rating. The results display claims such as “Barack Obama began his presidency ‘with an apology tour’.” or “Joe Biden calls Pennsylvania voters who don’t support him ‘chumps’.” which were rated as false by the (a) (b) fact-checkers published in the fact-checking website PolitiFact.
In recent years, the spread of diseases and the number of claims regarding them have been on the rise. We tried to have a closer look at this topic. Figure 6(a) fetches claims that contain mentions of any disease in ClaimsKG. The query outputs 806 results along with the source review URL and the disease entity that was mentioned. Disease entities were retrieved from DBpedia. From the results, we filter out the top 5 diseases that were mentioned in these claims which show that Cancer was the highest (Table 3). Apart from the diseases mentioned in this table, there were also other diseases like Plague, Strabismus Palliative care, and others that were sparsely mentioned in the Knowledge Graph. After analysis of the veracity labels, we plot a graph of the entities and their truth values in Figure 6(b). The plot shows that Cancer is the most discussed and mentioned disease, and almost 47% of the claims made about cancer are false. These included claims like “Drinking hot coconut water kills cancer cells” or “An association of pediatricians “admitted” that HPV vaccine Gardasil causes ovarian “failure” or cancer.” There are very few true claims, only 0.06% of the total which included claims like “Four in ten cancer patients lose their life savings after starting treatment.” which is highly alarming. There are mentions of other diseases like influenza, infection, diabetes, and mental disorder that are predominantly present in the Knowledge Graph. On a general inspection, it is observable that the number of false claims is always more than claims labeled as TRUE or MIXTURE for any kind of disease. There is also a considerable percentage of claims that are rated as OTHER which did not have any clear rating value associated with them. For example, the claim “Asparagus has miraculous cancer-fighting properties.” published in Snopes has a rating of “Unproven” which is clearly neither true nor false. Likewise, OTHER claims included claims with original ratings such as “research in progress”, “outdated”, etc, and sometimes no rating attached to the claim.
Diseases http://dbpedia.org/resource/Cancer http://dbpedia.org/resource/Influenza http://dbpedia.org/resource/Infection http://dbpedia.org/resource/Diabetes http://dbpedia.org/resource/Mental_disorder We present the latest release of ClaimsKG, a knowledge graph of fact-checked claims which enables structured queries about their related metadata. We describe the pipeline changes made in this version and provide detailed statistics of the data. We also demonstrate use cases to show how ClaimsKG can be used for data analysis for various social-science-related research questions. We observe that the NERD tool fails to recognize some entities. This gives us the scope for further improvements. In the future, we aim to enhance the entity annotation capabilities with more focus on the social science domain. We plan to include more sources with diverse languages and perform multilingual entity linking and disambiguation for enhancing the quality of ClaimsKG. We also intend to analyze how the same or similar claims are covered across diferent sources and the degree of agreement between the fact-checking websites. ClaimsKG will foster new discussions about topics related to specific domains and support users in the exploration of online truthfulness about specific facts.