Fact-checking is a common journalistic practice adopted to verify the truthfulness of claims and information items. Because of the demanding nature of fact-checking, a significant amount of research has been devoted to the use of crowdsourcing to scale up this practice. The idea to use laypeople to fact-check information allows accessing a vast amount of human computation resources, but introduces an issue of reliability: when these tasks are performed by laypeople instead of experts, their quality might be questioned. In this paper, we introduce an ontology for modeling crowdsourced datasets of information quality assessments. We emphasize that we allow modeling information about the items evaluated as well as important metadata such as the authors of such assessments. The goal of this model is to favor interoperability among diferent datasets of the same kind, as well as to support internal analyses of the dataset themselves in terms of bias and reliability of the collected assessments.
Fact-checking is a common journalistic practice employed in order to test the veracity of claims to be reported in newspapers. In order to fact-check information, journalists look up information sources, appraise their reliability, extract information out of them, and then reason on the resulting information set. This sequence of operations is rather time-consuming, while the amount of claims that require fact-checking online is constantly growing. On the hand, this means that the workload of specialists is high and, on the other hand, their possibility to intervene real-time is limited. Crowdsourcing has revealed a useful tool to allow scaling up fact-checking. Crowd workers can, in fact, be instructed so to produce expert-like assessments, and by collecting multiple assessments about the same item (wisdom of the crowds), quality can be assured. In order to maximize the usefulness of the fact-checking (and, more in general, information quality assessment) datasets obtained from crowdsourcing, it is important to annotate them with specific metadata. This paper describes a lightweight ontology to describe and annotate crowdsourced information quality assessments. The ontology allows characterizing information about the items being assessed, the author of the assessment, and information quality details. We leverage existing ontologies like the Data Quality Vocabulary (DQV) and Schema.org (Schema) and we select and specialize their elements to serve this specific niche of information. The paper is structured as follows. Section 2 presents related work. Section 3 introduces the ontology, while Section 4 discusses example applications. Section 5 concludes.
Relevant to this line of work is the Data Quality Vocabulary (DQV) [
The work presented in this paper is also relevant to the field of FAIR data principles, as it
aims at favoring findability (especially on principle F2) and interoperability (I1) of data. We
refer the reader to the work of Poveda et al. [
The ontology we present here aims at achieving the highest interoperability while allowing modeling the necessary information resulting from the crowdsourcing tasks of interest. In the ontology, we identify three main “superclasses” that represent the core of the information we intend to model. We describe them as follows, while Figure 1 provides an overview. Target Item The target item represents the item to be evaluated. This ontology means to model quality assessments on information items online, therefore this class specializes the DigitalDocument class of Schema.org.
Worker The worker class is meant to characterize the author of a quality assessment, and is
thus modeled from the Person class of Schema.org. The instances of this class could be more or
less populated depending on the level of anonymity granted to the crowd workers. Information
to populate instances of the worker class can be provided both by the crowdsourcing platform
or by the worker who responds to demographic questions in the crowdsourcing task.
IQ Assessment This class models the information quality assessment that the worker provides
about the target item. This class requires the specification of a quality dimension (e.g., precision,
accuracy, truthfulness; see [
prov:startedAtTime dqv:isMeasurementOf "^Y^xYsYdY:d-MatMeT-DimDeTHH:MM:SSZ" dqv:dimension
dqv:metric
The proposed ontology is meant to produce and annotate Linked Data representation of
crowdsourced information quality assessments. Given that these data are often produced in CSV
format, we refer the reader to the CSV on the Web standard [
In the real-world research project “The Eye of the Beholder” 2, we aim to building or leveraging
a platform for scholars to train machine learning pipelines to automatically assess quality of
online information. To train these pipelines, we make use of a dataset collected from a crowd
sourcing platform called CrowdFrame [
Through investigation and comparison, we chose the KNIME [
The desired tool is composed of several KNIME workflows working in tandem with each other. These workflows automate the process of data exploration, model training, result interpretation, and pipeline comparison. To achieve this goal, we designed the CrowdIQ ontology to design the data interfaces for these workflows, and we also need to validate the training data files corresponding to this ontology before feeding the to those workflows.
The requirements for integrating the ontology in KNIME are specified as follows. Upload data from the crowdsourcing platform Once the crowd sourcing task has been completed by enough workers, our user wants to load the resulting assessments into KNIME for further analysis. The platform provides our user with a URL of the metadata of the file (the url of the data itself can be inferred from the metadata). The KNIME component that we develop takes this URL as an input. The component downloads the data and metadata, and interprets the data according to the CrowdIQ ontology. It executes some sanity checks: it returns an error when, for example, no documents or no users are present. It outputs three tables: for workers, documents and assessments respectively. It also outputs a visualization of the realization of some of the attributes in the data: e.g., which dimensions and worker attributes are present. Specify metadata When the user wants to provide data from a platform that does not give machine readable metadata compatible with our ontology, the user needs to specify which fields relate to which attributes in the data. Instead of providing an URL to the metadata as above, the user should be able to specify in a graphical user interface, for each column of their CSV data, which attribute in the ontology it represents. Depending on the format of the data that the user provides, the CSV data can either consist of one table with all information combined, or three diferent tables for workers, assessments, and documents respectively.
To account for the user stories as described in the previous section, we propose three types of validation to integrate in KNIME workflows. First, if the training data is stored in CSV format, a CSV-on-the-Web validation is essential. In this step, metadata annotations are added to the input CSV files (manually or by referring to an existing metadata to interpret the data files (encoding, data types of fields, etc.). At the same time, these files will be converted into a linked data format. Next, the obtained linked data should also be tested for consistency with the ontology. This is done through an ontology-based validation that maps the data fields to the expected classes/properties defined in the ontology and further cleans the data. Finally, the resulting data should also be validated against the constraints defined in the ontology. This can include various checks such as missing/duplicate properties, missing or improper type arcs, and inconsistent value ranges. In addition, we propose to extend the CrowdFrame platform to output a metadata file describing the output CSV according to the CSV on the web standard. In the interface for creating a new crowd sourcing task, the user should be able to specify which ifelds correspond to which attributes in the ontology, for those fields for which it cannot be inferred automatically. A prototype of a Knime pipeline that reads CSV and validates against the ontology, can be found on GitHub3. It wraps the csvw library4 in a Python scripting node.
In this paper, we introduce CrowdIQ, an ontology for modeling crowdsourced information quality assessments. The ontology aims at achieving minimal commitement, while specializing existing models to represent common information about information quality crowdsourced tasks. The ontology is described along with a possible utilization scenario. The goals of this model are twofold: allow interoperability among existing crowdsourced datasets, as well as to allow inspecting each dataset separately in order to assess its reliability and bias. Future work will aim at extending the model further and at providing pipeline automation components that leverage the model’s expressivity.
3https://github.com/EyeofBeholder-NLeSC/assessments-ontology 4https://github.com/cldf/csvw