The FAIR Guiding Principles (Findable, Accessible, Interoperable, and Reusable) have been widely endorsed by the scienti c community, funding agencies, and policymakers. However, the FAIR principles leave ample room for di erent implementations, and several groups have worked towards manual, semi-automatic, and automatic approaches to evaluate the FAIRness of digital objects. This study compares and contrasts three automated FAIRness evaluation tools namely F-UJI, the FAIR Evaluator, and FAIR Checker. We examine three aspects: 1) tool characteristics, 2) the evaluation metrics, and 3) metrics tests for three public datasets. We nd signi cant di erences in the evaluation results for tested resources, along with di erences in the design, implementation, and documentation of the evaluation metrics and platforms. While automated tools do test a wide breadth of technical expectations of the FAIR principles, we put forward speci c recommendations for their improved utility, transparency, and interpretability.
MIs but o ers an alternate user interface and result representation. F-UJI [
We sought to compare and contrast three automated FAIRness evaluation tools (F-UJI, the FAIR Evaluator, and the FAIR checker) against their usability, evaluation metrics, and metric tests results. We generate evaluation results using three datasets from di erent data repositories. We discover the FAIRness evaluation tools have di erent coverage and emphases on the FAIR principles and apply di erent methods to discover and interpret the content of the digital objects. When assessing the comparable evaluation metrics, di erent tools may output con icting results because of the di erent implementation of the metric tests. We analyze these observed di erences and explore their likely bases. Our work is the rst to o er a systematic evaluation of current automated FAIRness evaluators, with concrete suggestions for improving their quality and usability. 2
This study critically examines the functioning of the FAIR Evaluator, FAIR Checker, and F-UJI. These FAIRness evaluation tools are implemented as web applications that use web service APIs to execute a FAIRness evaluation and o er an interactive user interface through a web browser (Figure 1). These tools implement new or apply existing FAIRness evaluation metrics. Each metric has one or more compliance metric tests to determine if the digital object meets the requirements of the metric. These metric tests are the actual implementation of the evaluation metrics. Users invoke an evaluation by providing a valid URL or persistent identi er (PID) of the digital object's landing page. The tool executes a strategy to harvest relevant metadata on the URL (or its redirected URL) using a combination of content negotiation, embedded microdata, and HTTP meta rel links. The tools then test the harvested metadata, and tabulate whether and/or how they pass or fail the metric test(s). Finally, the tools present the results of the metric tests as an HTML web page that may otherwise be downloadable as a structured data le. We conducted a comprehensive comparison of the automated FAIRness evaluation tools focusing on 1) the characteristics of the evaluation tools, 2) the FAIRness evaluation metrics, and 3) the testing results using three public datasets. 2.1
The automated evaluation tools are accessible via web applications and APIs. We extracted key features and speci cations and re ected on the transparency (in terms of documentation) and extensibility of the tools. The elements such as the availability of source code, web application, the required inputs, the quality, and interpretation of the outputs are included. 2.2
At the heart of automated FAIRness evaluation are programs that examine data
resources for the presence and quality of particular characteristics. F-UJI
implemented FAIRsFAIR Data Object Assessment Metrics [
The last comparison focuses on the representation and interpretation of the
evaluation results from F-UJI and the FAIR Evaluator. Three tested datasets in
Table 1 are from PANGAEA [
Name GeoData CORD-19 NL-Covid-19 RIVM
Host
Input for the assessment tools Input type
This section presents the results and analysis of comparing three evaluation tools. Comparison of the characteristics of the tools was performed with the FAIR Evaluator, FAIR Checker, and F-UJI, whereas a comparison of evaluation metrics was performed only with the FAIR Evaluator and F-UJI, as the FAIR Checker applies the same evaluation metrics as the FAIR Evaluator. 3.1
As table 2 shows, all tools are implemented as a standalone web application and API. Execution of the FAIRness evaluation is as follows: F-UJI requests a persistent identi er (PID) of the data or the URL of the dataset's landing page as input, while the FAIR Evaluator requests a global unique identi er (GUID) of the metadata. The following schemes are considered as PIDs by both tools: Handle, Persistent Uniform Resource Locator, Archival Resource Key, Permanent identi er for Web applications, and Digital Object Identi er. Both o er short descriptions about the input, while the FAIR Checker simply requests a URL or DOI without further explanation.
After the execution of the evaluation, each application presents the results di erently. The FAIR Checker starts with a radar chart outlining the FAIRness scores along 5 axes (Findable, Accessible, Interoperable, Reusable, Total). The FAIR Checker does not provide detailed logs except the error messages. The FAIR evaluator presents the results of metric tests with the detailed applicationlevel logs. The results are assigned with PIDs and stored in a persistent database where users can search, access, and download as a JSON-LD le. The F-UJI also provides application-level logs as feedback to the rationality of the test results. However, the logs are not as detailed as the FAIR Evaluator. The results from F-UJI can be downloaded as a JSON le. F-UJI and the FAIR Evaluator are both based on APIs to make their FAIRness evaluation services accessible.
F-UJI
FAIR Evaluator
FAIR Checker
Web application [www.f-uji.net](v1.3.5b) [w3id.org/AmIFAIR](v0.3.1) [mfaaitri-qchueec.fkre]r(.vfr0a.n1c)e-bioinfor
Requested input PID,URL of dataset GUID of the metadata URL,DOI
Results export JSON JSON-LD Not available
Output Application-level logs Application-level logs Error logs
Metrics [
F-UJI has two metric tests on data and three tests on metadata to assess the ndability, while the FAIR Evaluator has six tests on metadata. The FAIR Evaluator requires PID for both metadata and data, while F-UJI only requires for the data. Two tools both check if the metadata is structured using JSON-LD or RDFa. However, the FAIR Evaluator requires metadata to be grounded in shared vocabularies using a resolvable namespace. F-UJI checks the prede ned core elements in the metadata, such as title, description. and license.
Two tools evaluate the accessibility by assessing communication protocols
for retrieving (meta)data, ensuring the (meta)data can be accessed through a
standard protocol. The FAIR Evaluator requires authentication implementation
on the data and authorizations on metadata, while F-UJI only requires metadata
authorizations. The metadata persistence is discussed by both tools, but F-UJI
does not implement it in their tool. The argument is that programmatic
evaluation of the metadata preservation can only be tested if the object is deleted or
replaced [
To evaluate the interoperability, the FAIR Evaluator tests whether the metadata and data are structured and represented using ontology terms. F-UJI only focuses on the structure of metadata. Compared to F-UJI, the FAIR Evaluator has extensive measurements on both metadata and data to evaluate the interoperability. In the evaluation of reusability, F-UJI has more comprehensive measurements than the FAIR Evaluator. The FAIR Evaluator checks if license information is included in the metadata. By contrast, F-UJI setup four tests for metadata and one test for data to check the richness, licenses, and provenance of metadata and applied community-standards in metadata and data. 3.3
The evaluation results of three datasets are shown in Table 4. The full results are accessible on https://doi.org/10.5281/zenodo.5539823. Geodata scored perfect on all the metrics from F-UJI, but 17 out of 22 from the FAIR Evaluator. 4 out of 5 failed tests in the FAIR Evaluator assessed aspects that are not listed in F-UJI. The test on the persistence of the data identi er (F1-01D, F1-02D, MI F1B) had di erent results from F-UJI and the FAIR Evaluator. Additionally, if quali ed outward references in metadata (I3-01M, MI I3A) and licenses in metadata (R1.1-01M, MI R1.1) also had di erent results from two evaluators on the tested datasets. These di erences are examined further in the Discussion. F1-01D F1-02D F4-01M A1-01M
MI F1B MI F3
MI F4 I1-02M I3-01M MI I2B MI I3A
GeoData CORD-19 NL-Covid-19 F-UJI FE F-UJI FE F-UJI FE 33 7 73 7 37 7 - 3 - 7 - 7 3 3 3 3 7 7 3
CORD-19 failed 4 tests in F-UJI and 9 tests in the FAIR Evaluator mostly in the evaluation of the I and R. The poor quality of metadata of CORD-19 causes further failures in the other tests in both evaluation tools such as the persistence of the metadata identi er (F1-02D), metadata includes license (MI R1.1). NLCovid-19 had the lower FAIRness score from F-UJI among the three datasets (11 out of 16) and 13 out of 22 in the FAIR Evaluator. It has the same issue of the quality of metadata as the second dataset, but outperformed in the knowledge representation in data. Neither F-UJI nor the FAIR Evaluator detected the license information in the metadata of NL-Covid-19, but the metadata clearly indicates NL-Covid-19 comply with a valid license. 4
This study compares three automated FAIRness evaluation tools on the characteristics of the tools, the evaluations metrics and metric tests, and the results of evaluating 3 datasets. The outstanding feature of the FAIR Evaluator is the community-driven framework that can be readily customized, by creating and publishing an individual or collection of Maturity Indicators (MIs) to meet the domain-related and community-de ned requirements of being FAIR. The MIs and metric tests that are registered by one community are discovered and can be grouped to maximize the reusability across communities. All published MIs and conducted FAIRness evaluations are stored in a persistent database and can be browsed and accessed by the public. F-UJI visualizes the evaluation results and represents the output with better aesthetics. The source code is publicly available in Python, and well-structured for each metric test. The FAIR Checker uses the FAIR Evaluator API to perform the resource assessment, and has a more aesthetic presentation including recommendations to the failed tests, but does not allow the selection of particular metrics tests or collections, and does not o er the detailed output. 4.1
All the evaluation tools su er from some aspect of clarity and transparency. F-UJI's source code is open and each evaluation metric is described in an accompanying article. However, without technical speci cations of the application functioning, it is challenging to scan the whole code repository to learn how each metric was technically implemented. It is unclear what properties are assessed and how to improve the FAIRness of the objects. F-UJI gives a FAIRness score and a maturity score to the digital objects based on the metric tests. But it is lacks of description of how these tests are scored and how the scores are operated.
The FAIR Evaluator published its MIs and metric tests in a public Git repository. The web application of the FAIR Evaluator presents detailed log messages which potentially indicate what has been tested and what caused the test failure. However, the users still su er from the insu cient transparency of the implementation. The FAIR Checker only generates the nal test results (pass or not pass) without further explanations. 4.2
In the comparison of the evaluation metrics, F-UJI has comprehensive metrics for Reusability, while the FAIR Evaluator focuses on the Interoperability. The evaluation results from three datasets reveal more signi cant di erences between F-UJI and the FAIR Evaluator which result in con icting results for the same metric. We summarize the following three key reasons.
data, F-UJI recognizes the DOI (10.1594/PANGAEA.908011) as the data identi er. F-UJI considers DOI as a persistent identi er (PID) and determines that Geodata has a valid PID for the data. However, the FAIR Evaluator de ned the DOI as the identi er for the metadata instead of the data. The data download URL is recognized as the data identi er by the FAIR Evaluator. Thus, F-UJI and the FAIR Evaluator have di erent understanding and de nitions of data and metadata identi ers, which result in di ering test results.
Evaluator gave con icting results in determining whether metadata contained license information in CORD-19. F-UJI reported that license information was found, while the FAIR Evaluator did not recognize the license. From the output logs, two tools were both able to capture \Other (speci ed in description)" as the license information in the metadata. However, the FAIR Evaluator failed the \metadata contains licenses" test because the FAIR Evaluator requires a valid value of a license property (i.e. a URL). F-UJI passed the test but the given information for the license property is not recognized as a valid license.
When evaluating NL-Covid-19, F-UJI and the FAIR Evaluator both failed the test on \metadata contains licenses". However, the license information is clearly included in the metadata of NL-Covid-19 (RDF format) with two statements. F-UJI is unable to nd the license predicate in the metadata, while the FAIR Evaluator found the license predicate but only processed the rst statement \Geen beperkingen" as an invalid license. Unfortunately, the FAIR Evaluator did not continue to process the second statement which contains the valid license information. In this case, neither F-UJI nor the FAIR Evaluator are able to nd the valid licenses in the metadata of NL-Covid-19.
uator both examine whether the relationships within (meta)data between local and third-party data are explicitly indicated in the metadata (I2-01M, MI I3A). In the evaluation of NL-Covid-19, the FAIR Evaluator passed the test by discovering 26 out of 45 triples in the linked metadata pointed to resources that are hosted by a third party. F-UJI did not pass this test because it could not exact any related resources from the metadata. The con icting test outcome results from the di erent implementation of recognizing the relationship between the local and third-party data. F-UJI requires the relationship properties that specify the relation between data and its related entities have to be explicit in the metadata and use pre-de ned metadata schemas (e.g., \RelatedIdenti er" and \RelationType" in DataCite Metadata Schema). Compared to F-UJI, the FAIR Evaluator has a broader requirement for acceptable quali ed relationship properties by including numerous ontologies which include richer relationships. 4.3
This study has several limitations. The comparison of evaluation metrics between F-UJI and the FAIR Evaluator is based on the description of each metric, metric tests, and log messages. We did not conduct a detailed examination of their implementation. The FAIR Evaluator published technical speci cations for each Maturity Indicator and its metric tests as well as the source code of implementation. F-UJI shares its source code and descriptions of the metrics in an article. However, metric tests and their implementation have not been su ciently discussed. A possible solution for comparing the evaluation tools on the implementation level is to scan their entire source code. However, this will require an extensive e ort by experts in both Ruby and Python to conduct this task.
The discovery of the evaluation results from the three tools is possibly limited by our selection of the datasets. To increase the objectiveness of the evaluation, more representative datasets from various data repositories are required to test the di erent evaluation tools. A potential solution could be to construct a framework that evaluates and compares the FAIRness evaluation tools in an automatic and systematic manner. The framework executes the evaluation tools on a set of standard benchmarking datasets, examines what properties are being tested, and generates evaluation results automatically. This automated evaluation framework will overcome the qualitative nature of the current study and the shortcomings of requiring substantial manual e ort and proning to the errors. Finally, the evaluation tools in this study are all under active development. The evaluation metrics and implementations of metric tests in these tools can probably be changed over time. 5
This study conducted a comprehensive comparison among three automated FAIRness evaluation tools (F-UJI, the FAIR Evaluator, and the FAIR checker) covering the tool characteristics, evaluation metrics and metric tests, and evaluation results of three public datasets. Our work revealed di erences among the tools and o ers insights into how these may lead to di erent evaluation results. Finally, we presented the common issues shared by all FAIRness evaluation tools and discussed the advantages and limitations of each tool. We note the tools are under active development and are subject to change. Future work could focus on standardized benchmarks to critically evaluate the functioning of these and future FAIRness evaluation tools.