Today's exploitation of data shapes how we all live and function [1]. A growing number of electronics around us and on the Internet are allowing for the enormous growth of data [2]. International Data Center (IDC) predicts that by 2025 the total data produced would rise from 33 Zettabytes (ZB) in 2018 to 175 ZB [1]. If we look at planes, they produce around 2.5 billion Terabyte of data from the sensors mounted in the engines, per year [2]. It's like everywhere we turn to, we are surrounded by data.
As all things on the Internet, we would like the data to be easily discovered and consumed by users, just like websites on the Web. Therefore, research communities around the world have gathered to draft principles to improve the consumption of data on the Internet, thus the FAIR principles were born. FAIR is an abbreviation of the words Findable, Accessible, Interoperable and Reusable. The FAIR principles are a guide on achieving FAIR data, not a set of rules to follow. With that being said, FAIR is not a standard, it does not define the how-to's, it is completely open to different interpretations and should not be used to assess the quality of data [3]. Instead, FAIR are guidelines for better experience with the data on the Internet, for both humans and machines. First formulation of the FAIR data vision was actually in 2014 and the primary goal was to optimise data sharing and reuse by humans and machines. In 2016, this initiative resulted in the first formal publication of FAIR principles, "The FAIR Guiding Principles for scientific data management and stewardship" by Wilkinson et al [4].
"Findable" yields ways for others to easily discover the data, e.g. using public repositories for data storage and assigning Data Object Identifiers (DOIs) for citation. "Accessible" allows for maximum availability of the data i.e. designs the access (and restrictions) to the data using the Internet protocols (FTP, HTTPS). "Interoperable" will make sure that the data is easily integrated with other data and can easily be consumed by both machines and humans, meaning multiple file formats for the machines and usage of widely used languages for the humans. "Reusable" provides for easier usage and understanding of the data by other researchers by requesting metadata and documentation [5]. One common misconception about the FAIR principles is that: "FAIR data means open data". The FAIR principles allow (and encourage) licences, which can restrict the access to the data.
Open Access Infrastructure for Research in Europe (OpenAIRE) [6] which is one of the leading projects for open science in Europe and infrastructure for open scholarly and scientific communication, also promotes and supports FAIR principles. According to OpenAIRE, FAIR principles describe the organization of the research outputs so they can be more easily accessed, understood, exchanged and reused. In more details, Findable requires data with persistent identifier, rich metadata, searchable and discoverable online; Accesible means data retrievable online using standardised protocols and restrictions if necessary; Interoperable recommends using common formats and standards and controlled vocabularies and Reusable imposes well-documented data with clear licence and provenance information (Fig. 1). In this paper, we aim to provide insights at specific tools for FAIR data assessment and analyze their capabilities. In Section 2 we cover some related work. Section 3 provides descriptions about fair assessment tools characteristics, with the exception of the last subsection, which draws parallels between them based on multiple relevant characteristics. In Section 4 we evaluate open data sets using the tools we addressed in the previous section. At last, we draw the conclusion of this paper by revealing the advantages and disadvantages of the tools, with respect to their "FAIRness" evaluation and characteristics.
The FAIR data principles requirements and evaluation are relatively new topics that started to be researched in 2016 and have been rapidly increasing. With the need data to be easily discovered and consumed by users, researches have begun to access and evaluate data FAIRness.
Camilla Hertil Lindelöw et al. [7] talk about the Swedish Government allocating parallel assignments to the Swedish National Library in order to develop criteria and a mechanism for assessing how well research data and scholarly publications created in Swedish organizations comply with the FAIR principles. The discussion describes recommendations and the possibilities and setbacks that they have identified during the work, focusing primarily on evaluation at a national level. Thompson at al. [8] describe tools that aid the FAIR process, from FAIR data management planning to FAIR data creation, publication, evaluation and (re)use, revealing that there are a lot of ongoing efforts that contribute to the goal of making FAIR a reality. In [9], the authors describe a FAIR framework and execute compliance tests with the FAIR metrics. They demonstrate its usage in some commonly used repositories and provide feedback where semi-automated evaluations are performed. It's revealed that the distinction between manual and automatic assessment shows that automatic assessments are usually more rigorous, resulting in lower FAIRness scores, though more precise. In [10], the authors briefly outline the different kinds of FAIRness evaluations describing the pros and cons of each kind and provide guidelines on how FAIRness evaluations can be used and interpreted. They define discrete-answer questionnaire-based evaluations, open-answer questionnaire-based evaluations and semi-automated evaluation, concluding that evaluations should be assessed not at the overall FAIRness level, but at the maturity indicator level. Bishop et al. [11] explore how the FAIR principles can be measured for re-use from a consumer perspective, stating that some FAIR principles can be subjectively automated more than others and that requires more qualitative, subjective measures for automation. They provide recommendations to create context aware questionnaires to evaluate the FAIR principles in a way that captures the FAIR-ness from the perspective of data re-user/consumer. Mons et al. [12] discuss how the data will stay FAIR in the ever emerging cloud world. They explore the public and commercial domain of the cloud sphere and what they are willing to offer for open and FAIR data across cloud platforms. The FAIR principles are revised from the perspective of the European Open Science Cloud. Madduri et al. [13] examine tools designed to help implement complex "big data" computations in ways that allow the code and associated data to be FAIR. To highlight the usage of the tools, the authors present a case study on the implementation of a multi-stage DNase Hypersensitivity (DHSs) sequencing data analysis that retrieves massive data sets from a public repository and uses a combination of parallel cloud and workstation computing to identify binding sites of candidate transcription factors.
In this Section we provide descriptions and comparison of multiple tools for data FAIRness evaluation.
The Australian Research Data Commons (ARDC) is an organization that has been an active advocate for the adaptation of the FAIR principles. It provides multiple useful resources to the international research community to ensure the usage of best practices in their research, one of them being a FAIR data self assessment tool which provides a score describing the "FAIRness" of the data. It is important to note that the tool is made with the interpretation of the FAIR principles and it is made to trigger thinking and discussion around potential approaches of making data more FAIR [14]. The tool is hosted on the web site of ARDC [15]. It is designed in a form of a survey with 12 questions which are formulated to quantify the intensity of each FAIR category. It has 4 sections, titled by the four FAIR principles. There are single choice questions and questions in Yes/No format. The answers are evaluated based on previous answers for consistency. Every question essentially traces back to a certain FAIR principle. By answering a question, the bar on the bottom of the corresponding section fills up depending on the "strength" of the provided answer, i.e. it's measuring the presence of a FAIR principle. On the bottom (and top) the "Total across F.A.I.R" bar also fills up depending on the section score bars, representing the total "FAIRness" score of the data. The tool captures the FAIR essence. It addresses the FAIR principles in great detail and provides additional information on "FAIR terms". It is very concise and to the point, self-explanatory and has a nice design and layout. It does not disclose the algorithm behind the scoring system, but the pattern is easily noticeable after a few tries. It is easily accessible and needs no form of log-in.
Data Archiving and Networked Services (DANS) urges researchers to make their digital research data and related outputs FAIR [16]. To assist the process, a FAIR data self-assessment tool called SATIFYD (Self-Assessment Tool to Improve the FAIRness of Your Dataset) was created. It is intended primarily to evaluate datasets that will be published on EASY, which is an online archiving system for depositing and reusing research data [17]. SATIFYD is in the form of a 12question questionnaire divided into 4 sections, each section having questions centered around a certain FAIR principle. Some of the addressing aspects of the FAIR principles are guaranteed by EASY. For example, DOI requirement is omitted from the questions, thus meaning that SATIFYD relies on the repositories's ability to ensure some of the principles. Some questions are directly linked to services offered by EASY, and there is no work-around, meaning that the overall FAIR score would suffer if the data is not published on EASY. Also, some questions are repeated in different sections, since they capture the nature of several FAIR principles [16]. If the score is not perfect (100%), it offers guidance on how to lift it with pressing the "Want to improve?" button on that section. The consistency of the answers is tested by referring to previous question's answers and if anything does not add up, a pop-up appears to expose the inconsistencies. It does not disclose the algorithm behind the scoring system. This tool also includes a feature for getting printed reports on the answers, together with the tips on improving.
CSIRO's (Australia's Commonwealth Science and Industrial Research Organization) initiative OzNome develops tools and methods aimed at providing access to self-organizing, reliable, organized, and well-governed data ecosystems [18]. In that spirit, they developed a 5 star FAIR data self assessment tool, which is publicly hosted on their website [19]. The tool embodies the four FAIR categories, plus adding "Trusted" to the bunch, which is trying to determine if the data keeps records on how it's been used, by whom, and how many times. The tool is in the form of a survey with questions each yielding single-choice answers. Every FAIR principle is captured by a series of different questions which are designed to allow data rating according to its current state. The end result is a 5 star rating for each category for the data, showing the compliance to the 5 categories. The number of stars from the respective categories are adjusting with every answered question. The survey kicks off with the requirement for information about the name (title) and URL of the data, for which you can provide false information if none other information is available at the moment. These fields don't affect the score. The consistency of the answers is not checked and it does not provide any additional explanatory information on the site about the concepts covered by the tool. On the other hand, it discloses the rating scheme [20].
The European Association of Databases for Education and Training (EUDAT) Collaborative Data Infrastructure (CDI) project holds an infrastructure for integrated data services and resources supporting research in Europe [21]. The CDI provides a common infrastructure that enables data management across European research communities, allowing researchers from any research discipline to preserve, find, access and process data in a trusted environment. The EUDAT checklist was developed in order to help the researchers to test the "FAIRness" of their data [22]. The checklist is not a self-assessment tool, but rather a handy reference sheet that can be printed out for a quick check on the FAIRness of the data. It contains 4 sections, each yielding a brief summary of a FAIR principle and 4 statements associated with it. For the purpose of this paper, the final score shall be calculated as the percentage of the checked off answers. The questionnaire is available in two versions, one as a more simple checklist, the other as a data flyer with better design and some colors. This checklist is also supported by the OpenAIRE project [6].
Research Data Alliance (RDA) [23] is composed of multiple organizations from around the world with the purpose of strengthening the social and technological bridges to enable open data sharing and reuse. To stimulate data sharing, an interdisciplinary scientific interest group called SHAring Reward & Credit (SHARC) was set up by RDA. The primary aim of SHARC is to find ways to promote the value of the data sharing process and find ways of crediting those who comply. With these priorities in mind, SHARC developed a simplified evaluation grid which employs criteria against researcher's data to determine the presence of the FAIR principles [24]. The simplified grid is intended for use by researchers who produce and/or use data. It includes 4 sections representing the FAIR principles. Each section contains several questions, which for the purpose of this paper will be answered only by YES or NO. The questions from the sections are in the form of a decision tree, indicating that there are preconditions for some questions. For example, to answer: "Unique, global, persistent ID?", first the data has to be indexed, i.e. the question "Indexed identifier?" must be answered with "YES". For the purpose of this paper, the final score is the percentage of "YES" answered questions. The results of the grid are to be used for appreciating the researcher's practice and to spark discussion to keep the data management life cycle more "FAIR", but not for comprehensive data FAIRness assessment.
After a detailed study of the tools/checklists described in Subsection 3.1, we selected eight features (criteria) to make a comparison: 1.Automated evaluation -Does not require manpower for calculating the final score; 2.Disclosing the rating system -The algorithm behind the calculation of the final points is available to the user; 3.Additional explanatory information -Provides all the needed info about the FAIR terms mentioned in the questions; 4.Bound to a repository -It relies on features made available by a certain repository to impose certain FAIR principles; 5.Guidance to improve "FAIRness" -Provides tips and information on how to comply more heavily to the FAIR principles; 6.Printed report -Provides a printed report of all the answers; 7.Goes beyond FAIR -The questions cover principles outside of FAIR; 8.Checklist -The questions are only in a Yes/No format.
Table 1 presents the comparison of FAIR data assessment tools. First column denotes the tool name, while first row contains the eight features as explained in Subsection 3.2.1. The symbol "✓" represents that the tool "posses" the selected feature. For example, the tool ARDC has Additional explanatory information (feature 3).
Based on the tools analysis we provide the following recommendations:
15:6
• If EASY is not the chosen repository where the data will be published then SATIFYD tool should be avoided, because the overall score suffers greatly (as we will see in the next Section); • The checklists are more vague on how the FAIR principles are truly obtained. On the other hand, the non-checklists provide a palette of specific answers based on FAIR principles implementations, which can affect the final score in different ways; • ARDC and SATIFYD provide an abundance of additional information about the FAIR terms and specifically cite their sources, so these tools would be great for researchers that are not that familiar with the FAIR principles or don't know where to look for information on them; • CSIRO, EUDAT and RDA disclose their algorithms for researchers and they can give feedback to the creators and build their own tools on top of these algorithms; • If EASY is the repository of choice, then SATIFYD is definitely the right choice to assess the FAIRness. It provides unique features that other tools lack, e.g. guidance for improvement.
On the other hand, if the data is not/will not be published on EASY, then ARDC or CSIRO should be the go-tos if the researcher is not in need of a printed report, otherwise the checklists would do the job.
The evaluation of the data FAIRness was performed on seven open datasets chosen from the following repositories: PANGAEA -Open Access library aimed at archiving, publishing and distributing georeferenced data from earth system research and has been selected as one of the 6 data repositories that provides their expertise in testing practical solutions to enhance the FAIRness of data [25]; PhysioNet -repository of freely-available medical research data, managed by the MIT Laboratory for Computational Physiology [26]; DRYAD -international open-access repository of research data, especially data underlying scientific and medical publications. Dryad is a resource that makes research data discoverable, freely reusable, and citable [27]; EASY -online archiving system of the creators of the tool for FAIRness data evaluation SATIFYD, Data Archiving and Networked Services (DANS). EASY offers access to thousands of datasets in the humanities, the social sciences and other disciplines [17]; Dataverse -an open source data repository framework used by individual researchers, archives, academic institutions and publishers around the world to share, find, cite, and preserve research data [28]; datagovmk -contains datasets from different institutions from North Macedonia [29]; and Arctic Data Center -primary data and software repository for the Arctic section of NSF Polar Programs which is said to be in "large degree already compliant with the FAIR principles" [30]. Table 2 presents the metadata for the seven datasets: Dataset name, Repository, Short Desrciption, Dataset Size, Creators and Year of publishing.
We evaluated the seven datasets using each of the five tools/checklists (ARDC, SATIFYD, CSIRO, EUDAT checklist and RDA checklist). The evaluation results for the four FAIR principles are presented in Table 3. F stands for findable, A for accessible, I for interoperable, R for reusable same problem as EUDAT: vagueness, but with the opposite effect. Some of the questions simply cannot be answered and thereby left unchecked, which decreases the overall score. For example: "Do the data reuse control and data sharing arrangements meet the data protection and "local/national ethics requirements?". This question requires knowledge from the law/moral standards, but it is not clear whose law/moral standards should be examined (Which country's?).
The dataset is open and available for anyone in the world, thus this question becomes hard to answer (especially in YES/NO format).
On the non-manual side, it's observable the ARDC is the most "generous one". This happens because ARDC is a little loose on the "R" side of FAIR, compared to SATIFYD and CSIRO, who are more strict and demand the implementations of several standards to increase the reusability of the dataset.
SATIFYD scores are lower because of the "EASY nature" of the tool. For example: it demands file formats that are specifically preffered by EASY and if those are not provided, the score suffers, even though these "non-preferred" formats are widely used and standardized.
In 2016 a group of researchers published a measurable set of principles for the academia and industry known as FAIR Data Principles. The primary intent was to develop a guideline for researches who want to enhance the reusability of their data. However, these principles set up a lot of challenges that need to be addressed. To understand the current research in the field of FAIR principles, we have analyzed the provided FAIR evaluation options of five tools and identified main characteristics and differences. We also perform FAIR evaluation of seven datasets from different data repositories using the five tools. The findings have shown that this topic is still in its early phase. The results show that the FAIR requirements are still not unified. For some datasets, we obtained scores that differ by more than 30%. It indicates that tools are somehow designed for a specific data repository. We can conclude that there is still no agreement for the universally accepted requirements for data FAIRness evaluation tools. This study contributes to the theory by analyzing the evaluation options and requirements for data FAIRness and by providing a guidance for tool selection and improvement.
This work was supported in part by the European Union's Horizon 2020 research and innovation programme, project National Initiatives for Open Science -Europe, NI4OS-Europe, [857645] and by the Faculty of Computer Science and Engineering, Skopje, North Macedonia.
(A. Mishev) https://www.finki.ukim.mk/en/content/bojana-koteska-phd (B. Koteska); https://www.finki.ukim.mk/en/staff/anastas-mishev (A. Mishev) 0000-0001-6118-9044 (B. Koteska
and FAIR is the total score obtained as average value of the four principles. For the CSIRO tool there is only result for the total FAIR score because this tool provides visual 5-star output for each principle. The general overview shows that the manual checklist EUDAT gives the highest scores, which is a direct consequence of the vagueness of the questions in the questionnaire, thus shunning away from testing for concrete solutions/standards for certain principles.
On the other hand, RDA gives relatively lower scores than EUDAT, even though it has the