An experimental protocol describes a sequence of tasks executed to perform experimental research in biological and biomedical areas, e.g. genetics, immunology, neuroscience and virology. Such experimental protocols indicate, for each step, exactly how it should be executed, often including equipment, reagents, descriptions of critical steps, troubleshooting instructions, other kinds of tips, as well as any other information that researchers deem important for facilitating the reusability of the protocol. These protocols therefore have a clear systematic structure, but when published they are treated like any other scientific publication i.e. as a narrative text in HTML or PDF format. The formal structure is therefore not easily accessible and cannot be reused. This paper addresses this problem by extracting, representing and publishing steps from experimental protocols to make them Findable, Accessible, Interoperable, and Reusable (FAIR). Our work builds upon human annotations in combination with Named Entity Recognition delivering nanopublications. Our software toolkit, Nanotate, is based on a flexible web based annotation environment, namely Hypothes.is, the BioPortal NER web services and the nanopublications infrastructure. Our evaluation shows that our approach is viable and our tool user-friendly.
Experimental protocols are documents providing detailed descriptions of the processes by means of which results, often data, are generated in biological sciences. For reproducibility purposes, both data and protocols describing the steps followed to obtain the data, should be available. The protocols often include equipment, reagents, critical steps, troubleshooting guidelines, tips and other information that facilitates reusability. Experimental protocols are described in natural language lacking of a formal structure; it is not surprising that important details are sometimes missing, e.g. time or temperature to centrifuge a sample, precise storage conditions of a suspension, or specific features of an equipment or reagent used.
Copyright © 2022 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
To illustrate the importance and nature of experimental protocols, let us consider “Identification of QTL Associated with Drought Tolerance in Common Bean”; it involves the execution of a number of protocols, such as sample preparation, DNA isolation and amplification of the DNA via PCR. The study as a whole can be seen as a workflow that includes several steps, each of them a protocol on its own. Each protocol consists of a sequence of structured instructions. Each of the protocols and also each of their steps have inputs and outputs. In this sense, steps within each protocol could be seen as the smallest most granular part of the workflow. Following this analogy, understanding the study as a container of protocols, both steps within protocols as well as protocols on their own should be reproducible.
When protocols are published, they are treated like any other scientific
publication. Little attention is paid to the workflow nature implicit in this kind of
document, or to the chain of provenance indicating where it comes from and how
it has changed. The protocol is understood as a text-based narrative instead of a
self-descriptive Findable Accessible Interoperable and Reusable (FAIR) [
Our approach makes use of domain expert annotations in protocols to extract information about steps, samples, instruments and reagents participating in each step. In this work, we investigate how we can facilitate the participation of the domain experts when extracting steps from experimental protocols while hiding the complexity of representing and publishing such artifacts. We developed Nanotate. Nanotate combines Named Entity Recognition and Human Based Annotation in a single annotation framework. The results suggest that our approach is practical, the semantic annotations rich, consistent between annotators and meaningful; and the interface is perceived to be usable and user-friendly. 2
Open sharing of methods is an essential element to ensure reproducible research
in life science, e.g. through repositories like BioProtocols5, protocol exchange
[
Provenance is an important aspect in experiments, and Reproduce Microscopy
Experiments (REPRODUCE-ME) [
To address the problem of extracting, representing and publishing steps from experimental protocols we developed Nanotate, a web based tool that facilitates the publication of annotated steps as nanopublications. The goal of our approach is to allow end-users to directly publish nanopublications about annotated protocol steps. Our approach is highly scalable; Nanotate extends an existing annotation framework, Hypothes.is, which is widely used, and has an extensive community of users, thus supporting developers and end users. More importantly, Hypothes.is is highly adaptable; further adaptations of the interface are possible while reusing the backend and authentication mechanisms.
Nanotate prioritizes Human based annotation in specific parts of the
scientific discourse; in this case the identification of steps in experimental protocols
and the material entities participating in each one of them (samples,
equipment, reagents). Nanotate incorporates capability for the automatic recognition
of samples, equipment and reagents with classes from 8 ontologies (OBI [
The Nanotate architecture is shown in Figure 1. The workflow architecture starts
on the protocols website, called here “annotated site”. We replaced the
hypothes.is annotator and sidebar with our own annotation user interface. We
use the BioPortal API to consult the ontologies with terminology related to the
samples, equipment and reagents to be annotated. To produce nanopublications
about individual annotated steps we use the Python libraries nanopub8 and
fairworkflows9, which allow for searching and publishing nanopublications and
support the construction of FAIR scientific workflows using nanopublications[
Generation of nanopublications (from JSON to RDF). Here is generated the publication of each annotated step and their components as nanopublications. In this process participate the component Nanopub library. To generate the nanopublications, the users just press the button “Nanopub” located in each annotated step. The nanopub library was configured to group annotations according to their text position. These are grouped, if an annotation that does not 10 https://git.io/JDFOV 11 https://github.com/judell/HelloWorldAnnotated have the tag “step” is contained in an annotation that does have a tag “step”. All annotations are sent to Nanotate via API where validation is done and subsequent publication. In addition, the published nanopublications are stored locally in a MongoDB data storage.
Generation of RDF workflows. Once having all the nanopublications for the individual steps, the next step is creating the corresponding workflow. Here, Nanotate client, the users consult the nanopublications and press the button “new workflow”. Then, users register the fields: i) label: to add a name to the workflow to be created and ii) description: short description about the new workflow. Finally, the users select the nanopublications that are part of the workflow. The resulting nanopublications are then published as above. The published nanopublications about workflows are also stored locally in a MongoDB data storage.
Nanotate is a free and open source tool. The code behind the tool is available on Github https://github.com/nanotate-tool. End-users can install it by creating a bookmark. The documentation about how to install and use the Nanotate is available at http://doi.org/10.5281/zenodo.5101941 A running instance of the tool can be found at https://nanotate.bitsfetch.com/ 4
To assess our approach we carried out a controlled annotation study with domain
experts. We evaluated the nature and consistency of the annotations and, the
subjective usability of the tool. The methodology is presented below.
Materials. We worked with i) six open access protocols in molecular biology
from Bio-protocols and Nature Protocol Exchange; See Table 1. ii) human
annotators: Three annotators with experience in laboratory techniques. iii) the
annotation tool, Nanotate: a web-based tool used in this study. During the
training sessions the participants learnt how to use the tool. iv) training
documentation: We provided the participants with a detailed training document,
how to install and use the tool. It gives examples of samples, equipment, and
reagents and how this information should be annotated. And v) a questionnaire
to evaluate a subjective usability of the Nanotate tool: A usability
questionnaire with ten standard questions from the System Usability Scale (SUS) [
Methods. Our controlled annotation has a series of activities organized in the following stages: i) training session, ii) assignment of protocols to annotators, 12 https://forms.gle/o6JYQ7xY7wVFsqqG8 # Protocol ID Source 1 DOI:10.21769/BioProtoc.323 Bio-protocols 2 DOI:10.21203/rs.2.1347/v2 Nature Protocol Exchange 3 DOI:10.1038/protex.2013.007 Nature Protocol Exchange 4 DOI:10.21203/rs.2.1645/v2 Nature Protocol Exchange 5 DOI:10.21769/BioProtoc.1751 Bio-protocols 6 DOI:10.21769/BioProtoc.1077 Bio-protocols iii) review of annotations and iv) generating data for the analysis. In the first stage, a virtual session was organised with each annotator in order to train them in the use of the Nanotate tool and give them some tips about good annotation practices. The meetings were carried out by using Google Hangouts. In the second stage, the six protocols presented in table 1 were annotated by three human annotators. Then, in the third stage, virtual meetings were scheduled with annotators in order to solve doubts and inconsistencies. Specifically, when we had to deal with nanopublications validation problems –A subset of protocol steps were annotated but the corresponding nanopublication was generated incorrectly, probably due to problems in the HTML of those protocols. Finally, the data in the form of the semantic annotations structured as nanopublication as well as the answers to the usability questionnaire were analyzed. We focused on tag distribution, ontology coverage, completeness analysis, and inter-annotator agreement.
A total of 232 part-of-speech entities were tagged with one of the six available categories (sample, equipment, reagent, input, output, step). The results of this stage are summarized in Figure 2 (the full data is available online13). In this study, 6.4% (17) of the samples were also tagged as “input”; and 5.7% (15) of the samples were also tagged as “ output” (see Figure 2). 22 parts-of-speech were just tagged as “sample”. These results indicate that the categories “input” and “ output” were the least used to tag parts-of-speech. The results indicate that the 65 steps could be identified and annotated (see Figure 2). Finally, 19.7% (52) parts-of-speech were tagged as “equipment” and 21.6% (57) parts-of-speech were tagged as “reagent” (see left-hand side of Figure 2). 5.2
Parts-of-speech tagged as “sample”, “equipment” and “reagent” were mapped to ontology terms that come from the 8 aforementioned ontologies available in 13 http://doi.org/10.5281/zenodo.5089323 sample/input sample/output
sample equipment reagent step 17 15 22 equipment
31 reagent sample 0 1 12 25 23 52 57 54 0 10 20 30 40 50 60 70 80
Number of Annotations 0 10 20 30 40 50 60 70 80
Number of Annotations BioPortal. The results of this stage are summarized in Figure 2 (right; the full data is available online14). 59.6 % (31) of the text tagged as “equipment” were mapped to ontology terms from OBI (8), BAO (20), SP (2) and ERO (1). 43.9% (25) of the text tagged as “reagent” could be mapped to ontology terms from CHEBI (17), OBI (3), SP (4) and BAO (1). 39.7% (23) of the text tagged as “sample” could be mapped to ontology terms from NCBI TAXON (3), UBERON (1), CHEBI (11), BAO (2), SP (5) and EFO (1). We manually analyzed the cases that we not mapped to an ontology term to find out about the reasons for that. We found the following four main reasons: i) terminology not represented in the available ontologies; for instance reagent manufacturer names (e.g.: TRIsol) and acronyms or short names (e,g,: PBMCs, PBS). ii) name of specific type of reagents (e.g.: extraction bu↵er, this is a bu↵er), and samples (e.g.: human venous blood, this is a blood). In this second case, just superclasses are available in the ontologies used (See yellow bar in Figure 2). These ontologies do not reach a high level of granularity. iii) Plural words (like mosquitoes, bacteria), are not represented in our subset of ontologies, and iv) Typos, for instance Micropistle instead of Micropestle. 5.3
As the above analyses focused only on the annotated entities, we should also have a look at completeness. We are looking for indications of the extent to which entities were missed. We found that first steps should always specify an input, while last steps specify an output. In the set of analyzed protocols, we found that 100% of first steps included an annotated input; 83.33%, five of the six last steps, included an annotated output. For equipment annotations, 14 http://doi.org/10.5281/zenodo.5089726 steps in such protocols almost always involved the use of some equipment. We checked how often these were correctly covered. We found that in 55,38% (36) of the steps, at least one of the equipment was not explicitly mentioned in the text. The domain experts are able to infer the use of some equipment, due to the narrative; for instance, use of experimental actions like “centrifuge”, and additional information such as temperature of centrifugation (e.g.: 4 C) help to deduce the use of a “refrigerated centrifuge” in a particular step (more examples are available in a table online15). This table include nanopublication links where inferred equipment are missing in the assertion. We can therefore conclude that the completeness is far from perfect, but the annotations cover a substantial part of the mentioned entities. 5.4
Annotators categorized the annotations into the di↵erent workflow component classes (i.e., step, reagent, equipment, input, output) with perfect agreement. However, in some cases, the spanning text of these annotations overlapped but did not match perfectly. We observed that the annotators chose the same spanning text to annotate “step” and “sample/input” (samples that were also tagged as input). Partial matches were identified where one or more annotators highlighted slightly di↵erent text. That was the case of the 9.6% of the text tagged as “equipment”, 15,4% of the text tagged as “sample”, 17,5% of the text tagged as “reagent” and 20% of the text tagged as “sample/output” (samples that were also tagged as output). As was presented in subsection 5.2, we found a low incidence in annotations linked to ontology terms. However, the annotators have a high agreement on linking annotations to the ontology terms (Fleiss’s Kappa: 0.70). The full data, including the set of annotations used to calculate inter-annotator agreement is available online16. 5.5
The table https://doi.org/10.5281/zenodo.5528946 summarizes the results
of the SUS. The participants were three annotators of the annotation study
described above and one additional expert who used the tool but did not participate
in the annotation study. Overall, the tool got a SUS score of 93.12%; between
“excellent” and “best imaginable” on the adjective scale [
This work involves manual annotation of protocols in order to extract information about samples, instruments and, reagents participating in each step. The 15 https://doi.org/10.5281/zenodo.5528934 16 http://doi.org/10.5281/zenodo.5095720 methodological aspects involved the participation of domain experts, reusing existing resources while focusing on how end users could make that valuable information from experimental protocols readily available in a semantic manner. We developed Nanotate, a tool to publish nanopublications from annotated protocol steps. It extends the Hypothesis platform by making it compliant with the nanopublication workflow. It also adds NER capabilities from BioPortal in a single annotation framework.
From this experiment we concluded that the Nanotate tool, the guidelines
about what and how to annotate, and the knowledge that comes from experts in
laboratory techniques were key in achieving a high consistency in the annotations
and the subsequent nanopublications. This is a consequence of a standardized
annotation process to publish individual protocol steps and their participants
(samples, equipment, reagents). We also found some missing elements in
protocols. For example, no inputs or outputs. Some equipment could be inferred by
experts but it could not be annotated because there was no explicit mention
to it. Such inaccuracies limit the reproducibility and reusability of protocols. It
is necessary to improve reporting structures for experimental protocols, this
requires collective e↵orts from authors, peer reviewers, editors and funding bodies
[
This work was supported by the Dutch Research Council (NWO)(No. 628.011.011).
Giraldo et al.