In recent years there has been a growing interest in developing techniques to efectively extract knowledge from biomedical textual documents. Many solutions rely on Named Entity Recognition and Linking (NER+L) which consists in detecting entities in text and disambiguating them through the use of knowledge bases. Despite its potential, applying this approach to the biomedical domain is limited by the lack of large annotated corpora useful to train Machine Learning (ML) models. Nowadays, it is dificult to find large sets of annotated data covering a wide range of biomedical sub-domains: the creation of annotated corpora in fact, is an expensive and time-consuming task usually performed by experts. To address this problem and ease and speed up the annotation process, we propose MedTAG, a web-based, collaborative, customizable annotation tool for biomedical documents; it is platform-independent and it provides a straightforward installation procedure.
The availability of biomedical data stored in electronic form faced an exponential growth over
the last decade. As a consequence, the need to preserve and curate biomedical data is growing
[
Extracting knowledge and relevant information from medical data is not a trivial task. First,
the largest part of biomedical data are stored in an unstructured textual format not easily
machine-readable. Second, biomedical documents have plenty of abbreviations, symbols and
terms that need to be disambiguated [
A common limitation related to the application of NER+L techniques to the biomedical
domain is the lack of large annotated corpora needed to evaluate systems and training ML
models. Creating large annotated corpora, able to cover several biomedical sub-domains, is a
complex and demanding task which requires the supervision of experts [
To address these problems, we present MedTAG, a web-based, collaborative, customizable annotation tool for biomedical literature. MedTAG key features are the following. (i) Inter Annotator Agreement (IAA) based on majority vote; (ii) multilingual support: the same document can be uploaded and annotated in diferent languages; (iii) support for user roles: roles are based on the level of expertise of the users who share MedTAG; (iv) support for documentlevel annotations; (v) support for mention-level annotations; (vi) user annotation statistics; (vii) documents’ annotation statistics: for each biomedical document an overview of its annotations is provided; (viii) support for automatic annotations: we rely on Semantic knowledge Extractor Tool (SKET)1 for the automatic annotation process which is currently limited to three cancer use-cases: lung, colon, cervix cancer; automatically created annotations can be manually edited by the annotators; (ix) PubMed integration; (x) collaborative facilities: users who share the same instance of MedTAG can visualize the annotation of one team mate of their choice; (xi) support for ontologies; (xii) support for multiple file formats : annotations can be downloaded in several ifle formats (JSON, CSV, BioC/XML, BioC/JSON).
The rest of the paper is organised as follows: in Section 2 we present MedTAG and we describe the main aspects of the tool; in Section 3 we compare MedTAG to other biomedical annotation tools; in Section 4 we draw the conclusions.
MedTAG is a web-based annotation tool distributed as a Docker container. Docker ensures portability, code isolation, dependencies packaging and a fast installation procedure: MedTAG installation can be launched with a single command and takes a few minutes to complete. The source code and the documentation are publicly available at: https://github.com/MedTAG/ medtag-core.
We refer to the Github repository cited above and to [
1https://github.com/ExaNLP/sket
Configuration. MedTAG was designed to be easily configurable via a dedicated web-interface. At the moment of the configuration, the user is asked to provide the collection of clinical documents (or reports) to be annotated, the labels needed to perform document-level annotation and the concepts belonging to one or more ontologies. All the files must be uploaded in CSV format. In addition, it is possible to specify the parts of the uploaded reports where annotation is allowed. The file containing clinical reports can be replaced by (or provided with) a CSV containing a list of PubMed articles’ identifiers. MedTAG automatically extracts the identifiers and downloads the title, the abstract and some additional information of each article. Annotation is allowed on title and abstract sections. MedTAG supports the upload of new batches of reports, new sets of labels and concepts at will, without losing the existing annotations. In order to make the configuration easier and faster, MedTAG ofers downloadable CSV templates and provides the user with information about what changes need to be applied to the uploaded CSVs to make them comply with the required format.
User interface. In Figure 1, the MedTAG user interface is illustrated. The user interface we propose, its layout and components have been discussed with physicians and experts in the digital pathology domain.
We organized the MedTAG user interface into three main sections: Settings and Download (A in Figure 1), Report (B in Figure 1) and Annotation (C in Figure 1).
The Settings and Download section, placed at the top of the web page, allows the user to change the current parameters’ setting; it is possible to change in this order: (i) the clinical case, (ii) the language of the clinical reports, (iii) the institute (or hospital) where the reports have been produced and (iv) the annotation mode (it can be set either to Manual or Automatic if the reports have been automatically annotated). The button Download allows the user to download the annotations in JSON, CSV, BioC/XML and BioC/JSON file formats. The leftmost button allows the user to navigate to other web-pages. The rightmost button instead, allows the user to logout.
The Report section contains the textual report to annotate and some other relevant information such as the date of the last annotation performed by the user, the report’s identifier and the report’s translations (available if the report has been uploaded in diferent languages). The Reports navigation panel (B.1 in Figure 1) allows the users to navigate between reports. Clicking on the next and previous buttons, or using the keyboard arrows, it is possible to move to the next or previous reports. To skip to a specific report, the drop-down list allows the user not only to search for the desired report from its identifier, but also to check what reports have not been annotated yet (they are marked in boldface).
The Annotation section contains the annotation created by the user for the clinical report examined. MedTAG ofers four annotation types: Labels, Concepts, Mentions and Linking. The ifrst two are document-level annotations: in this case the annotation refers to the entire textual content. The latter two are mention-level annotations: in this case the annotation refers to one or more text spans (mentions) in the original textual content. Labels annotation consists in associating one or more labels to the textual report; a label is a property that describes the report. Concepts annotation consists in associating one or more ontological concepts to the report. Mentions annotation consists in identifying entity mentions. In order to identify a mention, the user can either click on the first and on the last words composing the mention, or click on every single adjacent word of the mention in the textual report on the left. The identified mentions are characterized by diferent colors so to be immediately detected. In Figure 1 the annotated mentions are highlighted in blue, yellow and green respectively. Linking consists in associating one or more ontological concepts to a mention. Ontological concepts in Linking and Concepts annotation can be added via a drop-down menu with auto-completion facilities, this allows the user to type the desired concept without examining all the ontological concepts. In Figure 1, each mention has been linked to its corresponding concept. Each annotation can be either saved or deleted by clicking on Save and Clear button respectively. Nevertheless, the annotation is automatically saved when the report or the annotation type change. Annotators can benetfi from annotations created by other users they share MedTAG with. The Annotation Panel (C.1 in Figure 1) provides the annotation created by the logged in user (left-side button), the automatically created one (button in the middle, if the automatic annotation is not available this button is disabled) and the one performed by another user of choice (right-side button). The automatic annotation of the report and the one of the team mate are read-only.
Firstly, we evaluated MedTAG from a qualitative perspective, which allowed us to compare its functionalities with those of other seven annotation tools exploited in the biomedical domain. Then, we evaluated MedTAG from a quantitative perspective which allowed us to study the performances of MedTAG compared to those of other four biomedical annotation tools.
In the qualitative analysis we compared MedTAG to: BioQRator [
MedTAG turned out to satisfy almost all the selected criteria (20 criteria over 22): the annotation of overlapping mentions and the relationships annotation are the only unsatisfied criteria.
In the quantitative analysis instead, we compared MedTAG performances to those of other
four web-based publicly available annotation tools: MyMiner [
The lack of large annotated corpora hinders the development of NER+L techniques based on ML. Creating annotated corpora is an expensive demanding task performed by experts in the biomedical domain. In order to ease and speed up the annotation process we presented MedTAG, a dockerized, web-based biomedical annotation tool.
Thanks to its Docker distribution, MedTAG is portable and it is easy to be installed. MedTAG is customizable with documents, labels and concepts, not necessarily tied to a single ontology. The PubMed integration allows the user to upload a list of PubMed articles’ identifiers whose main information are automatically downloaded. MedTAG provides four diferent annotation types: two of them are document-level annotation types, the others are mention-level annotation types. The annotations of each type can be downloaded in four diferent file formats: CSV, JSON, BioC/XML and BioC/JSON. MedTAG supports collaborative annotations: users can visualize other members’ annotations and copy them in their own profile.
In order to improve MedTAG, we plan to add the support for relationships annotation and overlapping mentions. Moreover, we plan to extend the use-cases available for the automatic annotation.
2https://github.com/MedTAG/medtag-core#medtag-benchmark 3https://www.selenium.dev/
This work was supported by the ExaMode Project, as a part of the European Union Horizon 2020 Program under grant 825292.
The authors wish to thank Stefano Marchesin for the work on SKET which has been integrated in this work. [14] S. M. Yimam, I. Gurevych, R. E. de Castilho, C. Biemann, Webanno: A flexible, web-based and visually supported system for distributed annotations, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2013, pp. 1–6. [15] S. Dobbie, H. Straford, W. O. Pickrell, B. Fonferko-Shadrach, C. Jones, A. Akbari, S. Thompson, A. Lacey, Markup: A web-based annotation tool powered by active learning, Frontiers in Digital Health 3 (2021). [16] M. Neves, J. Ševa, An extensive review of tools for manual annotation of documents, Briefings in bioinformatics 22 (2021) 146–163.