Ground truth creation plays a key role in Information Retrieval (IR), serving as the foundation for building test collections used to train and evaluate retrieval systems. However, this process is often time-consuming and demands considerable efort from human experts. Evaluation campaigns like TREC and CLEF highlight the scale of this challenge, requiring extensive manual annotation to ensure the accuracy and reliability of the data. To reduce this workload and support assessors more efectively, we present Doctron. Doctron is a collaborative, web-based, containerized platform designed to simplify ground truth creation in IR. It supports both text and image annotation, including features like entity tagging and linking, passages annotation, graded labeling, and object detection. The platform enables team collaboration through role-based permissions and integrates Inter Annotator Agreement (IAA) metrics to ensure annotation consistency and quality.
Creating ground truth datasets for large corpora is fundamental to Information Retrieval (IR), as they
are the foundation for training, evaluating, and enhancing search systems. Manual annotation – where
human assessors label documents – remains the standard approach, playing a key role in ensuring
the reliability and robustness of test collections that contribute to the progress in IR. In the context of
large-scale evaluation initiatives like Text REtrieval Conference (TREC) and Conference and Labs of
the Evaluation Forum (CLEF), the creation of high-quality annotated corpora is crucial for enabling
researchers to benchmark models and foster progress in the field [
However, generating such datasets is a demanding and resource-intensive process involving multiple
stakeholders – such as domain experts, annotators, and project coordinators – in a complex workflow.
This process encompasses defining guidelines, selecting and configuring tools, preprocessing data,
performing annotations, resolving inconsistencies, assessing quality, and making revisions. Due to its
complexity, this workflow often becomes a bottleneck in IR projects [
Choosing an appropriate annotation tool can streamline the process and ease assessors’ work. In
recent years, several reviews have assessed the efectiveness of annotation tools, compared their
functionalities, and provided guidance to researchers in selecting the most appropriate tool for their
specific needs [
To address the limitations of existing annotation tools, we introduce Doctron [
Doctron [
Doctron is a web-based, platform-independent annotation tool designed for document-topic pair annotation. Distributed as a Docker container, it enables easy deployment and ensures privacy by allowing local installation. Its architecture follows a three-tier design: a data layer with a PostgreSQL database storing documents, topics, and annotations; a business logic layer implemented with Django 2, handling core functionalities such as processing requests and interacting with the database; and a presentation layer developed with React.js 3, providing an interactive platform for annotators.
In Doctron, there are three user types: annotators, reviewers, and administrators. Annotators add annotations to documents. Reviewers, with the highest expertise, have full access to annotators’ work and can update annotations to ensure quality and consistency. Administrators manage the collection, oversee annotators and reviewers, set annotation guidelines, and configure templates and settings. They can modify and update all annotations, including those of reviewers, and track progress.
The user interface of Doctron is presented in Figure 1. Upon login, users select an annotation template, and the system loads the relevant collections and the last opened topic-document pair. The main header 1 provides access to the Home, Collections, and Statistics pages. The document header 2 displays key identifiers and ofers navigation and annotation reset options. A multifunctional left sidebar 3 enables quick access to features like role switching, document/topic lists, settings, annotation downloads, and tutorials. The left panel 4 summarizes the user’s annotations for the current pair, allowing edits and comments. The main area presents the topic details 5 and the document to annotate 6 , with the layout adapting to both text and image-based content.
Annoation templates. Annotators can annotate document-topic pairs in Doctron using seven annotation templates. These templates include (i) graded labeling, where annotators assign labels (e.g., relevance) with a range of values (e.g., 0 to 3) to a document (textual or image) regarding a specific topic; (ii) passage annotation, where annotators select specific passages within a textual document and assign a graded label indicating its relevance to a topic; (iii) object annotation, which involves identifying and labeling objects within images by selecting the object’s perimeter and assigning one or more graded labels to the selected area; (iv) NER, where annotators identify mentions of entities in a text and label them with predefined tags; (v) NER+L, which adds the step of linking identified entities to specific entries in external knowledge bases like Wikipedia or Wikidata; (vi) relationship annotation, where annotators identify relationships between a subject, predicate, and object, which may be represented by ontological concepts, tags, or mentions; and (vii) fact annotation, which involves annotating factual triples (subject, predicate, object), where all components are ontological concepts or tags, and none are textual mentions from the document.
Collections and Customization. A collection in Doctron includes a group of annotators (at least
one), a set of documents (either text or images), a set of topics (either text or images), and is associated
with an annotation template. Each user in Doctron has the ability to annotate multiple document
collections, which exist independently from one another. Documents in Doctron are schema-free and
can be uploaded in several formats including: JSON, CSV, TXT and PDF (thanks to the integration
with GROBID [
Collection Statistics. Doctron provides two levels of statistical reporting: individual and global. Individual statistics give each annotator an overview of their work, including the number of annotated and unannotated documents per topic, as well as annotation-specific metrics depending on the template used (e.g., graded labels, identified passages, tagged entities, or extracted relations and facts). Administrators and reviewers can also monitor these statistics to track annotator activity and spot inconsistencies or documents that need to be reannotated. Global statistics aggregate this information across all annotators in a collection, also listing which annotators worked on each document. To evaluate annotation quality and consistency, Doctron includes IAA metrics – Cohen’s Kappa, Fleiss’s Kappa, and Krippendorf’s Alpha – which are accessible by administrators and reviewers. These metrics provide insight into annotation reliability by quantifying the degree of agreement among annotators. Together, these features ofer a comprehensive view of annotation performance and ensure quality control throughout the annotation process.
We performed a qualitative evaluation where we compared Doctron functionalities to those provided by
other 9 tools: MetaTron [
We performed a quantitative analysis to assess the eficiency of five annotation tools – Doctron,
Doctag, INCEpTION, Doccano, and LabelStudio – by measuring the number of clicks and time required
for two tasks: (i) creating and configuring a document collection, and (ii) annotating 15 documents
using three typical IR-related templates: multilabel classification, passage annotation, and NER. While
the annotation phase yielded comparable results across tools – due to similar implementations of task
templates – significant diferences emerged in the setup phase. Doctron and Doctag proved more
eficient for creating collections, with fewer steps and a simpler setup—Doctron especially stood out
for its ease of use. In contrast, general-purpose tools like INCEpTION and LabelStudio required more
complex configurations to support IR-specific workflows. These findings show that general tools often
demand extra efort for IR tasks, while Doctron ofers a more integrated, ready-to-use solution. More
details can be found in the original Doctron paper [
We presented Doctron, a portable, web-based annotation tool designed for the IR. It supports collaborative workflows with role-based access, ofers a wide range of annotation templates (including graded labeling, passage annotation, NER and NER+L), and integrates with PubMed and ir_dataset ensuring easy and customizable collection setup. Doctron includes built-in IAA metrics and detailed statistics to ensure annotation quality. Through qualitative and quantitative evaluations, we demonstrated that Doctron stands out for its flexibility, rich feature set, and ease of use. As future work, we plan to integrate automatic annotation support using pre-trained models for NER/NER+L and Language Language Models (LLMs) for graded and passage-level annotation.
This work is supported by the HEREDITARY Project, as part of the European Union’s Horizon Europe research and innovation programme under Grant Agreement No GA 101137074.
Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 - System Demonstrations, Abu Dhabi, UAE, December 7-11, 2022, pages 327–337. Association for Computational Linguistics, 2022. [16] T. Perry. Lighttag: Text annotation platform. In Heike Adel and Shuming Shi, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2021, Online and Punta Cana, Dominican Republic, 7-11 November, 2021, pages 20–27.
Association for Computational Linguistics, 2021. [17] J. Pustejovsky and A. Stubbs. Natural Language Annotation for Machine Learning: A guide to corpus-building for applications. " O’Reilly Media, Inc.", 2012. [18] D. Salgado, M. Krallinger, M. Depaule, E. Drula, A. V. Tendulkar, F. Leitner, A. Valencia, and C. Marcelle. Myminer: a web application for computer-assisted biocuration and text annotation.
Bioinformatics, 28(17):2285–2287, 2012. [19] P. Stenetorp, S. Pyysalo, G. Topic, T. Ohta, S. Ananiadou, and J. Tsujii. brat: a web-based tool for nlp-assisted text annotation. In EACL 2012, 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 23-27, 2012, pages 102–107. The Association for Computer Linguistics, 2012. [20] D. Wilby, T. Armakharm, I. Roberts, X. Song, and K. Bontcheva. GATE teamware 2: An open-source tool for collaborative document classification annotation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. EACL 2023 - System Demonstrations, Dubrovnik, Croatia, May 2-4, 2023, pages 145–151. Association for Computational Linguistics, 2023. [21] J. Yang, Y. Zhang, L. Li, and X. Li. YEDDA: A lightweight collaborative text span annotation tool.
In Fei Liu and Thamar Solorio, editors, Proceedings of ACL 2018, Melbourne, Australia, July 15-20, 2018, System Demonstrations, pages 31–36. Association for Computational Linguistics, 2018.