The existing entity annotation systems that drive the extraction of RDF from unstructured data are hard to compare as their evaluation relies on different data sets and measures. We developed GERBIL, an evaluation framework for semantic entity annotation that provides developers, end users and researchers with easy-to-use interfaces for the agile, fine-grained and uniform evaluation of 9 annotation tools on 11 different data sets within 6 different experimental settings on 6 different measures. In this paper, we present the developed interfaces, data flows and data structures. Moreover, we show how GERBIL supports a better reproducibility and archiving of experimental results.
(a) Example spider diagram of recent A2KB (b) Spider diagram of correlations between anexperiments with weak annotation matching. notation results and data set features.
Feature 2: Matchings. GERBIL offers three types of matching between a gold standard and the results of annotation systems: a strong entity matching for URIs, as well as a strong and a weak annotation matching for entities.
Feature 3: Measures. Currently, GERBIL offers six measures subdivided into two
groups: the micro- and the macro-group of precision, recall and f-measure. As shown in
Figure 1(a), these results are displayed using interactive spider diagrams that allow the
user to easily (1) get an overview of the performance of single tools and (2) compare
tools. Explicit definitions can be found in Usbeck et al. [
Feature 4: Diagnostics. An important novel feature of our interface is that it displays the correlation between the features of data sets and the performance of tools (see Figure 1(b)). By these means, we ensure that developers can easily gain an fine-grained overview of the performance of tools and thus detect possible areas of improvement for future work.
Feature 5: Annotators. Currently, GERBIL offers 9 entity annotation systems with a variety of features, capabilities and experiments out-of-the-box.
Feature 6: Data sets. The latest version of GERBIL offers 11 data sets. Thanks to the large number of formats, topics and features of the data sets, GERBIL allows carrying out diverse experiments.
Feature 7: Output. GERBIL’s experimental output is represented as a table containing the results, as well as embedded JSON-LD2 RDF data for the sake of archiving results. Moreover, GERBIL generates a permanent URI for each experimental result.
In this paper, we will give a detailed explanation of the different RDF data structures underlying GERBIL’s architecture. We will explain the internal workflow of GERBIL and argue why it simplifies the implementation of further experiments, annotators, data sets, matchings and measures. We conclude by pointing at future work.
GERBIL unifies the different formats used by existing datasets and annotators. To this
end, GERBIL’s interfaces are mainly based on the NLP Interchange Format (NIF).
This is a RDF-based Linked Data serialization which provides several advantages such
as interoperability by standardization or query-ability. The NIF-standard assigns each
document an URI as starting point and generates another Linked Data resource per
semantic entity. Each document is a resource of type nif:Context and its content
is the literal of its nif:isString predicate. Every entity is an own resource with a
newly generated URI pointing to the original document via the nif:referenceContext
predicate. Additionally the begin (nif:beginIndex) and end position (nif:endIndex)
as well as the disambiguated URI (itsrdf:taIdentRef) and the respective KB (itsrdf:
taSource) are stored. NIF’s paramount position amongst corpora serialisation formats is
evident by the growing number of available datasets [
GERBIL’s main aim is to provide comprehensive, reproducible and publishable
experiment results. Thus, GERBIL enforces the use of a machine-readable description
for each experiment via JSON-LD4 RDF data using the RDF DataCube vocabulary [
GERBIL relies on the DataID ontology [
org/dc/elements/1.1/. 7 datid is a prefix for for http://dataid.dbpedia.org/ns/core#. "@graph" : [ { "@id" : "http://gerbil.aksw.org/gerbil/experiment?id=...#experiment_...", "@type" : [ "gerbil:Experiment", "qb:Dataset" ], "experimentType" : "gerbil:A2KB", "matching" : "gerbil:WeakAnnoMatch", "structure" : "gerbil:dsd", "label" : "Experiment 201503160001" }, { "@id" : "http://gerbil.aksw.org/gerbil/experiment?id=...#exper..._task_0", "@type" : "qb:Observation", "annotator" : "http://gerbil.aksw.org/gerbil/dataId/corpora/Babelfy", "dataset" : "http://gerbil.aksw.org/gerbil/dataId/annotators/ACE2004", "statusCode" : "-1", "timestamp" : "2015-03-16T12:31:52.469Z" } ], "@context" : {
... }
} 2.2
Figure 2 shows the architecture of GERBIL with the data sets at the bottom, the annotators in the top and the user interface as well as user defined annotator and data set at the right. A GERBIL session starts at the configuration screen with which a user defines the experiment he is interested in. Each experiment is divided into tasks. A task comprises the evaluation of a single annotator using a single data set, is encapsulated into fault-tolerant classes and runs inside an own thread. Our fault-tolerance classes at two types of errors: (1) an annotator may return error codes for single documents, e.g., because of the missing ability to handle special characters. While other evaluation frameworks tend to cancel the experiments after an exception thrown by the annotator, GERBIL counts these smaller errors and reports them as part of the evaluation result. The second type of fault tolerance aims at (2) larger errors, e.g., the data set couldn’t be loaded or the annotator is unreachable via its Web service. These run-time errors are handled by storing one of the predefined error codes inside the experiment database. Therewith, we ensure that the user gets instant feedback if some parts of the experiment couldn’t be performed as expected.
During a task, the single documents of a data set are sent to the annotator. After
finishing the last document, the responses are evaluated. Currently, the evaluation is
focused on the quality, i.e., precision, recall, F1-score and error counts, but can be
extended. Moreover, a runtime is also available [
In this paper, we presented GERBIL, a platform for the evaluation, publishing and archiving of semantic entity annotation experiments. GERBIL extends the state-of-the-art benchmarks by dealing with data sets and annotators that link to different knowledge bases. Furthermore it offers extensible interfaces, reliable experiment descriptions as well as diagnostics and decision support. Our future work will comprise a better experiment task scheduling to achieve a higher efficiency. Another task is the improvement of the user interface towards a better intelligibility. Finally, we will devise a solution to ensure that GERBIL remains available to the community for the years to come.
Parts of this work were supported by the FP7 project GeoKnow (GA No. 318159) and the BMWi project SAKE (GA No. 01MD15006E).