Background Text mining challenges have been organised to measure the performance of automatic text mining solutions against a manually annotated gold standard corpus (GSC). The preparation of the GSC is timeconsuming and costly and the final corpus consists at the most of a few thousand documents annotated with a limited set of semantic groups. To overcome these shortcomings, the CALBC project partners (PPs) have produced a large-scale annotated biomedical corpus with four different semantic groups through the harmonisation of annotations from automatic text mining solutions, the first version of the Silver Standard Corpus (SSC-I). The four semantic groups were chemical entities and drugs (CHED), genes and proteins (PRGE), diseases and disorders (DISO) and species (SPE). This corpus has been used for the First CALBC Challenge asking the participants to annotate the corpus with their annotation solutions.
All four PPs from the CALBC project and in addition, 12 challenge participants (CPs) contributed annotated data sets for evaluation against the SSC-I. CPs could ignore the training data and deliver the annotations from their annotation system, or could train a machine-learning approach on the provided preannotated data. In general, the performances of the annotation solutions were lower for the CHED and PRGE in comparison to the identification of DISO and SPE. The best performance over all semantic groups were achieved from two annotation solutions that have been trained on the SSC-I. The data sets from participants were used to generate the harmonised Silver Standard Corpus II (SSC-II), if the participant did not make use of the annotated data set from the SSC-I for training purposes. The performances of the participants’ solutions were again measured against the SSC-II. The performances of the annotation solutions showed again better results for DISO and SPE in comparison to CHED and PRGE.
The SSC-I delivers a large set of annotations (1,121,705) for a large number of documents (100,000 Medline abstracts). The annotations cover four different semantic groups and are sufficiently homogeneous to be reproduced with a trained classifier leading to an average F-measure of 85%. Benchmarking the annotation solutions against the SSC-II leads to better performance for the CPs’ annotation solutions in comparison to the SSC-I.
Biomedical text mining (TM) has developed into a bioinformatics discipline seeking IT methods delivering
accurate results from an automatic literature analysis into bioinformatics research. This objective requires
the development of benchmarks and the assessment of existing TM solutions against these benchmarks. A
number of challenges have been proposed to achieve this goal: BioCreAtive I and II, JNLPBA and others
[
The first CALBC Challenge is similar in the sense that the project partners (PPs) of the CALBC project
also provided an annotated corpus to the challenge participants (CPs) of the first CALBC challenge to
reproduce the annotations with automatic means. On the other side, the first CALBC Challenge was
different to the before-mentioned challenges with regards to the following modifications: (1) the annotated
corpus has been generated automatically and not manually (Silver Standard Corpus, SSC-I), and (2) the
size of the SSC-I is significantly bigger than the corpora mentioned produced for the other challenges, i.e.
the annotated corpus contains 50,000 Medline abstracts for training and the corpus for annotation consists
of 100,000 test documents. This difference in size requires that all assessment is performed fully
automatically, that the CPs apply annotation solutions that can cope with such a large-scale corpus and that
the assessment solutions can evaluate the contributions in a short period of time. The automatic annotation
of the corpus also requires new solutions to harmonise the contributions from different automatic
annotation solutions. The term “harmonisation” refers to the process that determines a set of annotation
boundaries in the text. Overall these annotations should have the characteristic that all annotation solutions
show high performance against the set of annotations, for example when measuring the F-measure of the
annotation solution [
When comparing different NER solutions, it becomes clear that they do not generate the same results
depending on the technology behind and the type of resources used for the instantiation of the solutions
(see BioCreative II). On the other side, when combining the results from different automatic annotation
solutions, we can achieve an improvement of the results of the combined solution (see BioCreative
MetaServer) [
In addition, each annotation solution is optimised for a single semantic type and solutions for a larger scope of semantic groups are still missing. This is again partly due to the fact that manually curated corpora can only cover a small number of semantic groups to focus the ongoing work to the amount of work that is achievable in a fixed period of time and according to the available budget. The proposed approach of the CALBC project can cover a larger number of annotations due to the fact that the annotations are produced automatically and harmonised with automatic means.
In this manuscript, we report on the results of the first CALBC challenge. The CPs have submitted one or several sets of annotated documents. All the submissions have been assessed against the SSC-I. In addition, the submissions have been used to generate the second Silver Standard Corpus (SSC-II) and all the submissions have been assessed against the SSC-II. The results are presented in this manuscript to support a better understanding to which extent the automatic generation of an annotated corpus contributes to the benchmarking of annotation solutions in a domain where a large number of NERs have to be identified inside a large number of scientific documents. In the CALBC project and challenge the PPs and CPs contribute their annotations on a given corpus to enable the harmonisation of all annotations for a large-scale annotated corpus. A priori the annotation solutions do not share any properties and the contributed annotations should be produced by independent systems, but should be similar in the sense that they contribute annotations for entities in the biomedical domain. This leads to the result that the different solutions make use of similar biomedical data resources for the representation of terms and concepts and thus should show similarities in the annotation
Solution Dictionary-based concept recognition Indexing of tokens
and terms Both, trained & rule- P03 based solutions Case-based reasoning CRF based, trained
NER solution
P02 P04 P06 P10 P13 P15 P09 P07 P16 P11 P12 P14
Use of PPs | Training CPs Data P01 [ / ]
All PPs annotated the corpus of 150,000 Medline abstracts with their annotation solutions. The project
partners P01, P02 and P04 used dictionary-based concept recognition methods with techniques for quality
improvements, whereas partner P03 applied a combination of solutions that are either dictionary-based or
is based on machine-learning techniques. All annotations were delivered in the IeXML format and concept
normalisation should make use of standard resources such as UMLS, UniProtKb, EntrezGene or should
follow the UMLS semantic type system [
The alignment is based on the methods described in [
The contributions of the CPs were evaluated against the SSC-I. Different evaluation schemes were used to
determine the performance of the solutions [
I from CPs frComPsall II 228,622 6 11 233,398 238,431 275,235 9 15 343,681 435,797 300,637 8 11 255,599 245,524 317,211 7 9 277,071 304,503
The best average F-measure performances were achieved when applying 0.9 and 0.98 cosine similarity scoring. All submissions from all participants have been evaluated and the contributions with the best Fmeasure performance against the SSC-I have been selected for the harmonisation into the SSC-II. For the harmonisation of the contributions (SSC-II), a varying number of contributions had to be considered for the different semantic groups, i.e. 6 for CHED, 7 for SPE, 8 for DISO and 9 for PRGE (see table 2). A 3-vote agreement in combination with a 0.98 cosine similarity score in the alignment was required for the acceptance of the annotation across the different contributions. For all semantic groups, different voting schemes (i.e. 2- to 6-vote agreement) were evaluated to determine the best performing voting scheme in terms of the highest average F-measure performance across the contributions of the CPs. The 3-vote agreement delivered the best balance between the recall and the precision of the contributions against the harmonised SSC-II. All presented results are based on the annotations on the subset of 1,000 Medline abstracts.
The alignments of the 100,000 documents were either performed on Sun Fire opteron servers (4 or 8 CPUs, RAM sizes from 32 to 256 Gb RAM, 9-12 hours) or on the compute farm of 700 IBM compute engines (dual CPU, 1.2-2.8 Ghz, 2 GB RAM, 3 hours).
12 CPs actively took part in the challenge. Each CP could contribute several submissions at any time. Overall the CALBC challenge received 19 valid submissions. 3 CPs used the SSC-I as training data and contributed in total 8 submissions (ref. to table 2). 2 CPs did not use the SSC-I, but used an annotation solution that has been trained on a different annotated corpus for the challenge. All other partners (in total 11) used dictionary-based solutions and in one case used a combination of different solutions without training on the SSC-I.
Five CPs only focused on a single semantic group. All the other CPs covered three or more semantic groups. CP P10 delivered for PRGE a very high number of annotations, which impaired the performance of the system against the SSC-I.
The PPs contributions have been aligned to generated the SSC-I. The SSC-I has been contributed to the public to train machine-learning based NER solutions on the corpus and to gather the annotations of the CPs for performance assessments. The contributions of the CPs have been used to generate the SSC-II.
The submissions of the CPs were compared against the SSC-I (see below). Table 3 shows that two
solutions that were based on the provided training data reproduced the SSC-I annotation „standard“ at a
high level of quality: for SPE the solutions achieved 93% F-measure and for the other semantic groups the
F-measure was above 80% [
Precision
The two best-performing machine-learning based solutions produce results that are comparable to known
solutions for the gene mention task [
Fig. 2 shows a distribution for the performance for the annotation for chemical entities. The two best performing machine-learning solutions outperform again all other annotation solutions and the PPs’ annotation solutions have performances that are rather similar to each other and quite different from the performances of the contributions from the CPs. Comparing the results in fig. 1 and fig. 2, we note that the best annotation solutions show better performance for chemical entities than the same solutions 67 demonstrate for the annotation of proteins/genes. We conclude that the annotation of PRGEs in the SSC-I have higher variability (or more noise) than the annotations for the chemical entities in the same corpora. The following figures (fig. 3 and 4) show the same distribution for disease and species mention identification. For these two tasks the annotation solutions show better performance than for the previous two tasks (PRGE and CHED). We can derive that a good performance on these two tasks can be reached by the majority of the annotation solutions in comparison to the other two tasks.
20.0% 40.0% 60.0% 80.0% 100.0%
Precision 20.0% 40.0% 60.0% 80.0% 100.0%
Precision
The diagram for the identification of the diseases (DISO, fig. 3) demonstrates that the majority of the proposed systems identified the diseases at a recall of 60% and above, and at a precision of 55% and above. Two rule-based solutions from CPs showed similar performances to the PP’s solutions. We can conclude that the representation of the diseases in the SSC-I is better standardised and thus includes less variability or noise than the representation of proteins/genes and chemical entities.
The identification of species could be solved to the best precision and the best recall values from the large majority of all proposed solutions. Again the two best performances were achieved by two machinelearning approaches that reproduced the annotations from the training data. The performances of the other solutions, i.e. the PPs’ solutions and the CPs’ solutions, had the best performances for the identification of species in contrast to the other tasks. It is clear that the identification of species can be performed at a level of quality which is above the measured performances of the other semantic groups.
SPE DISO CHEM PRGE The results of the CPs and the PPs were compared against the SSC-II in addition to the SSC-I. Both harmonised sets were generated by using 98% cosine similarity and the comparison against the corpus was done with the same measure.
Tagging of proteins/genes and chemical entities measured against the SSC-II The performances of the PPs’ annotation solutions for genes/proteins showed lower results in the assessment against the SSC-II than in comparison to the SSC-I (refer to fig. 1). Since the SSC-II represents the harmonisation of annotations across a larger number of contributions, it can be expected that the annotations in the SSC-II are more heterogeneous than in the SSC-I.
The performance of the CPs’ annotation solutions has improved against the SSC-II in comparison to the SSC-I: the precision against the SSC-II has increased in comparison to the SSC-I. Recall has also improved. This result shows that the SSC-II incorporates characteristic features that are shared amongst all annotation solutions.
In the SSC-II the annotation solutions of the PPs for chemical entities show lower performance in comparison to the SSC-I (refer to fig. 2). The performance of the CPs’ annotation solutions has improved. Altogether the distribution of the performances of the PPs’ annotation solutions and the CPs’ solutions is comparable.
SSC-I / PRGE
SSC-II / PRGE
SSC-I / CHEM
SSC-II / CHEM 0
P13 P24 P3 1 P4 2 P510 P68 P175 P89 P96 10 0
P13
P24
P31
P42
P510 P68
P715 8
As can be seen in fig. 5, the performances of the annotation solutions for the four PPs deteriorated when comparing the performance against SSC-II instead of SSC-I. Furthermore, the performance of the four PPs against the SSC-II shows an F-measure that seems to be more evenly distributed across the different PPs, i.e. the systems seem to be more equal.
The performances of the CPs’ annotation solutions have improved when moving from the SSC-I to the SSC-II. This result can be explained by the fact that the contributions of the CPs have been included into the SSC-II in comparison to the SSC-I.
The results from the comparison of the annotation solutions for the chemical entities are not as clear as the results for the annotation of proteins/genes. In the case of the chemical entities, the performances of the PPs’ solutions deteriorate except for one PP. The performance of the CPs’ solutions varies to a small extent.
Tagging of diseases and species measured against the SSC-II The PPs’ annotation solutions and the CPs’ solutions show similar performance against the SSC-II and the SSC-I. The two corpora seem to have the same characteristics concerning the annotated entities in the corpus. In other words, the contribution of the CPs to the harmonised corpus did not change the quality of the silver standard corpus when producing the SSC-II from the PPs’ and the CPs’ contributions in comparison to the SSC-I. We can conclude that the annotation of disease entities is better normalised than the two other semantic groups, i.e. chemical entities and protein/genes, respectively.
Similar to the assessment of disease annotations, the species tagging solutions of the PPs and the project CPs did not vary when the annotations were evaluated against the SSC-II in comparison to the SSC-I. For both corpora, the annotation solutions yielded similar results. This leads to the conclusion that the SSC-I and the SSC-II have similar annotations and also to the result that the different contributing systems had 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% similar performances right from the beginning. Overall, we can conclude that the representation of species is better normalised or standardised in the scientific literature than chemical entities or gene/protein representations.
In the last analysis, we compared the F-measures reached from the individual systems against the SSC-II directly against the F-measures from the SSC-I. This should give an overview on the solutions that gained performance in the SSC-II over the SSC-I and the other solutions that deteriorated their performance.
SSC-I / SPE SSC-II / SPE SSC-I / DISO SSC-II / DISO 0
P13 P24
P3 1 P42 P510 P68 P715 P89 9 0
P13 P24 P31 P4 2 P510 P68 P715 P89 P96 10
When analysing the performance of the different solutions for species annotations and for diseases annotations, we find only small differences in the performances of the systems against the SSC-I and the SSC-II.
Direct measurement of the SSC-I against the SSC-II
Reference: SSC-I (cos 0.98) SSC-II DISO SPE PRGE CHED Rec 89.0% 94.5% 59.7% 49.6% Prec 71.6% 90.0% 96.8% 49.4%
F-meas 79.3% 92.2% 73.8% 49.5%
In the direct comparison between the SSC-I and the SSC-II, the annotations for SPE and DISO show better agreement than the comparison of the annotations for PRGE and CHED. The latter shows the lowest performance indicating that higher diversity exists between the two corpora.
Manual inspection of the SSC-I and the SSC-II The manual analysis of the SSC-I and the SSC-II is ongoing work. Due to the size of the corpus, it requires special IT solutions to oversee the regularities and irregularities in the corpus. A selection of irregularities result from the methods applied. First, a number of annotations are not captured (“false negatives”, FN, reduced recall) if none of the solutions identifies the entities. An increasing number of contributing annotation solutions reduces the risk that annotations are missed: a bigger number of included annotation solutions lead to a bigger number of annotations that are captured. This achievement is counterbalanced by the number of agreements that have to be available at minimum to accept an annotation. Second, for the same type of entity, e.g. “insulin”, different annotation solutions use a different tag, e.g. PRGE instead of CHED and vice versa. The harmonisation of the corpus can account for this, but will not produce this type of polysemous annotation throughout the whole corpus, since not all mentions have been consistently annotated with the two different groups over the whole corpus.
Third, inflections of terms, e.g. “tumour” vs. “tumours” and “bear” vs. “bearing”, lead to disagreements between the different annotation solutions. In the first case, the inflectional variability could be resolved and would lead to higher agreement, in the second case assumptions about the usage of the verb or noun have to be made to resolve conflicts. The comparison of the proposed solutions against the SSC-I is a new approach to evaluate annotation solutions. Until now, no large-scale corpus was available to achieve this task. In addition, it became clear that the SSC-I is homogeneous enough to be used as training data to achieve the same annotation task across the different semantic groups.
The generation of a harmonised corpus is a challenging task, but the presented results demonstrate that the produced harmonised corpus integrates the characteristics from the different annotation solutions. As a result, we can determine the features in the harmonised corpus by the annotation solutions that contribute to the generation of the SSC.
From a different perspective, we can argue that each of the used annotation solutions represents a piece of the complete annotation task. The more solutions are combined, the more closely we approximate an assumed consensus in the annotation task, which can be reproduced with a machine-learning tagging solution.