We describe our approach to create gold standard corpora for biomedical concept recognition in multiple languages, including English, French, German, Spanish, and Dutch. The annotations are based on a subset of the Unified Medical Language System and cover a wide variety of semantic groups. Gold standard corpora (GSCs) are essential for the development and evaluation of systems that perform natural language processing tasks. Currently available GSCs are only in English, often contain annotations for a limited set of semantic types, and generally do not link the annotations to ontological information. In the MANTRA project (http://www.mantra-project.eu), community efforts are solicited to provide two types of resources: enriched multilingual biomedical terminologies and semantically annotated multilingual documents for a wide range of semantic types. To achieve these goals, the MANTRA project capitalizes on a variety of existing parallel corpora and terminologies. The corpora include multilingual titles of scientific abstracts, drug labels, and biomedical patents. The terminologies are drawn from the Unified Medical Language System (UMLS). The quality of the newly generated resources has to be evaluated on multilingual GSCs in which entity mentions of different semantic types are mapped to unique concept identifiers. In this paper, we describe our approach to construct such GSCs and report initial results.
The GCSs are based on three multilingual corpora that have been collected in the MANTRA project: abstract titles from Medline, drug labels from the European Medicines Agency (EMEA) (freely available through the OPUS collection, http://opus.lingfil.uu.se/EMEA.php), and patents in the biomedical domain from IFI Claims (http://ificlaims.com). The languages of interest in the MANTRA project include English, German, French, Spanish, and Dutch. The Medline titles are bilingual, always in English and one of the other languages. The EMEA labels are available in all languages, the patents only in English, German, and French. Each document in the MANTRA corpora consists of one or more units of text, where a unit may contain a title (Medline abstracts), sentence (EMEA labels), or a paragraph of text (patents). From each MANTRA corpus, units were randomly selected for constructing a GSC: 100 units from the EMEA labels, 100 units from each set of bilingual Medline titles (400 units in total), and 50 units from the patents. Another 20 English units (11 titles, 5 labels, 4 patents) were selected for the development of annotation guidelines.
The annotators had to make their annotations based on the terminology that is used
in the MANTRA project. The MANTRA terminology contains a subset of the
UMLS, including MeSH, MedDRA, and SNOMED-CT. For each concept in these
terminologies, all terms were culled together with their semantic type and concept
unique identifier (CUI). Concepts were included if their semantic type belonged to
one of the following semantic groups [
The annotations are made independently by at least three annotators, using the brat
rapid annotation tool [
For harmonization we use the e-centroid method, an extension of the centroid
method that was developed in the CALBC project [
English GSCs.
Annotation guidelines were established based on the 20 units that were selected for development purposes. In case of multiple pre-annotations of the same stretch of text, the annotators should try to disambiguate. If the difference in meaning between the concepts is not clear or the context provides insufficient information to disambiguate, all annotations are kept. When an entity is nested within another entity, only the most detailed description of the entity is annotated. The general principle is to annotate the entity that is more specific and informative. Only concepts that are part of the MANTRA terminology should be annotated.
Inter-annotator agreement was measured by the F-score (harmonic mean of recall and precision) between two annotators or between one annotator and the gold standard. Note that the F-score is invariant to who of the two annotators is taken as the reference when computing precision and recall. 3
Fig. 1 shows two screen shots of the brat annotation tool for one of the English units. Information on the (pre-)annotated concepts is shown when the cursor is hovered over the annotations. Double-clicking a word or phrase shows a window that allows to make modifications or to link out to further information.
The annotation guidelines were developed based on the annotations of the 20 English training units by three independent annotators and subsequent discussions. A harmonized annotation for this set was automatically constructed, and interannotator agreement scores were computed (Table 1). The F-scores indicate good to excellent agreement between annotators 1 and 3 and the harmonized set.
The annotation of the 550 units in the full GSC has been started and is still work in progress.
We described our approach to create multilingual GSCs for biomedical concept recognition. First steps have been taken, including the development of annotation guidelines and a flexible annotation environment, and the selection of multilingual text units from different document types. Inter-annotator agreement scores on a small development set suggest that the annotations of different annotators are in good agreement.
To our knowledge, this is the first attempt to create GSCs for biomedical concept recognition in languages other than English. Other distinguishing features are the wide variety of semantic groups that are being covered, and the diverse text genres from which units have to be annotated.
The creation of the GSCs is currently under development. Once available, the GSCs will be made publicly available.