This paper introduces WikiMed-DE, a silver-standard, automatically annotated biomedical entity linking dataset for the German language. WikiMed-DE encompasses a substantial collection of 53,981 articles from the German Wikipedia annotated with 1,951,081 mentions corresponding to 317,010 unique mention URLs. The hyperlinks of Wikipedia articles are used to connect concept mentions to Wikidata and transitively to three biomedical concept IDs: the Concept Unique Identifier from the Unified Medical Disease Ontology. A curated subset, WikiMed-DE-BEL, is released as a ready-to-use benchmark for biomedical entity linking in German. It features the same number of articles as WikiMed-DE, but only the highest-quality information is retained: 413,913 mentions corresponding to 35,012 unique concepts.
CEUR ceur-ws.org
Biomedical entity linking (BEL) is an important task for automatically processing text from the
medical domain. It enables the disambiguation of entities in text to unique identifiers in
ontologies like the Unified Medical Language System (UMLS) 1 [
Many datasets have been created in order to foster the development of reliable BEL systems,
e.g. the NCBI Disease dataset [
These datasets were manually annotated by domain experts and focus on entities of interest from various domains - e.g. diseases, genes, tissues, chemical compounds, etc. And while many of these datasets are not very large, they do provide invaluable information for training machine learning models for the automatic disambiguation of biomedical entities.
However, the vast majority of biomedical datasets provide annotations for English texts. For other languages, like German, datasets annotated with biomedical concepts are extremely scarce due to the resource-intensive nature of the manual creation process, which requires trained professionals to perform the annotation process.
This paper addresses this issue by introducing WikiMed-DE, a silver-standard dataset for
biomedical entity linking for the German language. The automatic annotation process makes
use of the links connecting the text of the German Wikipedia5 pages with the structured
information available in the Wikidata [
Our contributions are the following:
1. We build upon and extend the procedure introduced by Vashishth et al. [
The remainder of this article is organized as follows: Section 2 provides an overview of the related work, Section 3 introduces the knowledge sources used to create WikiMed-DE, Section 4 describes the methodology used to construct WikiMed-DE, Section 5 presents the statistics of the dataset and asseses its quality, Section 6 discusses the limitations of WikiMed-DE and concludes the paper.
3MeSH: https://www.ncbi.nlm.nih.gov/mesh/ 4SNOMED-CT: https://www.snomed.org/ 5German Wikipedia: https://de.wikipedia.org/wiki/Wikipedia:Hauptseite 6WikiMed-DE dataset: https://doi.org/10.5281/zenodo.8188966 7WikiMed-DE code repo:https://github.com/AI4MedCode/wikimed-de
Datasets in the BEL domain typically focus on annotating biomedical entities from existing
ontologies. A point in case is the BC6BioID [
Medical Concept Normalization (MCN) [
The NCBI disease corpus [
MedMentions [
Vashishth et al. [
The BRONCO [
In an efort to support the biomedical entity linking in languages other than English, Liu
et al. [
8GeneOntology: http://geneontology.org/ 9ICD-10-CM: https://www.cdc.gov/nchs/icd/icd-10-cm.htm 10OMIM: https://www.omim.org/ 11Wikidata: https://www.wikidata.org/wiki/Wikidata:Main_Page 12ICD-10-GM: www.dimdi.de/dynamic/de/klassifikationen/icd/icd-10-gm/
Wikidata [
The Unified Medical Language System (UMLS) [
Medical Subject Headings (MeSH) [
The Disease Ontology (DO) [
16Disease Ontology release: https://github.com/DiseaseOntology/HumanDiseaseOntology/releases
UMLS Wikipedia
Entity
UMLS Concept [C5200801]
The WikiMed-DE dataset is based on a recent database dump of the German Wikipedia from 20.06.202317. WikiExtractor18 was used to extract the article title, page id, URL and text from the archive dewiki-20230620-pages-articles-multistream.xml.bz2 and save them in JSON format. The text of each article contains HTML-encoded hyperlink tags to create clickable links, which are retained for the annotation step. The WikiExtractor outputs roughly ∼10,000 files from this archive, each containing on average 450 Wikipedia articles. These files are combined in a post-processing step into a single JSON file containing 4,579,135 Wikipedia articles.
We map the Wikipedia articles and their mentions to Wikidata QIDs using another file from the German Wikipedia database dump, namely dewiki-20230620-page_props.sql.gz. This file relates a page ID from the German Wikipedia to the corresponding Wikidata QID. For instance, the 17German Wikipedia dump, 20.06.2023: https://dumps.wikimedia.org/dewiki/20230620/ 18WikiExtractor: https://github.com/attardi/wikiextractor ifrst entry (1, 'wikibase_item', 'Q734916', NULL) in this file indicates that the Wikidata QID associated with German Wikipedia page ID 1 is Q734916.
The result of this mapping step is a CSV file containing the QID and the Wikipedia page ID for each entry. Among the 4,579,135 Wikipedia articles from the previous step, a total of 1,754,551 page IDs lack a corresponding QID. This is due to the fact that not all the page IDs in the archive dewiki-20230620-pages-articles-multistream.xml.bz2 appear in the archive dewiki-20230620-page_props.sql.gz. In the next steps we focus on the 2,824,584 articles that have a corresponding QID.
The oficial Wikidata SPARQL endpoint 19 was used to generate a mapping from QIDs to biomedical concept IDs. Three properties from Wikidata were targeted: • P2892:UMLS CUI20, which maps a Wikidata item to its UMLS CUI, if one is available • P486:MeSH descriptor ID21, which maps a Wikidata item to the Medical Subject Headings identifier, if it has one and • P699:Disease Ontology ID22, which connects a Wikidata item to its ID in the Disease
Ontology, if such a mapping exists.
All three properties feature a single-value constraint23 in Wikidata, which states that this property generally contains a single value per item. However, as we will show in Section 4.6, these constraints are not enforced in Wikidata, making it possible for a Wikidata item to have, for example, multiple CUIs associated to it. We obtained a mapping from QIDs to 763,859 UMLS CUIs, 38,607 MeSH IDs and 10,609 DOIDs.
WikiMed-DE is meant to serve as a training material for BEL models. It is therefore important to filter the German Wikipedia articles and retain only those related to the biomedical domain. The articles are filtered based on the mapping of QIDs to the three biomedical concepts of interest described in Section 4.3. We retain only those articles where the QID is associated with at least one of these three biomedical IDs, resulting in 54,514 articles. However, in some cases, an article will have a title, a QID and a valid mapping, but no text - such articles are also filtered out. At the end of the filtering step there are 53,981 German Wikipedia articles left.
In order to fulfill its goal as a training resource for BEL, the hyperlinked words or phrases in the WikiMed-DE articles need to be identified and annotated with biomedical concepts. These 19Oficial Wikidata SPARQL endpoint: https://query.wikidata.org/ 20P2892:UMLS CUI: https://www.wikidata.org/wiki/Property:P2892 21P486:MeSH descriptor ID: https://www.wikidata.org/wiki/Property:P486 22P699:Disease Ontology ID: https://www.wikidata.org/wiki/Property:P699 23Wikidata item for single-value constraint: https://www.wikidata.org/wiki/Q19474404 are extracted using regular expressions, by identifying HTML-encoded tags in the article text and decoding them to generate URLs and clean text.
In this step we store, for each article, it’s title, text, URL, and a list of mentions corresponding to the hyperlinked words or phrases in the text. For each mention we record the text of the mention and the corresponding URL. We then save a list containing 317,010 unique mention URLs, which we need to map to Wikipedia page IDs. However, the URLs that are originally associated with the mentions are diferent from the URLs needed to obtain the page ID for each mention. For example, for the page with the title Getreide we need to map from the original mention URL24 to the corresponding Wikipedia page information URL25.
To obtain this mapping we do a request for each Wikipedia page information URL and retrieve the corresponding Wikipedia page ID. 259,250 mention URLs are successfully matched to a corresponding page ID in this way. The rest of the mention URLs (57,760 links) yield no page information because some links do not exist in the German Wikipedia. For example, in Figure 2, the URL corresponding to the red mention Aktin-bindenden Proteinen lacks a corresponding page ID because it is a link to a page that does not yet exist in the German Wikipedia. Such cases are relatively frequent, as Wikipedia editors routinely add links to pages that will be created only in a subsequent step.
The next step is to map the page IDs corresponding to each mention to QIDs. We use the previously generated CSV file (from Section 4.2), which contains a mapping from Wikipedia page IDs to QIDs. Leveraging this resource 206,549 page IDs out of the 259,250 are uniquely mapped to a QID. The mapping is, however, incomplete, with 52,701 page IDs lacking corresponding QIDs. We examined a small sample of these URLs and discovered two issues: (i) some hyperlinks point to sections within the same article, and therefore cannot be mapped to a separate QID and (ii) some of the URLs extracted from the hyperlink tags are redirects. For example, the page Hydrophil26 redirects to Hydrophilie27. The page ID information is not typically stored on the redirect page, but only on the target URL.
After the two previous steps 110,461 mention URLs are still not successfully mapped to their corresponding QIDs: 57,760 mention URLs lack a page ID and 52,701 mention URLs have a page ID but no QID. To address the redirection problem, we used the wikipedia28 Python package to obtain a page ID for the URLs that were still missing a QID. This package allows one to look for a Wikipedia page given the title of the page and the language code of the wiki ('de' in our case). By specifying the flag redirect=True one can also find the page IDs for redirects. We applied wikipedia’s page function to these 110,461 mention URLs and obtained an extra 46,541 correct mappings to QIDs.
The methodology described in Section 4.3 is used to map the QIDs to three biomedical concepts: the UMLS CUI, the MeSH ID, and the DOID. When a particular Wikidata item contains state24Mention URL for Getreide: https://de.wikipedia.org/wiki/Getreide 25Wikipedia page info URL for Getreide: https://de.wikipedia.org/w/index.php?title=Getreide&action=info 26Hydrophil: https://de.wikipedia.org/wiki/Hydrophil 27Hydrophilie: https://de.wikipedia.org/wiki/Hydrophilie 28wikipedia Python package: https://pypi.org/project/wikipedia/ ments about biomedical IDs, it will, most frequently, contain a statement about the associated UMLS CUI of that item. However, in some cases, the Wikidata items contain statements about their MeSH IDs or their DOIDs, but does not include statements recording their UMLS CUIs. Consequently, to increase the number of linked biomedical concepts, we extract UMLS CUIs, MeSH IDs and DOIDs for each article. In the WikiMed-DE dataset, we use the tags wikidata_cui, mesh and doid to label this information. Figure 2 shows a sample of WikiMed-DE: an article together with its meta-information and its annotations.
Filamine (FLN) sind Proteine bei Eukaryoten und gehören zu den Aktin-bindenden Proteinen (ABP).
Sie sind an der Quervernetzung von Aktinfilamenten, einem Hauptbestandteil des Zytoskeletts, sowie der Vernetzung von Aktinfilamenten mit Proteinen in der Zellmembran beteiligt.
The MeSH hierarchy is integrated into the UMLS. Therefore we can use the file MRCONSO.RRF from the UMLS release to map the MeSH IDs to UMLS CUIs, resulting in a mapping that is not always unique. In WikiMed-DE, the CUIs mapped from MeSH IDs are saved under the tag mesh_cui. A similar mapping is also performed for the DOIDs, using the file doid.json from Disease Ontology’s current release. In WikiMed-DE the CUIs mapped from the DOIDs are saved as doid_cui.
Finally, we consolidate the CUI information saved under the tags wikidata_cui, mesh_cui and doid_cui. If, under all these tags, there is only a single CUI for a mention or an article, this unique CUI is saved under the tag cui in WikiMed-DE. This unique CUI is further mapped to one or more TUIs using the file MRSTY.RRF from the UMLS release. The list of TUIs and the corresponding semantic type labels are saved under tui and semantic_type, respectively.
A part of the mentions will not have a single CUI, but several possible CUIs. This can have multiple reasons: either the Wikidata item already maps to multiple CUIs, despite the fact that the property P2892 has a single-value constraint; or the mapping from MeSH ID to CUIs resulted in several CUIs; or the consolidation step lead to a list of CUIs rather than a single CUI. In any case, we have no automatic method to choose a single correct CUI among the given ones, so we will typically just record all this information.
To obtain the final version of WikiMed-DE, we reprocess each of the articles and add the information we extracted in the previous steps for each article. The text within each Wikipedia page is decoded, removing any HTML tags and producing clean text and mention URLs. The start and end indices for each mention are recorded, thus enabling precise identification of the mention’s position. In some cases, the extracted start and end positions will not overlap with natural token boundaries (see Figure 2 for an example). Each mention is associated with its corresponding QID and all the extracted biomedical information.
At the end of the extraction process, WikiMed-DE consists of a list of German Wikipedia articles. For each article we save the article’s title, text, URL, QID, the biomedical indices (CUI, TUIs, semantic type labels, Wikidata CUI, MeSH ID, MeSH-derived CUI, DOID and DOIDderived CUI) and mention list, where the mentions correspond to the hyperlinked words or phrases in the text. For each mention we record the start and end indices, URL, QID and the biomedical indices (same as for the article).
WikiMed-DE is a dataset consisting of 53,981 German Wikipedia articles, each containing multiple mentions. The filtering step described in Section 4.4 ensures that all the articles in the dataset describe biomedical concepts mentioned either in the UMLS, in the MeSH hierarchy or in the Disease Ontology. The WikiMed-DE articles and the mentions therein are mapped to QIDs and to biomedical concept IDs. We can therefore analyze the dataset at two levels: at the article level and at the mention level.
WikiMed-DE contains a total of 1,951,081 mentions corresponding to 317,010 unique mention URLs. As shown in Table 1, 95.79% of these mentions have an assigned QID, with the rest having either missing links or missing QID information. CUI information was assigned for 29.59% of the mentions. Note that both directly through Wikidata and through the MeSH ID we obtained a larger amount of assigned CUIs (38.30% and 35.46%, respectively). However, part of these mentions have multiple CUIs assigned, and are therefore ambiguous from the point of view of biomedical entity linking. The disambiguation step is non-trivial and cannot be done without the help of experts, so we decided to just keep all the information in the dataset. Disease Ontology information is available only for a small percentage of the mentions. From 317,010 unique mention URLs, 79.82% have an assigned QID and 14.94% have an assigned CUI. WikiMed-DE contains therefore links to 47,380 unique biomedical concepts. The number of unique CUIs is much smaller than the number of QIDs. However, we believe that a large number of the items that have a QID in this dataset will, at some point, be connected to the UMLS and assigned a CUI. This is because many of the linked entities are still biomedical entities that are just not yet marked as such in Wikidata, or are marked using other biomedical IDs (e.g. P351: Entrez Gene ID). By including the QID information in the dataset, we give the research community the possibility to customize the dataset annotations to include other relevant biomedical information.
For the convenience of the researchers interested only in a biomedical entity linking benchmark we provide a curated subset, called WikiMed-DE-BEL, which focuses exclusively on the mentions that have an unique UMLS CUI associated to them. WikiMed-DE-BEL contains 413,913 mentions corresponding to 35,012 unique mention URLs. All mentions are automatically annotated with a single CUI. Mentions annotated with multiple CUIs are discarded. We also discard any mentions where the mention start or end index does not coincide with the start or end of a token. The dataset was divided into train, test and development splits using an 80-10-10 ratio — leading to 43,184 train articles, 5,399 test articles and 5,398 dev articles. The dataset portions contain 330,233 (train), 41,120 (test) and 42,560 (dev) mentions, annotated with 222,247 (train), 9,123 (test) and 9,149 (dev) unique UMLS CUIs, respectively. 833 (9.12%) of the concepts in the test set do not occur in training.
To assess the data quality of WikiMed-DE-BEL, a sample of 50 mentions annotated with a single CUI was randomly selected from the dataset. One of the authors checked the automatically annotated CUI for each mention, comparing it to the information available in the UMLS, using the UMLS Metathesaurus Browser29. The information was also compared to the context available in the Wikipedia article. 100% of the mentions were found to link to the correct concept and to match their context accurately. This shows that the strict filtering of problematic instances (e.g. mentions with multiple CUIs, or with missing links or QIDs) lead to the creation of a high-quality dataset.
Accuracy of Information: WikiMed-DE’s annotation process heavily relies on the accuracy and completeness of information available in Wikipedia and Wikidata. However, these sources are not immune to errors, inconsistencies, or vandalism. As a result, inaccuracies or outdated information present in the source material can propagate into the annotations in WikiMed-DE. Noise and Ambiguity: Automated annotation processes can introduce noise and ambiguity in the dataset. The automated methods used to match Wikipedia articles with biomedical concepts may still encounter challenges in fully disambiguating mentions - for example, we cannot systematically choose a single CUI for an entity if multiple CUIs are annotated in Wikidata. We try to limit the noise and ambiguity by enforcing stricter constraints - e.g., we only consider as valid annotations the mentions with a unique CUI mapping. However, this makes the dataset a silver standard dataset, since not all the proposed annotations were manually verified by domain experts.
Link coverage: Not all entities mentioned in a Wikipedia page are exhaustively marked with hyperlinks, meaning that many possible mentions will not be annotated. Furthermore, because we focus on the quality of annotations, we also end up discarding a portion of the marked hyperlinks. This leads to a dataset that has a lower mention coverage than a typical biomedical dataset. WikiMed-DE is therefore less suited for biomedical named entity recognition tasks. However, we believe that it is a useful resource for training BEL systems, and it has great potential to be further developed as new information is added to Wikidata.
This paper presented a new resource for disambiguating biomedical entities in German, WikiMed-DE, and a benchmark dataset for biomedical entity linking in German, WikiMed-DEBEL, thus supporting biomedical entity linking research focusing on the German language.
29UMLS Metathesaurus Browser: https://uts.nlm.nih.gov/uts/umls/home
Yi Wang and Corina Dima were supported by the Ministry for Economics, Labour and Tourism from Baden-Württemberg, Germany via grant agreement number BW1_1456 (AI4MedCode). 11153. arXiv:1908.11153. [21] E. Mitraka, A. Waagmeester, S. Burgstaller-Muehlbacher, L. M. Schriml, A. I. Su, B. M.
Good, Wikidata: A platform for data integration and dissemination for the life sciences and beyond, in: J. Malone, R. Stevens, K. Forsberg, A. Splendiani (Eds.), Proceedings of the 8th Semantic Web Applications and Tools for Life Sciences International Conference, Cambridge UK, December 7-10, 2015, volume 1546 of CEUR Workshop Proceedings, CEURWS.org, 2015, pp. 69–73. URL: https://ceur-ws.org/Vol-1546/paper_38.pdf. [22] S. Federhen, The NCBI taxonomy database, Nucleic Acids Res. 40 (2012) 136–143. URL: https://doi.org/10.1093/nar/gkr1178. doi:10.1093/nar/gkr1178.