Strong privacy constraints heavily constrain the public availability of health record data in German which hinders the development of advanced NLP methods for clinical texts. As a remedy, work by Frei and Kramer [1] has leveraged the recent breakthroughs in generative large language models to generate a synthetic dataset for training Named Entity Recognition (NER) models in the clinical domain. Because the basis is synthetic, both the corpus and the NER model are publicly available. However, as clinical text is highly idiosyncratic, it is not clear how well this approach performs on real data. We evaluate the model on two real-world German clinical datasets from cardiology and oncology departments. Our analysis shows that learning on generated data for NER models do not transfer well to real-world data.
Some of the important information in electronic health records is found only in unstructured text and needs to be extracted using natural language processing (NLP). An important task to this end is named entity recognition (NER) [2,3]. However, this task is difficult in the clinical domain because the goal of medical text is to concisely capture large amounts of factual information, resulting in abbreviations, domain-specific terminology and awkward grammar. Differences between processes, hospitals and practitioners make it difficult to transfer models between different datasets [4]. In addition, privacy concerns limit the sharing of such data and make annotation expensive due to the need for anonymisation [5]. This problem is particularly exacerbated for German-language applications, not only due to strong EU privacy regulations such as GDPR, but also due to stronger national privacy regulations [6].
Recently, the first annotated German clinical datasets BRONCO150 [7] and CARDIO:DE [8] for NER have been published, which are distributable under a data use agreement (DUA). Both use anonymisation. Kittner et al. [7] goes a step further and shuffles and filters sentences such that a clinical document cannot be reconstructed and that parts with frequent personal information are not distributed. Annotation of both datasets was an iterative and time consuming process to improve inter-annotator agreement due to the aforementioned challenges of medical text. In contrast, Frei and Kramer [1] attempts to circumvent the privacy issue and the costly annotation process by using recent advances in large language models. They generate both the clinical text Table 1 Datasets: The GPTNERMED corpus has been used to train a clinical German NER model [1]. We evaluate the GPTNERMED model using BRONCO150 [7] and its NER annotations for diagnosis, medication and dosage using prompts with manually modified examples of clinical notes sentences, allowing both the data and the corresponding NER model to be made publicly available without DUA. However, it is not yet clear how such data and models trained on it can deal with the idiosyncrasies and heterogeneity of the real clinical setting, as clinical text varies widely between practitioners, departments and institutions. There are also differences in how the text has been pre-processed between different applications, how anonymisation is performed and which parts of clinical documents are included.
We want to evaluate how the approach proposed by Frei and Kramer [1], models trained on a language model generated annotated clinical text, performs on two real-world clinical NER datasets BRONCO150 and CARDIO:DE, which are described in Table 1.
Frei and Kramer [1] first generate 23,411 clinical sentences that are annotated with diagnosis, medication, and dosage entities using the large language model GPT-NeoX and example clinical sentences that have been hand written based on real clinical text. They then fine-tune different pretrained German medical language models based on the BERT architecture [9] to obtain the GPTNERMED model that can extract diagnosis, medication and dosage entities.
The model is published using the spacy library 1 , and we use it to evaluate and train all NER models in our experiments. We always use the best performing public model for GPTNERMED that is based on the German-MedBERT (G-MedBERT) medical language model [10]. For the evaluation on BRONCO150, we use the first predefined 20% split as the test and all the other splits as the training data. For the evaluation on CARDIO:DE, which does not have predefined splits, we use the first 80% of the documents as training and the rest as test data. Entity predictions are evaluated with exact span matches-creating a challenge due to variances in annotation schemes across datasets and even disagreements among human annotators, as seen
Named Entity Recognition Benchmarks on BRONCO150 and CARDIO:DE datasets using GPTNERMED model, baseline setting with fine-tuning pretrained clinical BERT model G-MedBERT on both datasets, and the in-corpus setting with scores reported by dataset authors. For BRONCO150, we only consider the intersection of entity labels diagnosis and medication. For CARDIO:DE, we map labels strength and frequency to dosage, and active ingredient and drug to medication and report the macro average F1 score for in-corpus setting as the authors do not report entity based precision and recall. in lower inter-annotator agreement compared to token based evaluation for both CARDIO:DE and BRONCO150 [8,7]. Despite the difficulty, correct span prediction is crucial for downstream tasks like entity normalization and linking.
Our first experiment evaluates the GPTNERMED model as it is, i.e. without any finetuning on BRONCO150 and CARDIO:DE. As a second experiment, we fine-tune the best performing GPTNERMED base model G-MedBERT with training configuration published by Frei and Kramer [1] on both datasets, which means resulting models have not seen the GPTNERMED data. We also provide the results provided by the respective dataset papers BRONCO150 and CARDIO:DE for comparison [7,8]. In a third experiment, we try different fine-tuning strategies to identify how to best utilize the work of Frei and Kramer [1]. We try following setups: (1) finetuning G-MedBERT, (2) fine-tune GPTNERMED model on BRONCO150, and (3) fine-tune G-MedBERT on both BRONCO and GPTNERMED data.
Results for the GPTNERMED are in Table 2. We observe unsatisfactory performance of off-theshelf GPTNERMED model on real-world datasets BRONCO150 and CARDIO:DE. Results from experiment probing the best fine-tuning strategy using the GPTNERMED data are presented in Table 3. We see that none of the strategies bring any considerable improvement.
When we inspect the GPTNERMED predictions, we see that it predicts 74% of tokens with more than one capital letter consecutively as medication or diagnosis, where as only 1% of these tokens are annotated as medication or diagnosis. False positives include BRONCO150's
Experiments with different fine-tuning strategies. First, we fine-tune the German-MedBERT (G-MedBERT), the base model of GPTNERMED, with exact configuration on BRONCO150. This is the same setting as the baseline in Table 2. Second, we fine-tune the GPTNERMED model on BRONCO150. Third, we fine-tune the G-MedBERT on BRONCO150 and GPTNERMED. In all cases we evaluate with BRONCO150. anonymisation replacement tokens such as "PATIENTen" and "KRANKENHAUS" being predicted as medication and B-SALUTE, B-PERSON being predicted as diagnosis. Some common treatments, e.g. "CT", and diagnosis, e.g. "HCC-suspekten Laesionen" that have capitalized abbreviations are also predicted as medication. Similar problems also exist with dosage predictions in CARDIO:DE as it includes full doctor's reports, whereas GPTNERMED dataset only includes sentences with a mention of medicine and potentially its dosage. This results in a lot of non-medical numerical information in CARDIO:DE, e.g. age and birth date of the patient, are confused with dosage label by GPTNERMED. Of all the tokens with at least one digit, GPTNERMED predicts dosage for 61%, whereas only 5% are labeled as Dosage.
We suspect both covariate and label shift occur with both dataset, which can be potentially addressed by different methods. For a good overview we refer the reader to the review by Hupkes et al. [11].
Generating annotated medical text using large language models can circumvent the privacy issues during dataset curation in clinical domain and address the lack of available training data. However, the evaluations show that current solutions cannot deal with idiosyncrasies of medical text. It is, however, important that published models consider down-stream usecases and perform according evaluations. For a clinical application of a model it is required that it is able to deal with anonymisation and sentences without any medical named entities. Furthermore generated datasets should consider real world scenarios, where there are typos and wrong punctuation.
Funded by Gemeinsame Bundesausschuss (G-BA, 01VSF22041).