Copyright c 2019 for this paper by its authors. Use permitted under CreativeCommons License Attribution 4.0 International (CC BY 4.0). IberLEF 2019, 24 September 2019, Bilbao, Spain.
De-identification is a prerequisite of clinical record accessing and sharing outside of
hospitals, which is very important for secondary use of clinical data. In the past few
years, de-identification had attracted plenty of attention and a large number of efforts
had been made for de-identification, especially for clinical documents in English. The
representative works are natural language processing (NLP) challenges including the
de-identification task of clinical text, such as the i2b2 (the Center of Informatics for
Integrating Biology and Bedside) 2006 [
In 2019, Martin Krallinger et al. organized a challenge task special for the
deidentification of medical documents in Spanish, called the MEDDOCAN (Medical
Document Anonymization) task [
The overview architecture of our system for the MEDDOCAN task is shown Fig.1.
We first tokenized raw clinical texts in Spanish, and then deployed two individual
deep learning methods (i.e., BERT+CRF and flair) for de-identification respectively.
Our system was described below in detail.
The organizers of the MEDDOCAN task provided participants with a synthetic
corpus of 1000 discharge summaries and medical genetics clinical records manually
annotated by medical experts according to a guideline defining 22 types of PHI. The
corpus were divided into three parts: a training set of 500 records with 11,333 PHI
mentions, a development set of 250 records with 5801 PHI mentions, and a test set of
250 records with 5661 PHI mentions. The test set was embedded in a background set
of 3751 clinical records that have been manually split into sentences. The statistics of
the corpus, including number of documents, sentences and PHI mentions are listed in
Table 1, where “NA” denotes unknown.
Sentence split and tokenization are two important preprocessing steps for natural
language processing (NLP). We developed a simple rule-based system for sentence split
and tokenization. A document was split into sentences by ‘;’, ‘?’, ‘!’, ‘\n’ or ‘.’ not in
numbers, and each sentence was tokenized by the method proposed by Liu et al. [
De-identification is a typical named entity recognition problem, which is usually recognized as a sequence labeling problem. In this study, we deployed two deep learning methods for the MEDDOCAN task, that is, BERT+CRF and flair as follows: BERT+CRF. a method that appends a condition random field (CRF) layer to BERT. In our study, we compared the cases using different settings.
Flair. a sequence labeling method based on contextual string embeddings. 2.4
As clinical records in the test set have been manually split into sentence, to fixed errors caused by sentence split, we mapped the split sentences back to the gold ones and combined the neighbor PHI mentions of the same type together. 2.5
All system performance was measured by micro-average precisions (P), recalls (R), and F1-scores (F1) under three criteria: “strict” at entity level (track 1), “strict” at span level (track 2), and “merged” at span level, where “strict” at entity level checks whether a recognized PHI mention exactly matches a gold one of the same type, “strict” at span level checks whether a recognized PHI mention has the same span as a gold one no matter their types, and “merged” at span level is a “strict” at span level after merging the spans of PHI mentions connected by non-alphanumerical characters. All evaluations were conducted on the independent test data set, and the measures were calculated by the tool provided by the MEDDOCAN organizers. 2.6
In this study, PHI mentions were represented by “BIO” (B-beginning of a PHI mention, I-insider a PHI mention, O-outside a PHI mention). The hyper-parameters and parameter estimation algorithm listed in Table 2 were used in the deep learning methods. The pre-trained neural language models (https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H768_A-12.zip and https://github.com/zalandoresearch/flair) were used in BERT+CRF and flair respectively. The other parameters were optimized on the development set, and all models are evaluated on the independent test set. The micro-average precisions, recalls and F1-scores of our system under the three criteria were listed in Table 3. BERT+CRF outperformed flair by about 0.3% in F1scores because of higher recalls. When POS features were added, the performance of BERT+CRF decreased a little bit. When we further fine-tuned BERT+CRF on the combination of training and development sets, BERT+CRF did not change very much. Our system achieved the highest “strict” F1-score of 0.9646 at entity level, a “strict” F1-score of 0.97 at span level and a “merged” F1-score of 0.9821 at span level. The new results (shown in Table 3) reported here are the results of our first submissions (shown in Table 4) after post-processing. The great differences between “strict” F1-scores (track 2) and “merged” F1-scores inspired us to find errors caused by sentence split. For example, in sentence ”Domicilio: Av. de Jaén, 28.”, “Av. de Jaén, 28” is a entity of “CALLE”, but was split into two entities of “CALLE”: “Av.” and “de Jaén, 28” as the sentence were split into two sentence “Domicilio: Av.” and “de Jaén, 28.” by ‘.’. The sentence split errors result in an F1-score difference of about 0.4 between “strict” F1-scores and “merged” F1-scores. We can see that the post-processing module brings a “strict” F1-score gain of 0.0245 for track 1 and a “strict” F1-score gain of 0.0243 for track 2. The differences between “strict” F1-scores (track 2) and “merged” F1-scores decrease sharply when the post-processing module is added.
To analyze errors in our system, we evaluated the performance on each category of entity and found that the F1-scores on “PROFESION” and “INSTITUCION” are much lower than other categories except “OTROS_SUJETO_ASISTENCIA”, on which the F1-score is zero. There are main three reasons why these three categories of entities are not well recognized. Firstly, entities in some categories are too few. For example, there are only 15 entities of “OTROS_SUJETO_ASISTENCIA” in the training set and development set in all, and only 7 in the test set. Secondly, entities of “INSTITUCION” vary greatly in format. Thirdly, there may be some entities wrongly labeled as gold standards. For example, “militar” and “ex-operario de industria textil”, which means “soldier” and “ex-textile industry operator” respectively, are recognized by our system but not labeled as gold standards. 5
In this study, we developed a deep learning-based system for the MEDDOCAN task, a challenge special for de-identification of clinical text in Spanish. The system achieves a promising performance. Besides, “BERT+CRF” outperforms flair. In the future, we will investigate whether BERT and flair can be combined together for further improvement.
This paper is supported in part by grants: NSFCs (National Natural Science Foundations of China) (U1813215, 61876052 and 61573118), National Key Research and Development Program of China (2017YFB0802204), Special Foundation for Technology Research Program of Guangdong Province (2015B010131010), Strategic Emerging Industry Development Special Funds of Shenzhen (JCYJ20170307150528934 and JCYJ20180306172232154), Innovation Fund of Harbin Institute of Technology (HIT.NSRIF.2017052).