This paper describes ReCRF, a named-entity recognition system submitted to the Medical Document Anonymization (MEDDOCAN) challenge in the IberLEF 2019 Workshop. We propose a general method based on a data-driven rule generator and Conditional Random Fields (CRFs) to automatically detect protected health information (PHI) in Spanish medical documents. The reported experiments show that our system achieves a micro-F1 of 96.33% on the test dataset for the rst sub-task and a micro-F1 of 96.86% and 97.50% on strict and merged metrics, respectively, on the test dataset for the second sub-task.
Medical documents containing detailed patients' data are of utmost importance
for research. When healthcare data are associated to individuals, they are
considered protected health information (PHI). The new European General Data
Protection Regulation (GDPR) [
In 2006 and 2014, i2b2 organized two shared tasks on document
anonymization [
only. IberLEF 2019 organizes the rst community challenge task speci cally
devoted to the anonymization of Spanish medical documents, called the
MEDDOCAN task [
The remainder of the paper is organized as follows. In Section 2 we brie y describe the data to be anonymized. Section 3 describes the methodology we propose. Results and discussions are presented in Section 4. Section 5 presents the conclusions and depicts some lines of future work. 2
The MEDDOCAN challenge task aims at identifying and extracting several types of PHI categories from plain text medical documents. The PHI categories are grouped into eight main categories with 22 sub-categories. The corpora released for the tasks consists of 1000 documents, divided into: 500 as training data, 250 as development data and 250 as test data. The distributions of PHI categories and sub-categories in the training, development and test data are shown in Table 1.
3
We developed an automatic system to detect PHI categories from Spanish medical documents. The next subsections describe the steps followed to train and use the system. 3.1
In this step, we tokenize the text at two levels: sentence-level and word-level.
First, a sentence tokenizer takes a single document as input and produces list
of sentences. Afterwards, we split each single sentence into a list of tokens. The
sentence tokenizer is based on newline delimiter whereas a manually-crafted
regular expression based tokenizer and a spaCy pre-trained model for Spanish[
In this step, we developed a data-driven regular expression generator so that we avoid implementing hand crafted regular expression rules. This generator analyses all the appearances of the PHI categories in the training data set and, from that, it generates rules to detect those categories. These rules are later used to extract sudo-labelled tokens that are used to guide the CRF tagger in taking the nal decision. 3.3
We extract a wide variety of linguistic features, similarly to previous studies [
Extracting these features from just the target word does not consider the context in which the word appears, which may lead to misclassifying tokens due to language ambiguity. To tackle this, we consider a window of 5 words centered at the target word (i.e., the two words on the left and the two words on the the right). 3.4
We used both a set of automatically-crafted rules (RegEx model) and
Conditional Random Fields[
Annotated Documents
Document Rules Generation
RegEx Rules
Annotated Documents
Tags
Document Text Tokenization Feature Extraction Training the System
CRF Model
RegEx
Rules
External Resources
Fig 2 shows that our system has two outputs: the RegEx model and the CRF model. The RegEx model is built by the automatic rule extractor by analyzing the PHI categories that appear in well-structured contexts (e.g. Nombre: Xxxxx., Fecha de nacimiento: dd/mm/yyyy.).
The CRF model is trained by passing all the extracted features from the tokens plus the decision of the RegEx model which add extra information and make the decision easier for the CRF model.
Unannotated
Documents RegEx
Rules External Resources
Annotated Documents
Document Text Tokenization Feature Extraction
CRF Model
Document Fig 3 shows how both RE and CRF models are used to make the annotations. Even though the RegEx model is accurate enough to detect well-structured entities, it is not e ective in front of small changes in the text format. So, we decided to use the RegEx model to perform a preliminary annotation, which is then passed to the CRF model that will make the nal decision. 4
The performance of the detection of PHI categories has been evaluated using Precision, Recall and F1 scores at the entity level. The results of our system on the test set for the di erent PHI categories are shown in Table 2; the confusion matrix is shown in Table 3. Notice that categories that have low frequency in the training dataset have less F1 score (e.g. NUMERO FAX appears only 15 times in the training set and CENTRO SALUD appears six times). This result is expected because the model didn't get enough examples in order to learn how to accurately detect them.
The overall results of our system for the two sub-tracks of the competition on the development and test datasets are shown in Table 4. We see very small di erences in F1 scores between the development and test datasets. This proves that our system generalizes well in front of new data. 5
We presented a hybrid system that automatically detects PHI entities from plain text medical documents. The system consists of an automatically constructed RegEx model and a trained CRF model. The design of the system, which includes using a variety of linguistic and semantic features to increase the accuracy, ensures that it generalizes well in front of new data.
Finally, because of the rules that we get from the automatic RegEx generator are not fully generalized, in future work, we plan to implement an automatic optimizer to get a better result.
This work was partly supported by the European Commission (project H2020700540 "CANVAS"), the Government of Catalonia (ICREA Academia Prize to J. Domingo-Ferrer and grant 2017 SGR 705) and the Spanish Government (projects RTI2018-095094-B-C2 \CONSENT" and TIN2016-80250-R \Sec-MCloud"). While the authors are with the UNESCO Chair in Data Privacy, the opinions expressed in this paper are the authors' own and do not necessarily re ect the views of UNESCO.