Information extraction of medical reports is key in order to improve timely discoveries of ndings and as an aid to improve decisions about medical treatments and budget. In order to develop information extraction methods, medical data has to be available. Since this data is extremely sensitive due to the presence of personal information, report de-identi cation is needed. We present two methods, a window classi er and an implementation of conditional random elds (CRF) in order to de-identify personal information of Spanish medical records provided by the MEDDOCAN challenge. CRF obtained the best results with a F1measure of 0.897 for named entity recognition with exact match (subtask 1), and 0.930 and 0.940 for inexact match (subtask 2 strict and merged respectively).
Last years' exponential growth of available biomedical texts and the contribution of structured information to the triggering of automatic alerts about urgent situations, to performing better diagnosis, and as input to clinical decision support systems among others, has led to the need of automatically extracting information through the use of disciplines such as natural language processing (NLP) and information extraction (IE). The area of study that deals with information extraction from biomedical texts is called BioNLP or biomedical text mining.
Medical data is of sensitive nature, due to the presence of personal information. Therefore, in order to process medical reports, they have to be anonymized.
In the clinical domain, anonymization or de-identi cation is the process of removing from medical records all information that could identify a patient or the physician performing the study or diagnosis. In some cases, names and patients ? All authors contributed equally to the work.
Many factors have to be taken into account when anonymizing medical
records. See [
Some de-identi cation challenges have been organized in the past, for
example i2b2 2006 De-identi cation and Smoking Challenge [
In this paper we present two methods we have developed for MEDDOCAN
[
The MEDDOCAN task was structured into two sub-tasks: 1) NER o set and entity type classi cation, and 2) sensitive token detection.
A synthetic corpus of 1000 clinical case studies augmented with protected
health information (PHI) from discharge summaries and medical genetics
clinical records was provided. Additionally, a number of linguistic resources were
supplied:11
{ AbreMES-DB,12 a Spanish medical abbreviation database, that contains
abbreviations and their potential de nitions automatically extracted from
the metadata of titles and abstracts of biomedical publications written in
Spanish,
5 https://www.hhs.gov/hipaa/index.html
6 https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:
31995L0046:en:HTML
7 http://servicios.infoleg.gob.ar/infolegInternet/anexos/160000-164999/
160432/texact.htm, http://servicios.infoleg.gob.ar/infolegInternet/
verNorma.do?id=19429,http://servicios.infoleg.gob.ar/infolegInternet/
anexos/60000-64999/64790/texact.htm
8 https://www.i2b2.org/NLP/HeartDisease/
9 MEDDOCAN: http://temu.bsc.es/meddocan/.
10 IBERLEF 2019: https://sites.google.com/view/iberlef-2019/.
11 http://temu.bsc.es/meddocan/index.php/resources/
12 https://github.com/PlanTL-SANIDAD/AbreMES-DB
{ MEDDOCAN-Gazetteer, composed of MEDDOCAN related entities. It
includes names, surnames, addresses, hospitals, professions, and di erent
types of locations (provinces, cities and towns, among others),
{ SPACCC POS-TAGGER,13 a Part-of-Speech tagger for Spanish texts in
the medical domain based on FreeLing [
The rest of the paper is organized as follows. Section 2 presents previous work of anonymization in the biomedical domain. Section 3 presents the data set used, the implemented methods and the performance evaluation metrics used. Section 4 presents the results of our methods and its analysis. Finally, Section 5 presents conclusions and future work. 2
In this section we describe the previous work in anonymization in the biomedical domain.
Emam [
A tutorial on Privacy Challenges and Solutions for Medical Data Sharing was
organized by IBM Research Zurich and Cardi University in 2011.15
GkoulalasDivanis et al. [
Emam et al. [
Many other analysis and surveys on the subject [
Rules and regular expressions have been used, eg. for medical reports in
Spanish [
3.1
Data This section presents the datasets used, the evaluation metrics, the preprocessing of data and the two proposed methods. A ow chart gure for the used processing pipeline can be seen in Figure 1.
We used the MEDDOCAN corpus, a synthetic corpus composed of 1,000 clinical cases manually selected by a practicing physician and enriched with PHI expressions extracted from discharge summaries and genetics clinical records. It has around 33,000 sentences, with an average of around 33 sentences per clinical case, summing up 495,000 words and an average of 494 words per clinical case.
The corpus was divided into three subsets. The training set comprises 500 clinical cases, and the development and test set 250 clinical cases each. The test set was not provided until the end of the challenge.
An additional test set, that includes the actual test set, the training, the development set, and a background set -of 2,000 documents-, composed of 3,751 clinical cases was provided in order to test the submissions.16 The actual test set (250 clinical cases) is called test set with Gold Standard Annotations and the test set provided before the end of the challenge (3,751 clinical cases) is called test set (including background set).
A conversion script between BRAT stando annotation format17 and i2b2 annotation format was provided.18
The annotation schema used by the challenge organizers to create the dataset de nes 29 entities and was inspired by the annotation schema used for the i2b2 de-identi cation tracks, adapting it to the speci cities of the MEDDOCAN document collection. An iterative process was followed until the nal guidelines were obtained. The inter-annotation agreement (IAA) calculated on a set of 50 double-annotated records is reported as 98% in exact match.19 The dataset annotation guidelines can be downloaded.20 3.2
Evaluation metrics Given that di erent uses might require di erent balance in terms of precision and recall, two di erent sub-tasks have been proposed: sub-task 1 (named entity recognition o set and entity type classi cation) and sub-task 2 (sensitive span detection). Sub-task 1 consists in exact match (ie. prediction and gold standard 16 Datasets can be found in http://temu.bsc.es/meddocan/index.php/data/. 17 https://brat.nlplab.org/standoff.html 18 https://github.com/PlanTL-SANIDAD/MEDDOCAN-Format-Converter-Script 19 An abstract of the annotation process and schema can be seen in http://temu.bsc.
es/meddocan/index.php/annotation-guidelines/. 20 http://temu.bsc.es/meddocan/wp-content/uploads/2019/02/guADas-de-anotacinde-informacin-de-salud-protegida.pdf
Window Classi ers and CRF for Medical Report De-Identi cation have to begin and end in the same o set and also have the same classi cation). Sub-task 2 allows inexact match and possibly wrong entity types, evaluating whether spans belonging to sensitive phrases are detected correctly. Sub-task 2 has two evaluations: strict and merged.
Micro-averaged precision, recall, and F1 score are used to evaluate both subtasks (see de nition below). Micro-balanced metrics calculate each metric over all the classes together.
P recision(P ) =
Recall(R) =
F 1 = Leaks =
T P T P + F P
T P T P + F N 2P R P + R
F N #sentences where TP are the number of true positives, FP the number of false positives, and FN the number of false negatives.
Two additional metrics were computed to evaluate the best methods of the tasks: leaks for sub-task 1 and merged for sub-task2. The rst computes the proportion of PHIs that were not detected. The merged results of subtask 2 merges the spans of PHIs connected by non-alphanumerical characters before evaluating the results. 3.3
Data Preparation We rst use standard text preprocessing techniques such as sentence segmentation and word tokenization. Then, we use the BIO encoding, a standard technique that models Named Entity Recognition as a sequence tagging problem. In the BIO encoding, each entity type <T> has labels B-<T> and I-<T> to mark, respectively, those tokens as the beginning or the continuation of an entity. The label O is used to tag \outside" tokens, this is, tokens that do not belong to any entity.
We also do Part-of-Speech tagging using our own classi er-based PoS
tagger.21 The tagger was trained using the AnCora 3.0.1 Spanish corpus [
Window Classi er The window classi er consists in the use of a classi er to tag each token in the text based on information of the token itself and of tokens in a xed window 21 Unpublished work. 22 https://stanfordnlp.github.io/CoreNLP/ surrounding it. Our current system uses the two previous tokens and the next token, this is, a window of size four with the center word at the third position.
Features for all tokens include the lowercased token, a pre x and a su x of
length 2, the PoS tag, and boolean features indicating if the word is in upper
case and if it is only composed of digits. For the center word, a word embedding
is also included as a feature. Here we use 300-dimension fastText embeddings,23
pre-trained on Spanish Wikipedia [
For the classi er, in this work, we use a Support Vector Machine (SVM) with
a linear kernel, as provided by scikit-learn [
As tags are independent to each other, the tagger may output sequences inconsistent with the BIO representation, such as an I-<T> tag after an O tag. To x this, we introduce a post-processing step, that converts all inconsistent I-<T> tags to B-<T> tags. 3.5
Conditional Random Fields Conditional Random Fields (CRFs) are probabilistic models used to predict sequences of labels, based on sequences of input samples.
Each token has an associated vector of features, such as the words' part of speech tag and the words' su x of a given length. The input of a CRF is the sequence of tokens of the text. The features of a token and the pattern of labels assigned to previous words are used to determine the most likely label for the current token. In a linear chain CRF, only the label of the previous token is used.
As features we used the word in lower case, a pre x of length 3, a su x of length 4, the truth value of whether the word is all written as upper case if all 23 https://fasttext.cc/
Window Classi ers and CRF for Medical Report De-Identi cation are digits, PoS tag and reduced PoS tag of length 2. Also, one word before and one word after were considered.
For the implementation, CRFsuite [
A single run was submitted for each method. Results for both methods can
be seen in Table 1. Our results are compared to the results obtained by other
MEDDOCAN participating teams in [
Table 2 displays classes with lowest F1 and a minimum of 50 instances and Figure 2 presents the confusion matrix of a subset of the named entities.
We can observe some common errors: e.g, TERRITORIO (territory) is tagged frequently as CALLE (street) or as some of the0 ID entities (postal codes are annotated as TERRITORIO ). Another type of common error is to misclassify NOMBRE SUJETO ASISTENCIA as NOMBRE PERSONAL SANITARIO (and viceversa); this is probably due to the fact that both refer to proper nouns. A model with more features (and more sophisticated, such as word embeddings) and eventually a di erent window size would probably overcome these problems.
We also noticed some annotation errors {such as Madrid and Mendieta Espinosa{ that were not annotated as entities. This kind of errors can in uence negatively the performance of the learning algorithm. Moreover, entities correctly discovered are erroneously computed as errors.
As future work we plan to include in our models some of the linguistic resources provided by the challenge: namely, AbreMES-DB, a Spanish medical abbreviation database; the MEDDOCAN gazetteer, a gazetteer that includes names, surnames, addresses, hospitals, professions, and di erent types of locations, such as provinces, cities and towns. We tried to integrate them to our system, but it resulted in degraded performance. We also plan to include the use of regular expressions for detecting e-mails and phone numbers. More feature engineering in CRF should also be performed.
Another line of work to tackle NER could be the use of recurrent neural networks (RNN); in particular, bidirectional RNNs. We trained a biLSTM but, due to the small size of the dataset, the model failed to converge.
Since we did not use the Spanish abbreviation database nor the MEDDOCAN gazetteer the only components that takes language into account are our PoS tagger (trained with the AnCora Spanish corpus) and the word embeddings. Therefore, we think that our methods could be easily adapted to other languages. Regarding Spanish, although there are many di erences in di erent countries (e.g. the Rioplatense Spanish of Argentina and Uruguay) the entities of interest in this task, that were recognized with our methods should have similar results with Spanish of di erent regions of the world. References
Window Classi ers and CRF for Medical Report De-Identi cation