There is an increasing interest in exploiting the content of electronic health records by means of natural language processing and text-mining technologies, as they can result in resources for improving patient health/safety, aid in clinical decision making, facilitate drug repurposing or precision medicine. To share, re-distribute and make clinical narratives accessible for text mining research purposes, it is key to fulll legal conditions and address restrictions related data protection and patient privacy. Thus, clinical records cannot be shared directly "as is". A necessary precondition for accessing clinical records outside of hospitals is their de-identi cation or exhaustive removal/replacement of all mentioned privacy related protected health information phrases. Providing a proper evaluation scenario for automatic anonymization tools is key for approval of data redistribution. The construction of manually de-identi ed medical records is currently the main rate and cost-limiting step for secondary use applications of clinical data. This paper summarizes the settings, data and results of the rst shared track on anonymization of medical documents in Spanish, the MEDDOCAN (Medical Document Anonymization) track. This track relied on a carefully constructed synthetic corpus of clinical case documents, the MEDDOCAN corpus, following annotation guidelines for sensitive data based on the analysis of the EU General Data Protection Regulation. A total of 18 teams (from the 51 registrations) submitted 63 runs for rst sub-track 1 and 61 systems for the second sub-track. The top scoring systems were based on sophisticated deep learning approaches, representing strategies that can signi cantly reduce time and costs associated to accessing textual data containing privacy-related sensitive information. The results of this track might help in lowering the clinical data access hurdle for Spanish language technology developers, showing also potentials for similar settings using data in other languages or from di erent domains.
1
There is an increasing interest in exploiting the content of unstructured clinical
narratives by means of language technologies. Therefore, and because there is
clear interest in the health sector by the language technology industry, one of
the agship projects of the Spanish National Plan for the Advancement of
Language Technology (Plan TL4) is related to the clinical and biomedical eld. The
Plan TL has promoted the generation of a collection of resources for Spanish
biomedical NLP5, including corpora [
Being able to transform automatically clinical documents into some structured representations is nonetheless needed to enable secondary use of EHRs to carry out population and epidemiological studies, to detect medication-related adverse events or for monitoring systematically treatment-related responses, just to name a few.
To be able to share, re-distribute and make clinical narratives accessible for
text mining and natural language processing (NLP) purposes, it is key to ful ll
legal conditions and address restrictions related data protection and patient
privacy legislations [
MEDDOCAN: Automatic de-identi cation of medical texts in Spanish
for the construction of de-identi ed textual corpora for research purposes [
Studies describing services for pseudonymization of EHRs based on
standards such as the ISO/EN 13606 were previously published for data in Spanish
[
The practical relevance of anonymization or de-identi cation of clinical texts
motivated the proposal of two shared tasks, the 2006 and 2014 de-identi cation
tracks [
In case of texts in Spanish, there has been so far a rather limited attempt
in developing and characterizing automatic de-identi cation strategies [
The MEDDOCAN track was one of the nine challenge tracks of the Iberian Languages Evaluation Forum (IberLEF 2019)6 evaluation campaign, which had the goal of promoting the development of language technologies for Iberian languages. MEDDOCAN was the rst community challenge track speci cally devoted to the anonymization of medical documents in Spanish and it evaluated the performance of the systems for identifying and classifying sensitive information in clinical case studies written in Spanish.
The evaluation of automatic predictions for this track had two di erent scenarios or sub-tracks: 1. NER o set and entity type classi cation : the rst sub-track was focused on the identi cation and classi cation of sensitive information (e.g., patient names, telephones, addresses, etc.). 2. Sensitive span detection: the second sub-track was focused on the detection of sensitive text more speci c to the practical scenario necessary for the release of de-identi ed clinical documents, where the objective is to identify and to mask con dential data, regardless of the real type of entity or the correct identi cation of PHI type. 2.2
For this track, we prepared a synthetic corpus of clinical cases enriched with PHI expressions, named the MEDDOCAN corpus. The MEDDOCAN corpus, of 1,000 clinical case studies, was selected manually by a practicing physician and augmented with PHI phrases by health documentalists, adding PHI information from discharge summaries and medical genetics clinical records.
To carry out the manual annotation, we constructed the rst public
guidelines for PHI in Spanish [
The MEDDOCAN corpus was randomly sampled into three subset: the train set, which contained 500 clinical cases, and the development and test sets of 250 clinical cases each. These clinical cases were manually annotated using a customized version of AnnotateIt. Then, the BRAT annotation toolkit was used to 6 http://hitz.eus/sepln2019/?q=node/21 correct errors and add missing annotations, achieving an inter-annotator agreement (IAA) of 98% (calculated with 50 documents). Together with the test set, we released an additional collection of 3,501 documents (background set7) to make sure that participating teams were not able to do manual corrections and also to promote that these systems would potentially be able to scale to larger data collections.
The MEDDOCAN annotation guidelines de ned a total of 29 entity types. track and the number of occurrences among the training, development and test sets.
The MEDDOCAN corpus was distributed in plain text in UTF-8 encoding, where each clinical case was stored as a single le, while PHI annotations were released in the BRAT format, which makes visualization of results straightforward, as you can see in Fig. 1 For this track, we also prepared a conversion script8 between the BRAT annotation format and the annotation format used by the 7 The background set included the train, development and test sets, and an additional collection of 2,751 clinical cases (totalling 3,751 clinical cases). 8 https://github.com/PlanTL-SANIDAD/MEDDOCAN-Format-Converter-Script
Marimon et al. previous i2b2 e ort, to make comparison and adaptation of previous systems used for English texts easier. We developed an evaluation script that supported the evaluation of the predictions of the participating teams. For both sub-tracks the primary evaluation metrics used consisted of standard measures from the NLP community, namely micro-averaged precision, recall, and balanced F-score, being the last one the only o cial evaluation measure of both sub-tracks:
T P Precision: P = T P +F P
T P Recall: R = T P +F N
To participate in the MEDDOCAN track it was necessary to register both on the o cial website9 and in the CodaLab competition10. Training and development sets were made available for download on the o cial website11, and the evaluation script was uploaded to GitHub12, to ensure a transparent evaluation.
Submissions had to be provided in a prede ned prediction format (BRAT or i2b2). The participants had a period of almost two months to develop their system. In the middle of this period, the text and background sets were released with the 3,751 documents that the participants had to process and label, although the nal evaluation was done on the 250 documents of the test set. As we have mentioned, the participants could submit a maximum of 5 system runs, and, once the submission deadline expired, we published the Gold Standard annotations of the test set, in order to ensure a transparent evaluation process.
A total of 18 teams participated in the track, submitting a total of 63 systems for sub-track 1 and 61 systems for sub-track 2. Teams from eight di erent nationalities participated in the track: ten from Spain, two from the United States, and one from Argentina, China, Germany, Italy, Japan, and Russia. Among all the participants, only one belonged to an institution of a commercial nature. Table 2 summarizes the most relevant information about the participants. 3.2
We produced a baseline system using a vocabulary transfer approach. Each annotation from the train and development datasets was transferred to the test dataset using strict string matching. For those cases where the text was the same, but the entity type was di erent, we decided to annotate all entity types that matched that text. 9 http://temu.bsc.es/meddocan/ 10 https://competitions.codalab.org/competitions/22643 11 http://temu.bsc.es/meddocan/index.php/data/ 12 https://github.com/PlanTL-SANIDAD/MEDDOCAN-CODALAB-EvaluationScript Marimon et al.
MEDDOCAN: Automatic de-identi cation of medical texts in Spanish set and entity type classi cation. with a recall of 0.98335, lukas.lange, with a recall of 0.98264, and, mhjabreel, with a recall of 0.97471.
An analysis of errors showed that some of the annotations in the Gold Standard (GS) corpus were not detected by any of the systems (at least not exactly). Some of them are listed here: { HOSPITAL: Hospital General de Agudos P. Pin~ero { FAMILIARES SUJETO
{ OTROS SUJETO
On the contrary, some systems annotated entities that were not in the GS but probably should be. For instance, "ex-operario de la industria textil " was annotated as PROFESION
by jiangdehuan, jimblair, and Jordi, but this annotation was not in the GS. MEDDOCAN: Automatic de-identi cation of medical texts in Spanish One of the primary goals of this track was to develop systems capable of completely de-identifying sensitive information from clinical documents. However, none of submitted systems managed to obfuscate all the sensitive information. In this section, we present two experiments we performed that evaluated the performance of combined systems to de-identify the test dataset without leaks. The rst experiment was based on a joint system, the second experiment, on a voting system.
Joint system The goal of this experiment was to nd the combination of individual systems that achieved the best possible performance. For this, rst, we ranked all the systems by F-score, and then we joined the annotations of the two best system. If the performance of the Joint system improved, we continued with the next best system, if not, we kept the previous system (or the previous joint system). We repeated this until no systems were left. We measured the performance of the joint system using three metrics: 1. Best F1: If the F-score of the joint system improved when we added the annotations from the next system, we updated the joint system with the new one. If the F-score did not improve, but it was maintained and the recall was better, we also updated the joint system with the new one (same F-score, better recall, worse precision). 2. Best Recall: If the recall of the joint system improved, we updated the joint system, regardless of the drop in the F-score. It tried to maximize the chances of completely de-identifying the documents. 3. Balanced: If the recall of the joint system improved, we updated the joint system only if the decrease of the F-score was at much four times the increase
MEDDOCAN: Automatic de-identi cation of medical texts in Spanish
Marimon et al. of the recall. That it, for every point of increase in recall, we allowed 4 point of decrease in F-score, but not more. It tried to increase the recall, but without hurting the F-Score too much.
The systems that were used to achieve the best results for these metrics were the following: { Best F1: { Recall: lukas.lange/run3 improves the F-score from 0 a 0.96961. lukas.lange/run2 improves the F-score from 0.96961 a 0.96997. lukas.lange/run1 improves the F-score from 0.96997 a 0.97033. lukas.lange/run3 improves the recall from 0 to 0.96944. lukas.lange/run2 improves the recall from 0.96944 to 0.97209. lukas.lange/run1 improves the recall from 0.97209 to 0.97492. lukas.lange/run4 improves the recall from 0.97492 to 0.97562. Fadi/15-7 improves the recall from 0.97562 to 0.97898.
Fadi/14-5 improves the recall from 0.97898 to 0.97951.
Fadi/17-3 improves the recall from 0.97951 to 0.98022.
Fadi/16-3 improves the recall from 0.98022 to 0.98039. nperez/ncrfpp improves the recall from 0.98039 to 0.98181.
FSL/run1 improves the recall from 0.98181 to 0.98393.
FSL/run2 improves the recall from 0.98393 to 0.9841. nperez/sp-test-03-empty improves the recall from 0.9841 to 0.98516. mhjabreel/run3 improves the recall from 0.98516 to 0.98551. mhjabreel/run2 improves the recall from 0.98551 to 0.98569. jiangdehuan/run3 improves the recall from 0.98569 to 0.98693. jiangdehuan/run2 improves the recall from 0.98693 to 0.9871. jimblair/run2 improves the recall from 0.9871 to 0.98763. jimblair/run3 improves the recall from 0.98763 to 0.98781. jiangdehuan/run1 improves the recall from 0.98781 to 0.98816. Jordi/run3 improves the recall from 0.98816 to 0.98869.
vcotik/run5 improves the recall from 0.98869 to 0.98887. { Balanced: lukas.lange/run3 improves the recall from 0 to 0.96944 (+0.96944) without losing too much F-score: 0.96961 (-0.96961). lukas.lange/run2 improves the recall from 0.96944 to 0.97209 (+0.00265) without losing too much F-score: 0.96841 (0.00112). lukas.lange/run1 improves the recall from 0.97209 to 0.97492 (+0.00283) without losing too much F-score: 0.96647 (0.00194).
Fadi/15-7 improves the recall from 0.97492 to 0.97863 (+0.00371) without losing too much F-score: 0.96181 (0.00466).
Fadi/17-3 improves the recall from 0.97863 to 0.97951 (+0.00088) without losing too much F-score: 0.95868 (0.00313). nperez/ncrfpp improves the recall from 0.97951 to 0.98128 (+0.00177) without losing too much F-score: 0.95308 (0.00560).
FSL/run1 improves the recall from 0.98128 to 0.98375 (+0.00247) without losing too much F-score: 0.94342 (0.00966).
MEDDOCAN: Automatic de-identi cation of medical texts in Spanish Voting The combination of individual systems from the previous experiment was done directly on the test set. It is very di cult for a given combination of systems to be transferable from one data set to another. Therefore, it should be taken as only an approximation of the upper bound that can be obtained by combining individual systems. In this experiment, we combined the systems using a voting scenario: we accepted as good the annotations that had predicted by N systems.
We created 50 systems for sub-track 1. The rst system accepted all the annotations predicted by, at least, one of the systems, while the last one accepted only the annotations that were predicted by, at least, 50 systems. The results of this experiment is shown in Table 9. As expected, as the value of N increased (we increased the number of required votes), the recall got worse and the precision improved. The maximum value of F-score on the train and development sets was obtained combining 17 systems (F-score of 0.9942). When we used the train and development sets as train corpus to select the optimal value of N and used this value on the test set, we obtained an F-score of 0.9757. This score was lower than the best one that could be obtained (0.9768, with N = 23), but the di erence was (in practice) negligible.
Comparing the results of the two experiments, we see that the voting system improved the joint system by 0.54 points. In addition, as we see in the Table 9, the values were very stable and a non-optimal choice of the value N did not vary much the result. The negative part was that the voting scenario required many systems to obtain this result (17 systems out of 63 had to agree in order to accept an annotation), while the joint system was a combination of only 3 systems. The voting system matched the performance of the joint system when N is 13, scoring 0.9701 (the joint system scored 0.9703) .
For reasons of space, we do not include the results of this experiment for sub-tracks 2A and 2B, but they showed a very similar behavior. In this section we analyze the performance of the systems on the di erent data sets. As we have said, the background set included, the train set and the develMarimon et al. opment set, which allowed us to measure the F-score of all the systems on the train, development and test set, and to analyze their behavior.
All the scores of this analysis are shown in table 10, where the drop column indicates the di erence of performance in the test set with respect to the development set (a negative value indicates a lower performance on the test set). There were two teams that achieved a F-score of 1.0 in both train and development set: jimblair (in all tracks) and m. domrachev (in sub-tracks 1 and 2A). The former had a performance drop of 6.25 points, and the latter of 9.99 points in the test set, probably because both systems of these competitors memorized the train and development data, obtaining a perfect score, incurring in over tting. This also suggested that they could have used the development set to train the system, and not just to tune it.
In contrast to this, we see that lukas.lange, which was rst team on the test set for sub-track 1, was also the rst on the development set (without taking into account those who had scored 1.0), but third on the train set (without taking into account those who scored 1.0). The performance of their system only dropped 0.14 points in the test set with respect to the development set. Probably they used the train set to build the system and the development only for tuning, not incurring in over tting. This demonstrated that the ability of the systems to generalize was very important.
Taking into account all the sub-tracks, the maximum performance drop was su ered by m.domrachev, losing 9.99 points in sub-track 1. Without taking into account those who had scores 1.0 on the development set, the system that lost more points was the one submitted by Jordi, which lost 5.25 points on track 2B (0.33 points in sub-track 1 ,and 0.29 points in sub-track 2A). The next participants with the highest loss of performance were VSP and FSL.
The maximum improvement in the test set with respect to the development set was 3.32 points, corresponding to the system submitted by jiangdehuan, in track 2A.
As a curiosity, ccolon scored exactly the same result on the development and test set. However, its performance decreased with respect to the train set (by 3.77 points). 4
The MEDDOCAN track attracted a considerable number of teams, not only from Spain, but also from other countries, stressing the global interest in solving the clinical data access hurdles and assuring patient data privacy requirements. Compared to previous e orts for English, namely the i2b2 de-identi cation tracks, MEDDOCAN could even reach a higher number of participation. It is important to point out that the MEDDOCAN track bene ted signi cantly from the experiences, setting and annotation process pioneered by the i2b2 e orts.
In case of the 2006 i2b2 shared task [
De-identi cation is still a very hard task, because for the special characteristics of clinical texts and the importance of recall, i.e. avoiding leakage of sensitive information. The top three teams are above 0.96 in F-score, for the track based on entity detection.
The top scoring systems make use of the most cutting-edge NLP techniques, i.e. exploiting Deep Learning. Their results are comparable to single manual anonymization done by humans. Automatic anonymization with manual revision to detect potential leakages might result in anonymized Spanish clinical records that allow data redistribution. Nevertheless, a follow up task, using real EHRs from various healthcare institutions, and assessing the practical user scenario with experts in the loop would be desirable to quantify also cost reduction and bene ts of the quality of anonymization strategies assisted by automated tools. 5
The results of the MEDDOCAN shared task and evaluation e ort on automatic
de-identi cation of sensitive information from texts in Spanish show that
advanced deep learning approaches in combination with rule based systems and
gazetteer resources can provide very competitive results when a high quality
manually labeled dataset is available. The construction of Gold Standard corpora
is key and require very detailed annotation guidelines and a carefully designed
corpus generation process with involvement of clinical domain experts. We
expect that such a corpus and evaluation will also be carried out for data in other
languages and that automatic anonymization and de-identi cation systems will
be bene cial beyond EHRs, such as medical surveys [
We acknowledge the Encargo of Plan TL (SEAD) to CNIO and BSC for funding, and the scienti c committee for their valuable comments and guidance. We would also like to thank Siamak Barzegar for his help in setting up MEDDOCAN at CodaLab, and Felipe Soares for input in preparing the manuscript and task.
MEDDOCAN: Automatic de-identi cation of medical texts in Spanish