Clinical coding requires the analysis and transformation of medical narratives into a structured or coded format using internationally recognized classi cation systems like ICD-10. These codes represent medical diagnoses and procedures. Clinical coding is critical for standardizing medical records, particularly for health information management systems used to carry out biomedical/ epidemiological research studies, monitor health trends or facilitate medical billing and reimbursement. The growing amount of clinical records has prompted the search for tools that assist manual coding. Inspired by the CCMC challenge and various eHealth CLEF shared tasks, we organized the CodiEsp track. Codiesp (eHealth CLEF 2020- Multilingual Information Extraction Shared Task) represents the rst e ort to promote the development and evaluation of automatic clinical coding systems for medical documents in Spanish. In this context, we have published a set of resources including (i) a manually coded Gold Standard corpus with inter-coder agreement and supporting textual evidence statements, (ii) an additional large collection of medical literature indexed with ICD-10 clinical codes and (iii) a machine translated corpus to enable multilingual approaches and testing of previous strategies developed for data in English. We have received a total of 168 runs submitted by 22 teams from 11 countries for at least one of our three sub-tracks: CodiEsp-D (Diagnosis Coding), CodiEsp-P (Procedure Coding) and CodiEsp-X (Explainable AI). Despite the considerable complexity of this task, which can be viewed as a hierarchical multi-label classi cation problem using ICD-10 codes as labels and documents as input, participants obtained very promising results, specially for codes that were well covered by the training data. Participants examined a variety of strategies, speci cally deep learning approaches, pre-trained language models and word embeddings (BERT, BETO, FastText, etc.), as well as NER, string lookup and knowledge graph approaches. CodiEsp Corpus: https://zenodo.org/record/3837305
Public health emergency situations, such as the COVID-19 global health crisis,
further highlight the need of e cient search, retrieval, analysis, integration as
well as exploitation strategies for a diversity of medical content types. This is
particularly true for the medical literature, where clinical case reports
characterizing in detail the symptoms and signs experienced by individual patients,
together with the diagnosis, treatment and follow-up information, constitute a
valuable evidence source for the possible pathogenesis of a disease and suitable
therapeutic approaches [
Structured clinical information, in the form of coded clinical data relying on controlled indexing vocabularies such as ICD-10 1is a key resource for statistical analysis techniques applied to patient data [43]. The results of clinical coding activities are being used, for instance, as aggregated data to analyze retrospective and prospective aspects of information contained in electronic health records (EHRs). Clinical coding is a complex and time-consuming process, carried out by trained experts. This task requires the assignment of codes from a clinical classi cation (typically the 10th revision of the International Statistical Classi cation of Diseases and Related Health Problems or ICD-10) that essentially represents diagnoses and procedures associated to electronic health records.
The use of automatic systems to assist coding experts is becoming increasingly relevant to keep up with the pace of newly generated clinical texts. Automated clinical coding systems represent also a mechanism to improve coverage and consistency during the transformation process of EHRs into their corresponding structured representations.
Clinical natural language processing and AI-based document indexing strategies can result in resources useful for automatic clinical coding, directly exploiting the unstructured content of EHRs. Such tools play an increasing role to generate results that complement health informatics approaches focusing on translational medicine challenges, by providing relevant diagnostic information extracted from clinical narratives. This implies that text mining generated clinical coding results can provide a rich clinical context for patient health information necessary for other downstream data analysis processes like bioinformatics and OMICS data exploration. Figure ?? shows a general view of a canonical text mining owchart and underlying tasks. 1 https://www.who.int/classi cations/icd/icdonlineversions/en/
Currently, most research on clinical NLP applies to English texts; however,
there is a considerable amount of biomedical documents generated in non-English
languages. The importance of clinical coding in languages other than English has
driven challenges to promote automatic clinical coding systems. There have been
community e orts to develop such clinical coding systems in English. In addition,
in recent years, shared tasks for clinical coding have been proposed for English
[37] as well as content in non-English languages such as French [15], German
[
In the following sections, we will summarize the CodiEsp shared task setting, evaluation metrics, corpus preparation/annotation process, as well as the results produced by participating teams and a short summary of the used methodologies.
The CodiEsp 2 track proposes participants the challenge of building an automatic clinical coding system for Spanish documents. Participant systems have to automatically assign ICD-10 codes (CIE-10 in Spanish) to clinical case documents. Evaluation is done by comparing automatically generated results against manually manually generated ICD-10 codi cations. 2.1
CodiEsp is structured into three di erent subtasks, two of them directly related to the two main branches of ICD-10 terminology. Moreover, to improve systems' acceptance, usefulness and practical integration into clinical coding support applications, results must be understandable, traceable to human-interpretable evidence sources and transparent. To that extent, in addition to two traditional coding subtasks, the CodiEsp shared task proposes a novel subtask on Explainable/Interpretable AI. Systems that participated in this subtask had to recognize the correct clinical codes and return the corresponding evidence text supporting the code assignment. The CodiEsp track comprised the following three subtracks: { CodiEsp Diagnosis Coding sub-task (CodiEsp-D): required automatic ICD10-CM [CIE10 Diagnostico] code assignment. This sub-track evaluated systems that predict ICD-10-CM codes (in the Spanish translation, CIE10Diagnostico codes). A list of valid codes for this sub-task with their English and Spanish description was provided by the task organizers 3. { CodiEsp Procedure Coding main sub-task (CodiEsp-P): required automatic ICD-10-PCS [CIE10 Procedimiento] code assignment. This sub-track evaluated systems that predict ICD-10-PCS codes (in the Spanish translation, CIE10-Procedimiento codes). A list of valid codes for this sub-task with their English and Spanish description was provided by the task organizers. { CodiEsp Explainable AI exploratory sub-task (CodiEsp-X). Participating systems were asked to return in addition to clinical code assignments supporting evidence texts extracted from documents. Both ICD-10-CM and ICD10-PCS codes were used for this subtask. Evaluation was done against a collection of manually labeled evidence texts. 2.2
The CodiEsp track was organized in the form of three basic participation periods: 1. Training phase. During the initial participation period, a random subset of the entire corpus was published corresponding to the training data collection. 2 https://temu.bsc.es/codiesp 3 https://zenodo.org/record/3706838 It consisted of plain text documents and their corresponding annotations, i.e. ICD-10 code assignments and manually labeled evidence texts. During this period, teams started implementing their automatic clinical coding strategies by exploiting this dataset. 2. Development phase. Next, a second subset of the corpus was released (development data). This dataset served to ne tune and improve the initial predictive coding systems. 3. Test phase. Finally, the test set was released. This third subset of the corpus was distributed without providing manual annotations/code assignments. Participants had to return for all test set documents their corresponding ICD-10 codes. After the submission deadline, the shared task organizers evaluated team predictions against manual code assignments/annotations. A total of 5 runs were allowed for each subtrack per team, so that participants could explore di erent strategies and methodological approaches. 2.3
In case of the CodiEsp-Diagnostic and CodiEsp-Procedure subtasks, automatic predictions returned by teams had to consist in ranked codes. The primary evaluation metric for these two subtasks was Mean Average Precision (MAP).
Mean Average Precision (MAP) is a widely used evaluation score for ranking problems:
P(P (k) rel(k)) AveP =
number of relevant documents where, P(k) is the precision at the position k, and rel(k) is an indicator function equaling 1 if the item at rank k is a relevant document, zero otherwise.
MAP has shown good discrimination and stability [29]. For completeness, error analysis, and comparison to previous e orts, other metrics were also computed: MAP@k (MAP taking into account just the rst k results), f1-score, precision, and recall.
MAP MAP@30 MAP@10
X X X X
X X X X X X X x
Participants of the CodiEsp-Explainability subtask were evaluated with micro balanced f1-score, precision, and recall since its scope is di erent and more complicated.
Precision (P) =
Recall (R) = F1 score (F1) =
true positives true positives + false positives
true positives true positives + false negatives 2 (P R)
(P + R)
A complete overview of all used evaluation metrics for the CodiEsp track is shown in Table 2. In addition, Table 1 shows an evaluation example. CodiEsp baseline system. To provide context to the obtained task results, we implemented a baseline system using dictionary lookup and vocabulary transfer. This baseline system selects manually labeled text spans from the training and development collections, using these mentions afterwards as a gazetteer for lexical looking up in the test set documents. The lookup was strict. However, texts had been previously tokenized and normalized (transformed to lowercase, accents were removed, extra blank spaces or punctuation signs were ignored).
The CodiEsp corpus4 is a collection of 1,000 clinical case reports written in Spanish that cover a diversity of medical specialities. The training subset consisted in 500 documents, while the development and test set consisted in 250 documents each. All documents were exhaustively, manually annotated by professional clinical coders with codes from the Spanish version of ICD-10 (procedure and diagnostic). Additionally, human annotators had to label or mark up clinical-coding evidence text fragments. Figure 2 shows an example document with manually annotated codes, and Table 3 shows an example of an annotated sentence in Spanish and in English.
The manual annotation process followed o cial clinical coding guidelines published for Spain. CodiEsp documents were coded with the 2018 version of CIE-10 (the o cial Spanish version of ICD-10-Clinical Modi cation and ICD10-Procedures) and inspired by the \Manual de Codi cacion CIE-10-ES Diagnosticos 2018" and the \Manual de Codi cacion CIE-10-ES Procedimientos 2018" provided by the Spanish Ministry of Health. To cover aspects and particularities relevant to the sub-tasks and documents used for CodiEsp, together with the corpus itself, a document describing the annotation guidelines was published 5. Clinical codes (diagnostic and procedure) were linked to textual evidence fragments that support their assignment (see Figure 3). 4 https://doi.org/10.5281/zenodo.3625746 5 https://zenodo.org/record/3730567
Spanish: El paciente presento un cuadro brusco de disnea, vomitos y perdida de conocimiento.
English automatic translation: The patient developed sudden dyspnea, vomiting, loss of consciousness
r55 r06.00 r11.10
disnea vomitos perdida de conocimiento
To guarantee annotation quality, we performed an iterative process of guideline re nement and consistency analysis through comparison between independent annotations provided by multiple clinical coders. Several initial annotation rounds were necessary until an acceptable level of manual annotation quality was obtained. For the annotation of diagnostic codes, the nal pairwise percentage agreement obtained was 88.6%, 88.9% for procedure codes and 80.5% for the annotation of textual evidence.
Corpus format. Gold Standard CodiEsp corpus is distributed in the CodiEsp
format: documents are provided in plain text format, and annotations are
released in a tab-separated le. Each line of the le corresponds to a code
assignment. This format is coherent with the data format used in the 2019 CLEF
clinical coding shared task [
Corpus statistics. In total, the entire CodiEsp corpus contains 18435 annotations, with the DIAGNOSTIC class more common than the PROCEDURE class: 77.8% of the annotations correspond to diagnostics. The 18435 annotations contain 3427 unique ICD-10 codes. Again, there are more diagnostic than procedure codes: 2557 and 870, respectively. We hypothesise that it is more complicated for an automatic system to predict procedures, since the corpus contains fewer examples. However, since there are also less unique procedure codes, this di culty may be partially addressed. Table 4 contains a summary of the corpus statistics. 3.2
We have generated a collection of additional resources to overcome the size limitation of our Gold Standard CodiEsp corpus. These resources included: CodiEsp MT corpus 6. The CodiEsp shared task attracted participants from many non-Spanish speaking countries. In addition, there are already many clinical coding systems for data in English. To ease the comparison with such systems, provide support to participants working previously in English and to explore the use of machine-translated corpora, we generated a machine translated version of the CodiEsp corpus (CodiEsp MT corpus). The used machine translation system was adapted to the language characteristics of the medical domain [42]. CodiEsp-abstracts 7. To increase the size and number of possible training instances, we prepared a dump of medical literature abstracts from the Lilacs [26] and IBECS [18] bibliographic resources. Those were indexed manually with either DeCS or MeSH terms. Using a mapping chain [DeCS ! MeSH ! UMLS ! ICD-10], we generated a collection of medical literature abstracts with associated ICD-10 codes. The resulting collection contains 176,294 Spanish medical abstracts indexed with ICD-10 codes.
A mapping chain was generated. DeCs is a terminological resource created to index journal articles, technical reports, and other health-related documents. It is based on MeSH (Medical Subject Heading), developed by the U.S. National Library of Medicine [27]. Additionally, we used the UMLS Metathesaurus tool [47] to map MeSH terms to ICD-10 codes (see Figure 4). This enabled us to build a mapping from DeCS terms to ICD-10 codes with this mapping chain [DeCS ! MeSH ! UMLS ! ICD-10]. 6 https://doi.org/10.5281/zenodo.3625746 7 https://doi.org/10.5281/zenodo.3606625
PubMed machine-translation 8. A large collection of PubMed abstracts was automatically translated into Spanish using the same translation engine employed to translate the CodiEsp corpus [42]. PubMed abstracts are manually indexed with MeSH and easily mapped to ICD-10 terms using the same mapping chain employed for CodiEsp-abstracts. This resource was also provided to participants.
CodiEsp Silver Standard 9. The CodiEsp test set documents were released together with an additional collection of 2,751 clinical case documents (called the background set). Participants were asked to provide code predictions for the entire collection of 3,001 documents (background set plus test set). Such a setting tried to examine if participating systems were able to scale to larger data collections. Code predictions for the background set were released as a CodiEsp Silver Standard corpus, similar to the CALBC initiative [40]. 4 4.1
We received submissions from a total of 22 teams. In the CodiEsp-Diagnostic subtask, there were 22 participants (78 runs). For the CodiEsp-Procedure track, we received 64 runs from 17 teams. The exploratory CodiEsp-Explainability subtask had 8 participants, which returned a total of 25 runs. In total, 167 novel clinical coding systems were generated in the context of CodiEsp. These numbers are shown in Table 6
A detailed description of the participant teams is included in Table 5. 4.2
8 https://doi.org/10.5281/zenodo.3826553 9 https://doi.org/10.5281/zenodo.3859869 { CodiEsp-Diagnostic. IXA-AAA, reached a MAP of 0.593. They obtained a high recall and a moderate precision. The highest f1-score was achieved by the IAM team, with 0.817 precision and 0.592 recall. { CodiEsp-Procedure. The IAM team achieved the highest MAP, 0.493, and f1-score, 0.522. The precision of this team was 0.691, and the recall was 0.42. { CodiEsp-Explainability. The top-performing team was FLE, with the best f1-score (0.611). It obtained a precision of 0.687, and its recall 0.562. In an uno cial run, the IAM team obtained the same f1-score. Since this subtask required identifying not only the right codes, but also the correct textual evidence, MAP metric was not computed.
Codes present in training and development. The division of the CodiEsp corpus into training, development, and test set was performed using randomly generated non-overlapping samples. Since the ICD-10 terminology has more than 170.000 distinct codes, some codes present in test documents were not covered previously by training or development set annotations. When evaluating systems using only the subset of codes present in training and development sets, all evaluation scores increase considerably. In the Figure 5, this e ect is clearly observed. In red, we have the f1-score values computed taking into account all codes. In blue, we have the same metric taking into account just the codes present Fig. 5. F1 comparison. In red, main test set results. In blue, test set results considering only codes that were present in training and development sets. In black, test set results computed from code categories. in training and development sets. Three teams in CodiEsp-Diagnostic subtask developed systems with an f1-score above 0.7. For a complete relation of metrics computed evaluating these codes, see Table 9, Table 10 and Table 11. Code categories. Figure 5 also includes a third f1-score value in black. It corresponds to the same metric computed on the categories. ICD-10 terminology is tree-shaped (for Diagnostics) and axial (for Procedures). This characteristic means that the rst digits of the code give di erent information than the last digits. As digits are located more to the right of the code, their information is more granular. Therefore, f1-score, precision, and recall were computed taking into account only the rst three digits for CodiEsp-Diagnostic and the rst four digits for CodiEsp-Procedure.
Systems that correctly tag the category but fail on the more granular information could be a starting point that requires ne-tuning. And this is the case of most participant systems of CodiEsp (Figure 5, Table 9, Table 10 and Table 11). Indeed, when observing these metrics, we can see that the best prediction run of the IAM team reaches 0.773 f1-score.
For a complete list of all metrics for all runs, check the Table 9, Table 10 and Table 11 at Appendix. 4.3
In this error analysis, the focus was placed on codes that, despite being present more than four times in the training or development sets, were predicted correctly by less than 20% of the runs. Such codes could be considered as \di cult".
continuous text evidence were, in general, more di cult to detect correctly. As seen in Figure 6, in the training, development, and test sets the percentage of codes with discontinuous textual evidence is below 15%. However, in the subset of di cult codes, it approaches 30%.
Besides, it is clear from Table 7 that results for CodiEsp-Procedure are worse than those for CodiEsp-Diagnostic. In the former, 38.7% of the textual evidence of the test sets are discontinuous. In the latter, 14.3%.
Additionally, codes that are well predicted by most teams include continuous pieces of evidence. In fact, among the codes successfully predicted in more than half of the runs, the proportion of discontinuous texts of evidence is 4.3%, while this proportion increases to 14.3% in the whole test set.
continuous references are di cult to detect. Also, codes whose textual reference is longer are more challenging. In Figure 6, we observe how the distribution of textual reference lengths is almost identical for the codes in the training, development, and test sets (blue and green). However, for our subset of codes of interest, the average length increases.
codes ending with a 9 tend to be less speci c than the others. For example, code M25.561 represents "pain in the right knee", while M25.569 is used for "pain in an unspeci ed knee". The same happens with procedure codes ending with Z. We have evaluated the codes predicted by more than 50% of participant runs. Those codes could be seen as \easier" codes since most systems assign them. And there are 34.2% of 9-ending codes in the training and development sets, 36.8% of such codes in the test set, while there are 52.5% of such codes in the \easy" subset of codes.
breviations are a common problem when processing medical narratives. Their presence is ubiquitous in this type of texts, and their meaning varies from one medical specialty to another. We have looked for Spanish medical abbreviations, collected in the Spanish Medical Abbreviation DataBase [22], in the textual pieces of evidence that justify the code assignment.
Abbreviations appear in the di cult subset of codes in a similar percentage as in the rest of the corpus. For instance, 18.9% of diagnostic code evidence have abbreviations, and that percentage in the di cult codes is 19.1%. However, abbreviations appear much less in a subset of \easy" codes (codes predicted by more than 50% of the runs), 13%. For procedures, the phenomenon is the same. Indeed, there are more abbreviations in the procedure than in the diagnostic textual evidences: 40% of procedure textual evidences contain abbreviations from the Spanish Medical Abbreviation DataBase, against 19.1% for diagnostic textual evidences. This might contribute to the di culty of assigning procedure codes. 5
Methodology distribution. With participants from diverse backgrounds, the
range of methodologies employed is broad. For speci c details of the
participants' systems, we refer you to the particular articles. However, a simpli ed
classi cation is presented in this paper. Participant systems were divided into
those that employ language models (such as the popular BERT [
In the three subtasks, there are successful and unsuccessful teams in the three methodological groups. For example, the highest MAP in CodiEsp-Diagnostic is obtained by a team employing machine learning. In contrast, the top MAP scores for CodiEsp-Procedure was obtained by non-machine learning systems. The second-best f1-score in both subtasks was obtained by a ne-tuning Multilingual BERT (a language model) approach. Finally, it is noteworthy that in case of the CodiEsp-Explainability track, more teams were using non-machine learning strategies.
Participants descriptions. There have been three main approaches to
automatic clinical coding:
{ Classi cation. This approach considers that there are a set of documents
that must be categorized. Every ICD-10 has its own category. For example,
team ICB-UMA [28] followed this classi cation schema.
{ Named Entity Recognition. In this case, automatic systems must detect
whether each clinical case word (or set of words) is a diagnostic, a
procedure, or none of them. Examples of NER systems are FLE [14] or IAM
[
NER system.
Additionally, each of the two schemas (classi cation vs. NER) may be tackled
using di erent technologies. In this overview, we have clustered the technologies
in 3 classes:
{ Non-machine learning approaches. For instance, IAM [
XGBoost to perform document classi cation.
{ Language models. Within the teams using machine learning, a signi cant
number of them employ language models. For example, the FLE team [14]
ne-tuned BERT Multilingual and The Mental Strokers [
In the following paragraphs, the approaches followed by some of the teams
are brie y described to illustrate the di erent methodologies. Descriptions of
other participant teams are found in their system description papers.
{ IXA-AAA. They combined a Machine Learning engine with a string
similarity system. First, a binary XGBoost classi er was trained for each label. Since
the XGBoost outputs a probability for each prediction, its output could be
used directly in subtasks CodiEsp-Diagnostic and CodiEsp-Procedure. Texts
were expanded to improve the XGBoost models, concatenating the medical
entities extracted from the documents itself. Second, the string similarity
system compares text fragments and ICD-10 code de nitions using
Levenshtein distance, Jaro Winkler algorithm, and Cosine Similarity on
Multilingual BERT representations. ICD-10 standard de nitions were expanded
with non-standard terms, single-word descriptions, and even phrases
frequently associated with the codes. The best system for CodiEsp-Diagnostic
and CodiEsp-Procedure subtasks is a combination of XGBoost and Jaro
Winkler string similarity outputs. This combination obtained the highest
MAP for the CodiEsp-Diagnostic subtask, 0.593 [
Combined methodologies. Since teams had approached the challenge from di erent perspectives, we analyzed what would happen if we combined predictions from di erent approaches. This rationale was already followed by the IXAAAA team when they combined an XGBoost classi er with a string matching system. For instance, we may combine predictions of the FLE (that ne-tuned mBERT for NER and Levenshtein distance to nd ICD-10 code) and the IAM (that employed a tuned dictionary lookup) teams. The combination is an uncomplicated union. Any code predicted by any of the two systems is considered.
Compared with the manual gold standard, we observe in Table 8 how the f1score increases, and the resulting prediction is more balanced than the previous two. Both systems lacked a high recall, and combining them makes recall to increase. Precision decreases, but not as much. Then, the result is a prediction higher than any of the individual participant predictions. 6
The CodiEsp task (eHealthCLEF 2020 Task 1 on Multilingual Information Extraction), has attracted a considerable number of participants. There were only 3 CLEF subtasks with more registrations (ImageCLEF Task 3, medical; CheckThat Task 1, Check-Worthiness on tweets; and LifeCLEF Task 2, BirdCLEF). Indeed, participant teams came from very diverse countries, despite the highly specialized and complex application domain (medical/clinical data) and the use of non-English textual data. In this sense, CodiEsp could be perceived as a more challenging task setting when compared to other CLEF competitions, such as ImageCLEF or LifeCLEF.
Novelty and Impact. To the best of our knowledge, the CodiEsp corpus is the rst publicly available, manually annotated, clinical coding gold standard text corpus in Spanish. Despite the use of commercial tools for automatic clinical coding in hospitals and private companies, a setting for direct comparison and benchmarking using a common evaluation set, format and metrics was missing. CodiEsp has partially solved this problem. However, there is still a need for large data collections, including high quality manually coded anonymized EHRs from multiple hospitals.
Access to patient reports is a recurrent problem in clinical NLP due to privacy issues. Creative solutions, such as using clinical cases that have been carefully selected as a surrogate data of real clinical reports, might contribute to the development of clinical NLP infrastructures. In this sense, we have also employed document and individual sentence similarity (lexical/surface as well as semantic similarity) strategies comparing real EHRs (mainly discharge summaries and radiology reports) with clinical case reports to retrieve sentences that are basically equivalent for corpus construction and public release purposes. We call this Fig. 8. Fields of knowledge of CodiEsp participants and the time invested in the shared task. kind of creative corpus construction, circumventing data privacy issues as the wandering corpus strategy, where data with legal redistribution issues can at least be emulated by publicly available data resources that are highly similar.
CodiEsp is a clinical coding shared task in Spanish. Despite this apparent language constraint, there have been participants from 9 non-Spanish speaking countries. One of the reasons may be that 57.9% of participants reported that their system is multilingual and not Spanish-speci c. CodiEsp not only attracted participants from di erent countries, but also di erent backgrounds (Figure 8). Even though Natural Language Processing, Arti cial Intelligence, and Machine Learning were the most numerous background, there were also participants coming from elds such as bioinformatics, linguistics, and medicine or biomedicine. Finally, 57.9% of the participants reported experience in clinical coding systems (or similar tasks) before CodiEsp, and most of them said that their motivation was \to be able to compare their results with other strategies/methods/teams."
The participation of such diverse pro les has allowed the creation of heterogeneous resources, available to the community. Such resources are centralized in the CodiEsp webpage11. In this sense, 68.4% of the participant teams indicated that they would provide software or web service based on their system if there is speci c technical or nancial support, whereas 31.6% were not interested in assistance to advance their clinical coding system into a software product or startup.
CodiEsp has been a challenging shared task according to the participants. Most of them rated it as di cult or very di cult. However, 78.9% would be interested in participating in a second CodiEsp track. When asked about the time invested, the most common answer has been 1 to 4 weeks, followed by 4 to 10 weeks (Figure 8).
Possible improvements. One of the CodiEsp limitations has been the Gold Standard size. CodiEsp corpus contains 1,000 annotated clinical case reports, with 16,504 sentences and 396,988 tokens. In the shared task setting, 750 docu11 https://temu.bsc.es/codiesp ments were used for learning, and 250 were employed for results evaluation. As previously discussed, some codes present in the test set had not been employed in the training and development sets. Indeed, 1856 codes appear just once in the entire gold standard. Even though several systems achieved high metrics, a more signi cant Gold Standard would allow more examples for systems to learn from, and a more representative set of documents to evaluate them. An extension of the CodiEsp corpus would now require fewer resources, since the guidelines employed are already publicly available [31].
Additionally, as some participants have pointed out, the gold standard of the exploratory subtask CodiEsp-Explainability had a limited inter-annotator agreement. CodiEsp corpus was annotated independently by two clinical experts. On the subset of documents annotated by both experts, they achieved an IAA of 80.5% for annotating the textual references that justify the code assignment. Arguably, a higher IAA would have resulted in more top metrics in the CodiEspExplainability subtask.
These two limitations, gold standard size and consistency in the text evidence annotation, should act to spur future research groups to extend the CodiEsp corpus. Most participants(78.9%) reported interest in a second CodiEsp edition. Additionally, annotation guidelines are already publicly available, together with the entire CodiEsp corpus, CodiEsp abstracts corpus, and many other resources employed by participants to tune their systems. A second CodiEsp edition could make use of these materials to further promote clinical coding in Spanish. Closing remarks. The task has been relevant not only in terms of determining the most competitive approaches for this particular data and track, but it also explains how to generate new clinical coding tools for other languages and data collections. The former includes participant systems that are open source and the baseline, all compiled in the CodiEsp webpage 12. The latter comprises the range of resources described in Section 3: CodiEsp corpus, annotation guidelines, CodiEsp abstracts, terminology mapping chain [DeCS ! MeSH ! UMLS ! ICD-10], machine-translated version of CodiEsp corpus, machine-translated PubMed dump and Codiesp Silver Standard.
We acknowledge the Encargo of Plan TL (SEAD) to BSC for funding, and the scienti c committee for their valuable comments, their guidance, and their help with the review of the proceedings. Besides, we would also like to thank the organization of CLEFeHealth, Lorraine Goeuriot, Hanna Suominen, and Liadh Kelly. Finally, we would particularly like to thank the team of Bitac, Gloria Gonzalez, and Toni Mas, who worked with us to create the dataset. 12 temu.bsc.es/codiesp/participants-systems/
TeamX SWAP LIIR FLE IAM BCSG SINAI DCICUNS IMS 0 PA 3P@ P R F1 M A
M .284 .277 .123 .448 .192 .299 .28 .011 .713 .022 .259 .24 .052 .58 .096 .265 .24 .004 .858 .009 .065 .053 .011 .713 .022 .202 .202 .295 .323 .308 .013 .013 .026 .041 .032 .117 .117 .272 .218 .242 .169 .16 .135 .442 .207 .013 .013 .04 .03 .034 .044 .044 .124 .055 .076 .011 .011 .066 .029 .041 .015 .015 .073 .032 .044 .002 .002 .013 .006 .008 .006 .006 .04 .018 .024 .519 .519 .732 .633 .679 .481 .48 .733 .588 .652 .501 .501 .74 .604 .665 .46 .46 .739 .556 .635 .521 .521 .817 .592 .687 .511 .511 .789 .591 .676 .242 .242 .375 .285 .324 .259 .259 .407 .287 .337 .231 .231 .457 .244 .318 .21 .21 .342 .28 .308 .128 .128 .235 .215 .225 .301 .298 .412 .538 .467 .314 .311 .443 .544 .488 .302 .299 .418 .54 .471 .251 .251 .45 .433 .441 .291 .288 .402 .528 .456 .097 .097 .385 .09 .146 .084 .084 .282 .14 .187 .074 .074 .482 .061 .108 .078 .074 .128 .261 .172 .449 .446 .373 .652 .474 .391 .383 .306 .672 .42 .389 .378 .299 .682 .416 .395 .373 .079 .699 .143 0 PA 3P@ P R F1 M A
M .334 .326 .123 .522 .199 .351 .329 .011 .831 .022 .304 .282 .052 .676 .097 .311 .282 .004 1 .009 .08 .066 .011 .831 .022 .236 .236 .295 .376 .331 .015 .014 .026 .048 .034 .136 .136 .272 .254 .262 .195 .185 .135 .515 .214 .022 .022 .099 .033 .049 .052 .052 .124 .064 .084 .013 .013 .066 .034 .045 .017 .017 .073 .037 .049 .004 .004 .032 .007 .011 .006 .006 .04 .021 .027 .598 .597 .767 .699 .731 .553 .553 .768 .646 .702 .576 .576 .775 .665 .716 .528 .528 .774 .61 .682 .605 .605 .843 .672 .748 .605 .605 .789 .689 .736 .288 .288 .375 .333 .352 .306 .306 .407 .335 .367 .275 .275 .457 .285 .351 .244 .243 .342 .327 .334 .149 .148 .235 .25 .243 .391 .39 .513 .615 .559 .414 .413 .551 .621 .584 .397 .395 .519 .616 .564 .328 .328 .559 .496 .526 .377 .376 .51 .604 .553 .159 .159 .772 .099 .175 .151 .151 .64 .148 .24 .12 .12 .738 .067 .123 .184 .184 .436 .271 .334 .527 .524 .373 .76 .5 .459 .45 .306 .783 .44 .461 .452 .306 .785 .441 .462 .439 .079 .807 .144
P
R
F1 .162 .538 .249 .02 .858 .038 .082 .681 .147 .01 .968 .021 .02 .858 .038 .346 .405 .373 .04 .064 .049 .3 .254 .275 .163 .542 .25 .096 .082 .089 .131 .066 .088 .085 .043 .057 .091 .046 .061 .063 .019 .03 .051 .025 .033 .802 .734 .766 .804 .687 .741 .807 .714 .758 .809 .662 .728 .877 .69 .773 .837 .682 .752 .425 .332 .373 .461 .333 .387 .52 .282 .366 .407 .332 .366 .284 .268 .276 .53 .646 .582 .567 .642 .602 .535 .641 .583 .586 .519 .551 .513 .624 .564 .768 .123 .212 PA 10P@ P R F1 PA 10P@ P R F1 P R F1 M A M A
M M .186 .168 .011 .685 .022 .207 .186 .011 .83 .023 .013 .716 .026 .182 .168 .042 .415 .077 .202 .187 .042 .503 .078 .048 .454 .087 .19 .169 .011 .685 .022 .212 .188 .011 .83 .023 .013 .716 .026 .16 .147 .029 .423 .054 .176 .161 .029 .513 .054 .032 .455 .06 .166 .147 .004 .825 .008 .183 .161 .004 1 .008 .005 .857 .01 .221 .219 .186 .38 .25 .25 .247 .186 .461 .265 .206 .416 .276 .137 .127 .122 .399 .187 .152 .141 .122 .484 .195 .133 .43 .203 .141 .14 .155 .323 .209 .154 .153 .155 .392 .222 .16 .339 .218 .17 .15 .097 .513 .164 .191 .168 .097 .622 .169 .107 .557 .18 0 0 0 0 0 0 0 0 0 0 0.001 .001 .017 .017 .051 .034 .041 .018 .018 .051 .041 .046 .053 .036 .043 .007 .007 .015 .01 .012 .008 .008 .015 .012 .014 .017 .012 .014 .02 .02 .046 .02 .028 .022 .022 .046 .025 .032 .053 .036 .043 0 0 0 0 0 0 0 0 0 0 .068 .046 .055 0 0 0 0 0 0 0 0 0 0 .068 .046 .055 .434 .433 .587 .448 .508 .515 .514 .627 .539 .58 .665 .468 .549 .433 .432 .587 .446 .507 .513 .512 .626 .537 .578 .665 .465 .548 .443 .443 .643 .428 .514 .525 .525 .692 .514 .59 .687 .462 .552 .44 .44 .642 .424 .511 .52 .52 .692 .51 .587 .687 .458 .55 .426 .426 .659 .373 .476 .496 .496 .659 .452 .536 .761 .431 .55 .493 .493 .691 .42 .522 .569 .569 .691 .509 .586 .764 .467 .579 .28 .275 .367 .452 .405 .33 .326 .393 .548 .457 .441 .492 .465 .293 .289 .37 .476 .416 .351 .347 .39 .577 .465 .437 .514 .472 .271 .267 .342 .455 .391 .327 .323 .368 .552 .442 .405 .494 .445 .25 .245 .343 .422 .378 .298 .293 .373 .512 .432 .402 .456 .427 .254 .249 .318 .458 .376 .303 .297 .344 .556 .425 .377 .5 .43 .386 .383 .455 .52 .485 .438 .435 .456 .63 .529 .521 .542 .531 .442 .442 .601 .412 .489 .509 .509 .602 .499 .546 .714 .427 .535 .404 .402 .501 .503 .502 .457 .454 .502 .608 .55 .586 .525 .554 .168 .168 .34 .018 .035 .17 .17 .34 .022 .041 .468 .025 .048 TeamX SWAP LIIR FLE IAM SINAI MEDIA [13] ExeterChiefs. eda-classi cation. https://github.com/aollagnier/eda_ classification. 2020. [14] Nuria Garc a-Santa and Kendrick Cetina. \FLE at CLEF eHealth 2020: Text Mining and Semantic Knowledge for Automated Clinical Encoding". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [15] Lorraine Goeuriot et al. \CLEF 2017 eHealth evaluation lab overview".
In: International Conference of the Cross-Language Evaluation Forum for European Languages. Springer. 2017, pp. 291{303. [16] Hulat. Codiesp-CLEF-2020-eHealth-Task1. https://github.com/pqueipo/
Codiesp-CLEF-2020-eHealth-Task1. 2020. [17] IAM. IAMsystem. https://github.com/scossin/IAMsystem. 2020. [18] IBECS. https://ibecs.isciii.es/. Accessed: 2020-08-26. [19] ICB-UMA. CLEF-2020-CodiEsp. https : / / github . com / guilopgar /
CLEF-2020-CodiEsp. 2020. [20] Iker de la Iglesia et al. \MEDIA team: CLEF-2020 eHealth Task 1: Multilingual Information Extraction - CodiEsp". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [21] IMS. https://github.com/gmdn. 2020. [22] Ander Intxaurrondo. AbreMES-DB. Version 2018-12-01. Funded by the Plan de Impulso de las Tecnolog as del Lenguaje (Plan TL). Zenodo, Nov. 2018. doi: 10.5281/zenodo.2207130. url: https://doi.org/10.5281/ zenodo.2207130. [23] Ander Intxaurrondo et al. \The Biomedical Abbreviation Recognition and Resolution (BARR) track: benchmarking, evaluation and importance of abbreviation recognition systems applied to Spanish biomedical abstracts".
In: (2017). [24] Ander Intxaurrondo et al. \Finding Mentions of Abbreviations and Their De nitions in Spanish Clinical Cases: The BARR2 Shared Task Evaluation Results." In: IberEval@ SEPLN. 2018, pp. 280{289. [25] Rishi Vardhan K et al. \Transformers in Semantic Indexing of Clinical Codes". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [26] LILACS. https://lilacs.bvsalud.org/es/. Accessed: 2020-08-26. [27] Carolyn E Lipscomb. \Medical subject headings (MeSH)". In: Bulletin of the Medical Library Association 88.3 (2000), p. 265. [28] Guillermo Lopez-Garc a, Jose M. Jerez, and Francisco J. Veredas. \CBUMA at CLEF e-Health 2020 Task 1: Automatic ICD-10 coding in Spanish with BERT". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [29] Christopher D Manning, Hinrich Schutze, and Prabhakar Raghavan. Introduction to information retrieval. Cambridge university press, 2008. [30] Montserrat Marimon et al. \Automatic De-identi cation of Medical Texts in Spanish: the MEDDOCAN Track, Corpus, Guidelines, Methods and Evaluation of Results." In: IberLEF@ SEPLN. 2019, pp. 618{638. [31] Toni Mas et al. CodiEsp guidelines. Version 1. Funded by the Plan de Impulso de las Tecnolog as del Lenguaje (Plan TL). Mar. 2020. doi: 10. 5281 / zenodo . 3730567. url: https : / / doi . org / 10 . 5281 / zenodo . 3730567. [32] Elias Moons and Marie-Francine Moens. \Convolutional Attention Models with Post-Processing Heuristics at CLEF eHealth 2020". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [33] nlp4life. CLEFeHealth2020-multilabel-bert. https://github.com/sarahESL/
CLEFeHealth2020-multilabel-bert. 2020. [34] Giorgio Maria Di Nunzio. \As Simple as Possible: Using the R Tidyverse for Multilingual Information Extraction. IMS UniPD ad CLEF eHealth 2020 Task 1". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [35] Anas Ollagnier and Hywel Williams. \Text Augmentation Techniques for Clinical Case Classi cation". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [36] Jose M. Perea-Ortega et al. \SINAI at CLEF eHealth 2020: testing different pre-trained word embeddings for clinical coding in Spanish". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum.
CEUR Workshop Proceedings. 2020. [37] John Pestian et al. \A shared task involving multi-label classi cation of clinical free text". In: Biological, translational, and clinical language processing. 2007, pp. 97{104. [38] Marco Polignano et al. \A study of Machine Learning models for Clinical Coding of Medical Reports at CodiEsp 2020". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [39] Paula Queipo-Alvarez, Paloma Mart nez-Fernandez, and Israel GonzalezCarrasco. \Classifying clinical case studies with ICD-10 at Codiesp CLEF eHealth 2020 Task 1-Diagnostics". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [40] Dietrich Rebholz-Schuhmann et al. \CALBC silver standard corpus". In: Journal of bioinformatics and computational biology 8.01 (2010), pp. 163{ 179. [41] Henning Schafer and Christoph M. Friedrich. \Multilingual ICD-10 Code Assignment with Transformer Architectures using MIMIC-III Discharge Summaries FHDO Biomedical Computer Science Group (BCSG)". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum.
CEUR Workshop Proceedings. 2020. [42] Felipe Soares and Martin Krallinger. \BSC Participation in the WMT Translation of Biomedical Abstracts". In: Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2). 2019, pp. 175{178. [43] Mary H Stan ll et al. \A systematic literature review of automated clinical coding and classi cation systems". In: Journal of the American Medical Informatics Association 17.6 (2010), pp. 646{651. [44] Pontus Stenetorp et al. \BRAT: a web-based tool for NLP-assisted text annotation". In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. 2012, pp. 102{107. [45] SWAP. CODIESP-10. https://github.com/marcopoli/CODIESP- 10.
2020. [46] Yuki Tagawa et al. \TeamX at CLEF eHealth 2020: ICD Coding with N-gram Encoder and Code- ltering Strategy". In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. 2020. [47] UMLS - Metathesaurus. https://www.nlm.nih.gov/research/umls/ knowledge_sources/metathesaurus/index.html. Accessed: 2020-08-26. [48] Marta Villegas et al. \Esfuerzos para fomentar la miner a de textos en biomedicina mas alla del ingles: el plan estrategico nacional espan~ol para las tecnolog as del lenguaje". In: Procesamiento del Lenguaje Natural 59 (2017), pp. 141{144.