Overview of the eHealth Knowledge Discovery Challenge at IberLEF 2020 Alejandro Piad-Morffisa , Yoan Gutiérrezb,c , Hian Cañizares-Diaza , Suilan Estévez-Velardea , Rafael Muñozb,c , Andrés Montoyob,c and Yudivian Almeida-Cruza a School of Math and Computer Science, University of Havana, La Habana, 10400, Cuba b Department of Language and Computing Systems, University of Alicante, Alicante, 03690, Spain b University Institute for Computing Research, University of Alicante, Alicante, 03690, Spain Abstract This paper summarises the results of the third edition of the eHealth Knowledge Discovery (KD) challenge, hosted at the Iberian Language Evaluation Forum 2020. The eHealth-KD challenge proposes two computational tasks involving the identification of semantic entities and relations in natural language text, focusing on Spanish language health documents. In this edition, besides text extracted from medical sources, Wikipedia content was introduced into the corpus, and a novel transfer-learning evaluation scenario was designed that challenges participants to create systems that provide cross- domain generalisation. A total of eight teams participated with a variety of approaches including deep learning end-to-end systems as well as rule-based and knowledge-driven techniques. This paper analyses the most successful approaches and highlights the most interesting challenges for future research in this field. Keywords eHealth, Knowledge Discovery, Natural Language Processing, Machine Learning 1. Introduction The vast amount of clinical text available online has motivated the development of automatic knowledge discovery systems that can analyse this data and discover relevant facts. These discoveries can be the base for novel treatments, understanding disease and drug interactions. Computational systems designed for this task are often trained on manually annotated corpora. To foster research in this area, the community has organised competitive challenges to identify, classify, extract, and link knowledge, such as in SEMEVAL 1 and CLEF campaigns 2 . Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) email: apiad@matcom.uh.cu (A. Piad-Morffis); ygutierrez@dlsi.ua.es (Y. Gutiérrez); hian.canizares@matcom.uh.cu (H. Cañizares-Diaz); sestevez@matcom.uh.cu (S. Estévez-Velarde); rafael@dlsi.ua.es (R. Muñoz); montoyo@dlsi.ua.es (A. Montoyo); yudy@matcom.uh.cu (Y. Almeida-Cruz) orcid: 0000-0001-9522-3239 (A. Piad-Morffis); 0000-0002-4052-7427 (Y. Gutiérrez); 0000-0002-5334-7468 (H. Cañizares-Diaz); 0000-0001-6707-1442 (S. Estévez-Velarde); 0000-0001-8127-9012 (R. Muñoz); 0000-0002-3076-0890 (A. Montoyo); 0000-0002-2345-1387 (Y. Almeida-Cruz) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings CEUR Workshop Proceedings (CEUR-WS.org) http://ceur-ws.org ISSN 1613-0073 1 http://alt.qcri.org/semeval2020/ 2 http://www.clef-initiative.eu/ The eHealth Knowledge Discovery (eHealth-KD) challenge, in its third edition, leverages a semantic model of human language that encodes the most common expressions of factual knowledge, via a set of four general-purpose entity types and thirteen semantic relations among them. The challenge proposes the design of systems that can automatically annotate entities and relations in clinical text in the Spanish language. In this new edition, an alternative evaluation scenario (not related to the health domain) is also considered, which challenges participants to design systems that can successfully transfer their internal semantic representations from the health domain to an arbitrary new domain with considerably reduced training data. The challenge has been hosted at the Iberian Languages Evaluation Forum 2020, and included the participation of eight teams of researchers from different institutions. This paper presents the design of the challenge as well as the data and tools provided to participants, and analyses the results obtained by each team. The remainder of the paper is organised as follows: Section 2 provides a detailed description of the tasks defined in the eHealth-KD challenge and the data provided for training and evaluation of knowledge discovery system, as well as all relevant evaluation metrics. Section 3 briefly describes all the solutions that were submitted to the challenge and introduces a set of characteristics that allow a qualitative comparison among them. Section 4 presents the main results of the challenge, divided into four evaluation scenarios, and analyses the most successful and promising approaches deployed by each team. Finally, Section 5 presents the conclusions of the research and recommendations for future editions. 2. Challenge description The eHealth-KD challenge involves the identification of semantic entities and relations in natural language text. Although the focus has been on the health domain in past editions, the nature of the entities and relations extracted are general and can be applied to any domain. Figure 1 shows an example of three sentences with the relevant entities and relations annotated. An in-depth explanation of the annotation model is provided in Piad-Morffis et al. [1]. The evaluation of the challenge consists of submitting a set of natural language sentences with annotations automatically produced by a knowledge discovery system. Participants are provided with a set of manually annotated sentences (training and development corpus) that can be used for training and/or fine-tuning system as well as raw sentences that are used for evaluation (test corpus). The training and development corpus was provided two months in advance, but the test corpus was released only two weeks prior to the evaluation date, to discourage any fine-tuning on the test data. Although the actual source code of the system is not required, participants are encouraged to upload their code to open source code sharing services like Github. To simplify the evaluation and provide more fine-grained comparisons the task is divided into two subtasks: one concerned with the identification and classification of entities, and the other concerned with the extraction of the semantic relations between these entities. 72 subject target is-a Con Concept Action Con Concept 1 El asma es una enfermedadque afecta las vías respiratorias. causes in-time target in-context target Action Concept Con Concept Action Con 2 La exposición prolongadaal sol en verano provoca daños en la piel. target arg subject domain Ref ++ Action Concept Pred C Con 3 Esta afecta principalmentea las personas mayores de 60 años. Figure 1: Example annotation of three sentences from the eHealth-KD challenge. 2.1. Subtask A: Entity Recognition Given a list of eHealth documents written in Spanish, the goal of this subtask is to identify all the entities per document and their types. These entities are all the relevant terms (single word or multiple words) that represent semantically important elements in a sentence. The following figure shows the relevant entities that appear in a set of example sentences. Some entities (“vías respiratorias” and “60 años”) span more than one word. Entities will always consist of one or more complete words (i.e., not a prefix or a suffix of a word), and will never include any surrounding punctuation symbols, parenthesis, etc. There are four types for entities: Concept: identifies a relevant term, concept, idea, in the knowledge domain of the sentence. Action: identifies a process or modification of other entities. It can be indicated by a verb or verbal construction, such as “afecta” (affects), but also by nouns, such as “exposición” (exposition), where it denotes the act of being exposed to the Sun, and “daños” (damages), where it denotes the act of damaging the skin. It can also be used to indicate non-verbal functional relations, such as “padre” (parent), etc. Predicate: identifies a function or filter of another set of elements, which has a semantic label in the text, such as “mayores” (older), and is applied to an entity, such as “personas” (people) with some additional arguments such as ‘‘60 años” (60 years). Reference: identifies a textual element that refers to an entity —of the same sentence or of different one—, which can be indicated by textual clues such as “esta”, “aquel”, etc. 73 2.2. Subtask B: Relation Extraction Subtask B continues from the output of Subtask A, by linking the entities detected and labelled in the input document. The purpose of this subtask is to recognise all relevant semantic relationships between the entities recognised. Eight of the thirteen semantic relations defined for this challenge can be identified in Figure 1. The semantic relations are divided into the following categories: General relations (6): general-purpose relations between two concepts (it involves Concept, Action, Predicate, and Reference) that have a specific semantic. When any of these relations apply, it is preferred over a domain relation –tagging a key phrase as a link between two information units–, since their semantic is independent of any textual label: is-a: indicates that one entity is a sub-type, instance, or member of the class identified by the other. same-as: indicates that two entities are semantically the same. has-property: indicates that one entity has a given property or characteristic. part-of: indicates that an entity is a constituent part of another. causes: indicates that one entity provokes the existence or occurrence of another. entails: indicates that the existence of one entity implies the existence or occurrence of another. Contextual relations (3): enable an entity to be refined (it involves Concept, Action, Predicate, and Reference) by attaching modifiers. These are: in-time: to indicate that something exists, occurs or is confined to a time-frame, such as in “exposición” i n - t i m e “verano”. in-place: to indicate that something exists, occurs or is confined to a place or location. in-context: to indicate a general context in which something happens, like a mode, manner, or state, such as “exposición” i n - c o n t e x t “prolongada”. Action roles (2): indicate what role the entities play related to an Action: subject: indicates who performs the action, such as in “[el] asma afecta […]”. target: indicates who receives the effect of the action, such as in “[…] afecta [las] vías respiratorias”. Actions can have several subjects and targets, in which case the semantic interpreted is that the union of the subjects performs the action over each of the targets. Predicate roles (2): indicate which role play the entities related to a Predicate: domain: indicates the main entity on which the predicate applies. arg: indicates an additional entity that specifies a value for the predicate to make sense. The exact semantic of this argument depends on the semantic of the predicate label, such as in “mayores [de] 60 años”, where the predicate label “mayores” indicates that “60 años” is a quantity, that restricts the minimum age for the predicate to be true. 74 2.3. Evaluation Scenarios The eHealth-KD 2020 Challenge proposes four evaluation scenarios to measure different char- acteristics of the participant systems. We propose using a micro-averaged 𝐹1 that weights all individual annotations equally, both entities and relations. Scenario 1 evaluates the solution to both tasks simultaneously, while Scenario 2 and 3 evaluate each task independently. Finally, Scenario 4 challenges systems to a novel domain with significantly less training data. This allows a more fine-grained comparison among systems with respect to specific capacities. 2.3.1. Main Evaluation (Scenario 1) This scenario evaluates both subtasks together as a pipeline. The input consists only of a plain text, and the expected output is a BRAT . a n n file with all the corresponding entities and relations found. The measures will be precision, recall and F1 as follows: 1 𝐶𝐴 + 𝐶𝐵 + 𝑃𝐴 2 𝑅𝑒𝑐𝐴𝐵 = 𝐶𝐴 + 𝐼𝐴 + 𝐶𝐵 + 𝑃𝐴 + 𝑀𝐴 + 𝑀𝐵 1 𝐶𝐴 + 𝐶𝐵 + 𝑃𝐴 2 𝑃𝑟𝑒𝑐𝐴𝐵 = 𝐶𝐴 + 𝐼𝐴 + 𝐶𝐵 + 𝑃𝐴 + 𝑆𝐴 + 𝑆𝐵 𝑃𝑟𝑒𝑐𝐴𝐵 ⋅ 𝑅𝑒𝑐𝐴𝐵 𝐹1𝐴𝐵 = 2 ⋅ 𝑃𝑟𝑒𝑐𝐴𝐵 + 𝑅𝑒𝑐𝐴𝐵 The exact definition of Correct(C), Missing(M), Spurious(S), Partial(P) and Incorrect(I) is presented in the following sections for each subtask. 2.3.2. Optional Subtask A (Scenario 2) This scenario only evaluates Subtask A. The input is a plain text with several sentences and the output is a BRAT .ann file with only entity annotations in it (relation annotations are ignored if present). To compute the scores we define correct, partial, missing, incorrect and spurious matches. The expected and actual output files do not need to agree on the ID for each entity, nor on their order. The evaluation matches are based on the start and end of text spans and the corresponding type. A brief description about the metrics follows: Correct matches are reported when a text in the development file —DEV— matches exactly with a corresponding text span in the gold file for START and END values, and also the entity type. Only one correct match per entry in the gold file can be matched. Hence, duplicated entries will count as Spurious. Incorrect matches are reported when START and END values match, but not the type. 75 Partial matches are reported when two intervals [START, END] have a non-empty intersec- tion, such as the case of “vías respiratorias” and “respiratorias” in the previous example (and matching LABEL). Notice that a partial phrase will only be matched against a single correct phrase. For example, “tipo de cáncer” could be a partial match for both “tipo” and “cáncer”, but it is only counted once as a partial match with the word “tipo”. The word “cáncer” is counted then as Missing. This aims to discourage a few large text spans that cover most of the document from getting a very high score. Missing matches are those that appear in the GOLD file but not in the DEV file. Spurious matches are those that appear in the DEV file but not in the gold file. From these definitions, we compute precision, recall, and a standard F1 measure as follows: 1 𝐶𝐴 + 𝑃𝐴 2 𝑅𝑒𝑐𝐴 = 𝐶𝐴 + 𝐼 𝐴 + 𝑃 𝐴 + 𝑀 𝐴 1 𝐶𝐴 + 𝑃𝐴 2 𝑃𝑟𝑒𝑐𝐴 = 𝐶𝐴 + 𝐼𝐴 + 𝑃𝐴 + 𝑆𝐴 𝑃𝑟𝑒𝑐𝐴 ⋅ 𝑅𝑒𝑐𝐴 𝐹1𝐴 = 2 ⋅ 𝑃𝑟𝑒𝑐𝐴 + 𝑅𝑒𝑐𝐴 2.3.3. Optional Subtask B (Scenario 3) This scenario only evaluates Subtask B. The input is plain text and a corresponding .ann file with the correct entities annotated. The expected output is a .ann file with both entities and relations. For this to happen, the entity annotations from the provided .ann file can be copied with the relation annotations appended. To compute the scores we define correct, missing, and spurious matches. The expected and actual output files do not need to agree on the ID for each relation (which is ignored) nor on their order. The evaluation matches are based on the start and end of text spans and the corresponding type. A brief description about the metrics follows: Correct: relationships that matched the GOLD file exactly, including the type and the corre- sponding IDs for each of the participants. Missing: relationships that are in the GOLD file but not in the DEV file, either because the type is wrong, or because one of the IDs did not match. Spurious: relationships that are in the DEV file but not in the gold file, either because the type is wrong, or because one of the IDs did not match. We define standard precision, recall and F1 metrics as follows: 𝐶𝐵 𝑅𝑒𝑐𝐵 = 𝐶𝐵 + 𝑀𝐵 76 𝐶𝐵 𝑃𝑟𝑒𝑐𝐵 = 𝐶𝐵 + 𝑆𝐵 𝑃𝑟𝑒𝑐𝐵 ⋅ 𝑅𝑒𝑐𝐵 𝐹1𝐵 = 2 ⋅ 𝑃𝑟𝑒𝑐𝐵 + 𝑅𝑒𝑐𝐵 2.3.4. Optional Alternative Domain Evaluation (Scenario 4) This scenario evaluates a set of 100 sentences from an alternative domain (not health related), to experience with transfer learning techniques. A small development dataset with 100 sentences and their corresponding annotations will be provided when the general test set is released. Participants will need to train their systems in the full eHealth-KD 2020 corpus, and then apply some fine-tuning techniques in the additional 100 sentences from the alternative domain in order to successfully approach this scenario. The input and output format, and evaluation metrics are the same as for Scenario 1. The purpose of this scenario, which we consider a complex challenge, is to stimulate the development of systems that can generalise to new knowledge domains without too many additional training examples. Hence, we encourage participants to focus not only on ehealth- specific features and techniques, but also consider more generalizable approaches. 2.4. Corpus Description The corpus used in this edition of the challenge is composed of several sources reused from previous challenges, as well as new annotated content. The annotation guidelines and procedure followed were as described in Piad-Morffis et al. [2]. A total of 1000 training and development sentences are reused from the previous edition of the challenge, which is based on the same annotation model and methodology. For the test corpus, a new set of 300 sentences from Medline were manually annotated. An additional 200 sentences were selected from Wikinews, of which 100 were provided for development and 100 for testing in the evaluation Scenario 4. Finally, based on the submissions of the previous edition, an ensemble of 3, 000 sentences automatically annotated was constructed by aggregating the annotations produced by previous participants. These sentences have not been manually revised, hence they are provided as an additional resource for fine-tuning but should be used with care when training a new system. The general statistics of the corpus are summarised in Table 1. 3. Systems Description This section briefly describes the eight systems that were submitted to the challenge. In contrast with previous editions, there was a high degree of uniformity among participants, in the sense that most approaches involve the use of deep learning architectures with contextual or static embeddings. Nevertheless, there are interesting differences among the approaches which proved significant with respect to the results obtained. The participant teams and their corresponding systems are described next: 77 Table 1 Summary statistics of the eHealth-KD Corpus v2.0. Key phrases and relation labels are sorted by the number of instances in the training set. The training and development collections (marked with ∗ ) have been reused from previous editions. Metric Total Training DEV/Main DEV/Transfer Test Ensemble ∗ ∗ Sentences 3400 800 200 100 300 3000 Entities 25225 5012 1305 1242 2921 14745 - Concept 16207 3112 797 841 1944 9513 - Action 6431 1319 340 278 628 3866 - Predicate 1902 412 124 104 299 963 - Reference 685 169 44 19 50 403 Relations 20504 4571 1204 1241 2710 10778 - target 6376 1281 350 270 562 3913 - subject 3156 674 170 251 438 1623 - in-context 2503 502 140 193 380 1288 - is-a 2013 458 104 119 262 1070 - in-place 1250 304 77 111 237 521 - causes 890 292 71 30 92 405 - domain 994 269 74 82 196 373 - argument 857 254 73 47 185 298 - entails 308 117 43 11 28 109 - in-time 489 126 26 81 127 129 - has-property 1088 134 18 18 91 827 - same-as 346 93 31 19 66 137 - part-of 234 67 27 9 46 85 Vicomtech [3] presented an end-to-end deep neural network with pre-trained BERT models as the core for the semantic representation of the input texts. They experimented with two models: BERT-Base Multilingual Cased and BETO, a BERT model pre-trained on Spanish text. They model all the output variables—entities and relations—at the same time, modelling the whole problem jointly. Some of the outputs are fed back to the latter layer of the model, connecting the outcomes of the different sub-tasks in a pipeline fashion. TALP-UPC [4] presented an end-to-end deep neural network, for simultaneously identifying key-phrases and their relationships, that does not rely on any domain-specific knowledge nor handcrafted features. Input documents are parsed using FreeLing and encoded using either a BERT, a Word2Vec or a FastText pre-trained word-embedding model. In order to generate all possible relations, the model should be run for every input token and have the all raw likelihoods combined across every one of them. UH-MAJA-KD [5] presented a hybrid model for Subtask A that uses Stacked Bidirectional LSTM layers as contextual encoders, and linear chain Conditional Random Fields as tag decoders. The system addresses Subtask B in a pairwise query fashion, encoding information about the sentence and the given pair of entities using syntactic structures derived from the dependency parse tree, by the means of LSTM-based Recurrent Neural 78 Networks. IXA-NER-RE [6] presented a two-step model for the NER and RE sub-task, each of them independently developed from the other. The Name Entity Recognition task has been envisaged as a basic seq2seq system applying a general-purpose Language Model and static embeddings. In the RE sub-task, two approaches were explored: transfer learning methods and Matching the Blank to tackle the problem of the reduced size of the training corpus by producing relation representations directly from unlabelled text. UH-MatCom [7] presented several deep-learning models trained and ensembled to automati- cally extract the entities and relations. Their models use a combination of state of the art techniques such as BERT, Bi-LSTM, and CRF. They also explore the use of external knowledge sources such as ConceptNet. SINAI [8] presented a BiLSTM+CRF neural network where different word embeddings are combined as an input to the architecture: custom-generated medical embeddings, con- textualised non-medical embeddings, and pre-trained non-medical embeddings based on transformers. HAPLAP [9] presented a joint AB-LSTM neural network which combines a Bi-LSTM with max pooling and an attentive Bi-LSTM for the relation extraction task. The Joint AB-LSTM is fed with the pre-processed sentences, their entities and relations between those, and distance embeddings. ExSim [10] presented an information retrieval approach in which entities and relations in the training set are compared via word-embedding similarity to determine the most likely label. Baseline is a basic implementation that stores all pairs of entities and labels, and all triplets of tow entities and relation labels found in the training set, and simply outputs for the test set a label if it finds an exact match. The purpose of the baseline is to provide participants with a starting point that already takes care of loading the data, parsing the annotation format, and producing the right output. By far the most common type of approach corresponds to recurrent deep learning architec- tures (e.g., LSTM layers) with contextual embeddings (e.g., BERT). This combination is the basis of seven out of eight participant systems. This is not surprising given the recent success of these approaches in several NLP tasks, and in fact it was suggested in the Overview of previous editions of the eHealth-KD Challenge [11] [12]. Variations within this trend include the use of custom rather than pre-trained embeddings and the introduction of knowledge-based features. However, the most significant difference in approach corresponds to systems that perform an end-to-end strategy versus systems that solve each subtask separately. In the previous two editions of the challenge, the best performing system has used an end-to-end strategy. In this edition, two team (Vicomtech and TALP) deploy different end-to-end strategies. 79 3.1. Systems characteristics For describing each system we define a set of characteristics that group the different approaches used by the participants. These characteristics span from abstract concepts as using external knowledge to implementation details such as using transformers or other contextual embeddings. The purpose of these characteristics is to analyse what is common among the systems that perform best in each scenario and possibly identify interesting or unexplored techniques. The characteristics are described below. NLP: Using classic natural language processing features and strategies, such as TF-IDF encod- ing, stemming, lemmatization, dependency parsing, etc. Static embeddings: Using pre-trained word embeddings such as Word2Vec or Glove, trained on standard corpora. Contextual embeddings: Using contextual embeddings such as BERT or GPT, trained on standard corpora. Custom embeddings: Using any type of embedding with a custom dataset selected for this task or a fine-tunning process. Recurrent Network: Using any variant of recurrent neural networks, such as GRU or LSTM, possibly combined with other deep learning architectures. Knowledge Bases: Using any source of external semantic knowledge either to define features or to enrich the training set. End to end: Designing a single system that is simultaneously trained on both subtasks and shares at least a part of the features, representation or learning parameters for both entities and relations. 4. Results Table 2 summarises the results obtained by each participant in each evaluation scenario. Results are sorted by 𝐹1 in Scenario 1 which is considered the main evaluation. The top three results in each scenario are highlighted in bold. Overall the best performing system was presented by Vicomtech [3] which not only obtains the best result in Scenario 1 (by a significant margin), but also ranks among the top three in all scenarios. Likewise, the system proposed by Talp-UPC [4] obtains the top result in Scenario 4, which is considered the most difficult scenario given the short number of training examples. It is also worth mentioning the results obtained by UH-MAJA-KD, who also rank among the top results in all scenarios, and the difference with the previous best result is less that 0.001 in two scenarios, which can be considered statistically insignificant. Finally, it is interesting to note that the systems that obtained the best results for each individual task (i.e., SINAI in Scenario 2 and IXA-NER-RE in Scenario 3) do not rank among the top three in the general scenarios. This suggests an interesting trade-off between focusing on solving one specific task or designing a generally well-performing system. 80 Table 2 Results (𝐹1 metric) in each scenario, sorted by Scenario 1 (column Score). The top results per scenario are highlighted in bold. Score (𝐹1 ) Team Scn 1 Scn 2 Scn 3 Scn 4 Characteristics Vicomtech 0.665 0.820 0.583 0.563 Recurrent Network, Contextual embedding, End-to-end Talp-UPC 0.626 0.815 0.574 0.583 Recurrent Network, Contextual embedding, Static embedding, NLP, End-to-end UH-MAJA-KD 0.625 0.814 0.598 0.547 Recurrent Network, Contextual embedding, NLP IXA-NER-RE 0.557 0.691 0.633 0.478 Recurrent Network, Contextual embedding, Custom embedding UH-MatCom 0.556 0.794 0.545 0.373 Recurrent Network, Contextual embedding, NLP, Knowledge Bases SINAI 0.420 0.825 0.461 0.281 Recurrent Network, Contextual embedding, Custom embedding, Knowledge Bases HAPLAP 0.395 0.541 0.316 0.137 Recurrent Network, Contextual embedding ExSim 0.245 0.314 0.131 0.122 NLP, Static embedding 4.1. Analysis of Systems Performance According to the characteristics defined in Section 3.1, we performed a qualitative analysis of the most successful strategies in each scenario. Figure 2 shows a box-plot of the ranking obtained by systems with each of the characteristics above defined, per evaluation scenario. The box-plot shows the mean, inter-quartile ranges, and the minimum and maximum score among all systems with a given characteristic. As observed, the common strategy of using contextual embeddings and recurrent networks is capable of producing results in the full range of rankings. However, several systems have deployed and tailored this strategy, producing results with a range of variations. Hence, the use of BERT or LSTM layers alone does not guarantee a successful strategy. Likewise, as observed in previous editions, the use of custom embeddings seems to incur a marginal disadvantage, perhaps given that training high-quality embeddings in domain-specific corpora is difficult. On the other hand, the use of enternal knowledge bases to enrich semantic representations seems to be helpful in the entity recognition subtask, as exemplified by the result obtained by SINAI [8]. The single most successful approach seems to be the design of end-to-end architectures as opposed to solving both subtasks separately. This has been a trend in all the editions of the eHealth-KD challenge and is one of the most significant insights. The fact that end-to-end systems consistently outperform other approaches indicates that there is an interesting interaction between the semantic representation of entities and relations. Both end-to-end approaches presented provide an important advantage in terms of internal feedback exchange when resolving Subtask A and Subtask B, enhancing the discovery of entities and relations. This approach supports the idea that both subtasks are not completely independent of each other. However, as explained in Section 4, while end-to-end systems outperform all other approaches in Scenario 1 and 4, where both subtasks are performed, there are subtask-specific approaches that perform best when only one of the tasks is evaluated. 5. Conclusions and Future Work The eHealth-KD 2020 proposed –as with the previous editions eHealth-KD 2019[11] and eHealth- KD 2018[12]– the modelling of human language in a scenario in which Spanish electronic health documents could be machine-readable from a semantic point of view. With this task, we 81 Figure 2: Box-plot of the distribution of ranking for the systems that applied each of the approaches defined in Section 3.1 . expected to encourage the development of software technologies to automatically extract a large variety of knowledge from eHealth documents written in the Spanish Language. For this purpose, a new Spanish language corpus was manually annotated. Likewise, we provided tools to simplify the construction of knowledge discovery systems based on this corpus. In the challenge, eight systems were presented achieving a maximum F1 score of 0.665. All participants presented algorithms in all scenarios, with the the end-to-end systems obtaining best results. The most used significant change to 2020’s edition with respect to previous ones is the use of contextual embeddings (i.e., transformer architectures, and specifically BERT) as a replacement of static word embeddings. The results indicate that although promising approaches were presented in the challenge, the extraction of general-purpose semantic relations from natural language text is still an open area of research. Moreover, even though modern deep learning approaches are the most successful, we believe there is still a margin for improvement 82 by incorporating knowledge-based components that can exploit the structure of the annotation model. Acknowledgments This research has been partially supported by the University of Alicante and University of Havana, the Generalitat Valenciana (Conselleria d’Educació, Investigació, Cultura i Esport) and the Spanish Government through the projects SIIA (P R O M E T E O / 2 0 1 8 / 0 8 9 , P R O M E T E U / 2 0 1 8 / 0 8 9 ) and LIVING-LANG (R T I 2 0 1 8 - 0 9 4 6 5 3 - B - C 2 2 ). References [1] A. Piad-Morffis, Y. Guitérrez, S. Estevez-Velarde, R. Muñoz, A general-purpose annotation model for knowledge discovery: Case study in spanish clinical text, in: Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019, pp. 79–88. [2] A. Piad-Morffis, Y. Gutiérrez, Y. Almeida-Cruz, R. Muñoz, A computational ecosystem to support ehealth knowledge discovery technologies in spanish, Journal of Biomedical Informatics (2020) 103517. [3] A. García-Pablos, N. Perez, M. Cuadros, E. Zotova, Vicomtech at eHealth-KD Challenge 2020: Deep End-to-End Model for Entity and Relation Extraction in Medical Text, in: Proceedings of the Iberian Languages Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [4] S. Medina, J. Turmo, TALP at eHealth-KD Challenge 2020: Multi-Level Recurrent and Convolutional Neural Networks for Joint Classification of Key-Phrases and Relations , in: Proceedings of the Iberian Languages Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [5] A. Rodríguez Pérez, E. Quevedo Caballero, J. Mederos Alvarado, R. Cruz-Linares, J. P. Consuegra-Ayala, UH-MAJA-KD at eHealth-KD Challenge 2020: Deep Learning Models for Knowledge Discovery in Spanish eHealth Documents, in: Proceedings of the Iberian Languages Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [6] E. Andrés, O. Sainz, A. Atutxa, O. Lopez de Lacalle, IXA-NER-RE at eHealth-KD Challenge 2020: Cross-Lingual Transfer Learning for Medical Relation Extraction, in: Proceedings of the Iberian Languages Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [7] J. P. Consuegra-Ayala, M. Palomar, UH-MatCom at eHealth-KD Challenge 2020: Deep- Learning and Ensemble Models for Knowledge Discovery in Spanish Documents, in: Proceedings of the Iberian Languages Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [8] P. López-Ubeda, J. M. Perea-Ortega, D.-G. Manuel C., M. T. Martín-Valdivia, L. A. Ureña- López, SINAI at eHealth-KD Challenge 2020: Combining Word Embeddings for Named Entity Recognition in Spanish Medical Records, in: Proceedings of the Iberian Languages 83 Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [9] S. Santana, A. Pérez, A. Casillas, HapLap at eHealth-KD Challenge 2020, in: Proceedings of the Iberian Languages Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [10] Z. Hamzah Almugbel, ExSim at eHealth-KD Challenge 2020, in: Proceedings of the Iberian Languages Evaluation Forum co-located with 36th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2020, 2020. [11] A. Piad-Morffis, Y. Gutiérrez, J. P. Consuegra-Ayala, S. Estevez-Velarde, Y. Almeida-Cruz, R. Muñoz, A. Montoyo, Overview of the ehealth knowledge discovery challenge at iberlef 2019, in: Proceedings of the Iberian Languages Evaluation Forum co-located with 35th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2019, Bilbao, Spain, September 24th, 2019, 2019, pp. 1–16. URL: http://ceur-ws.org/Vol-2421/ eHealth-KD_overview.pdf. [12] E. M. Cámara, Y. Almeida-Cruz, M. C. Díaz-Galiano, S. Estévez-Velarde, M. Á. G. Cumbreras, M. G. Vega, Y. Gutiérrez, A. Montejo-Ráez, A. Montoyo, R. Muñoz, A. Piad-Morffis, J. Villena- Román, Overview of TASS 2018: Opinions, health and emotions, in: Proceedings of TASS 2018: Workshop on Semantic Analysis at SEPLN, TASS@SEPLN 2018, co-located with 34nd SEPLN Conference (SEPLN 2018), Sevilla, Spain, September 18th, 2018, 2018, pp. 13–27. URL: http://ceur-ws.org/Vol-2172/p0_overview_tass2018.pdf. 84