This paper summarises the results of the third edition of the eHealth Knowledge Discovery (KD) challenge, hosted at the Iberian Language Evaluation Forum 2020. The eHealth-KD challenge proposes two computational tasks involving the identification of semantic entities and relations in natural language text, focusing on Spanish language health documents. In this edition, besides text extracted from medical sources, Wikipedia content was introduced into the corpus, and a novel transfer-learning evaluation scenario was designed that challenges participants to create systems that provide crossdomain generalisation. A total of eight teams participated with a variety of approaches including deep learning end-to-end systems as well as rule-based and knowledge-driven techniques. This paper analyses the most successful approaches and highlights the most interesting challenges for future research in this ifeld.
The vast amount of clinical text available online has motivated the development of automatic knowledge discovery systems that can analyse this data and discover relevant facts. These discoveries can be the base for novel treatments, understanding disease and drug interactions. Computational systems designed for this task are often trained on manually annotated corpora. To foster research in this area, the community has organised competitive challenges to identify, classify, extract, and link knowledge, such as in SEMEV1AaLnd CLEF campaigns2.
The eHealth Knowledge Discovery (eHealth-KD) challenge, in its third edition, leverages a semantic model of human language that encodes the most common expressions of factual knowledge, via a set of four general-purpose entity types and thirteen semantic relations among them. The challenge proposes the design of systems that can automatically annotate entities and relations in clinical text in the Spanish language. In this new edition, an alternative evaluation scenario (not related to the health domain) is also considered, which challenges participants to design systems that can successfully transfer their internal semantic representations from the health domain to an arbitrary new domain with considerably reduced training data. The challenge has been hosted at the Iberian Languages Evaluation Forum 2020, and included the participation of eight teams of researchers from diferent institutions.
This paper presents the design of the challenge as well as the data and tools provided to participants, and analyses the results obtained by each team. The remainder of the paper is organised as follows: Sectio2nprovides a detailed description of the tasks defined in the eHealth-KD challenge and the data provided for training and evaluation of knowledge discovery system, as well as all relevant evaluation metrics. Sec3tiborniefly describes all the solutions that were submitted to the challenge and introduces a set of characteristics that allow a qualitative comparison among them. Section4 presents the main results of the challenge, divided into four evaluation scenarios, and analyses the most successful and promising approaches deployed by each team. Finally, Sectio5n presents the conclusions of the research and recommendations for future editions.
The eHealth-KD challenge involves the identification of semantic entities and relations in natural language text. Although the focus has been on the health domain in past editions, the nature of the entities and relations extracted are general and can be applied to any domain. Figure1 shows an example of three sentences with the relevant entities and relations annotated. An in-depth explanation of the annotation model is provided in Piad-Morfis et a1l]. [
The evaluation of the challenge consists of submitting a set of natural language sentences with annotations automatically produced by a knowledge discovery system. Participants are provided with a set of manually annotated sentences (training and development corpus) that can be used for training and/or fine-tuning system as well as raw sentences that are used for evaluation (test corpus). The training and development corpus was provided two months in advance, but the test corpus was released only two weeks prior to the evaluation date, to discourage any fine-tuning on the test data. Although the actual source code of the system is not required, participants are encouraged to upload their code to open source code sharing services like Github.
To simplify the evaluation and provide more fine-grained comparisons the task is divided into two subtasks: one concerned with the identification and classification of entities, and the other concerned with the extraction of the semantic relations between these entities. is-a subject
in-time target in-context causes
target Con
Concept
Action
Con
Concept 1 El asma es una enfermedadque afecta las vías respiratorias.
Ref 3 Esta subject
++ Action Action
Concept
Con
Concept
target Action
Con 2 La exposición prolongadaal sol en verano provoca daños en la piel.
target
arg domain Concept
Pred
C Con afecta principalmentea las personas mayores de 60 años.
Given a list of eHealth documents written in Spanish, the goal of this subtask is to identify all the entities per document and their types. These entities are all the relevant terms (single word or multiple words) that represent semantically important elements in a sentence. The following ifgure shows the relevant entities that appear in a set of example sentences.
Some entities (“vías respiratorias” and “60 años”) span more than one word. Entities will always consist of one or more complete words (i.e., not a prefix or a sufix of a word), and will never include any surrounding punctuation symbols, parenthesis, etc. There are four types for entities: Concept: identifies a relevant term, concept, idea, in the knowledge domain of the sentence. Action: identifies a process or modification of other entities. It can be indicated by a verb or verbal construction, such aasfe“cta” (afects), but also by nouns, such as “exposición” (exposition), where it denotes the act of being exposed to the Sun, anddañ“os” (damages), where it denotes the act of damaging the skin. It can also be used to indicate non-verbal functional relations, such apsa“dre” (parent), etc.
Predicate: identifies a function or filter of another set of elements, which has a semantic label in the text, such asm“ayores” (older), and is applied to an entity, such apser“sonas” (people) with some additional arguments such a‘6s0‘ años” (60 years).
Reference: identifies a textual element that refers to an entity —of the same sentence or of diferent one—, which can be indicated by textual clues such aesst“a”, “aquel”, etc.
Subtask B continues from the output of Subtask A, by linking the entities detected and labelled in the input document. The purpose of this subtask is to recognise all relevant semantic relationships between the entities recognised. Eight of the thirteen semantic relations defined for this challenge can be identified in Figur1e. The semantic relations are divided into the following categories: General relations (6): general-purpose relations between two concepts (it involves Concept, Action, Predicate, and Reference) that have a specific semantic. When any of these relations apply, it is preferred over a domain relation –tagging a key phrase as a link between two information units–, since their semantic is independent of any textual label: is-a: indicates that one entity is a sub-type, instance, or member of the class identified by the other. same-as: indicates that two entities are semantically the same. has-property: indicates that one entity has a given property or characteristic. part-of: indicates that an entity is a constituent part of another. causes: indicates that one entity provokes the existence or occurrence of another. entails: indicates that the existence of one entity implies the existence or occurrence of another.
Contextual relations (3): enable an entity to be refined (it involves Concept, Action, Predicate, and Reference) by attaching modifiers. These are: in-time: to indicate that something exists, occurs or is confined to a time-frame, such as in “exposición” in-time “verano”. in-place: to indicate that something exists, occurs or is confined to a place or location. in-context: to indicate a general context in which something happens, like a mode, manner, or state, such ase“xposición” in-context “prolongada”.
Action roles (2): indicate what role the entities play related to an Action: subject: indicates who performs the action, such as in[el“] asma afecta […]”. target: indicates who receives the efect of the action, such as in[ …“] afecta [las] vías respiratorias”. Actions can have several subjects and targets, in which case the semantic interpreted is that the union of the subjects performs the action over each of the targets.
Predicate roles (2): indicate which role play the entities related to a Predicate: domain: indicates the main entity on which the predicate applies. arg: indicates an additional entity that specifies a value for the predicate to make sense.
The exact semantic of this argument depends on the semantic of the predicate label, such as in “mayores [de] 60 años”, where the predicate labeml“ayores” indicates that “60 años” is a quantity, that restricts the minimum age for the predicate to be true.
The eHealth-KD 2020 Challenge proposes four evaluation scenarios to measure diferent characteristics of the participant systems. We propose using a micro-avera g1etdhat weights all individual annotations equally, both entities and relations. Scenario 1 evaluates the solution to both tasks simultaneously, while Scenario 2 and 3 evaluate each task independently. Finally, Scenario 4 challenges systems to a novel domain with significantly less training data. This allows a more fine-grained comparison among systems with respect to specific capacities.
This scenario evaluates both subtasks together as a pipeline. The input consists only of a plain text, and the expected output is a BRAT.ann file with all the corresponding entities and relations found.
The measures will be precision, recall and F1 as follows:
= = 1 1 1 + + 2 + + + + +
= 2 ⋅ + + + + + + + 2 ⋅
+
The exact definition of Correct(C), Missing(M), Spurious(S), Partial(P) and Incorrect(I) is presented in the following sections for each subtask.
This scenario only evaluates Subtask A. The input is a plain text with several sentences and the output is a BRAT .ann file with only entity annotations in it (relation annotations are ignored if present).
To compute the scores we define correct, partial, missing, incorrect and spurious matches. The expected and actual output files do not need to agree on the ID for each entity, nor on their order. The evaluation matches are based on the start and end of text spans and the corresponding type. A brief description about the metrics follows: Correct matches are reported when a text in the development file —DEV— matches exactly with a corresponding text span in the gold file for START and END values, and also the entity type. Only one correct match per entry in the gold file can be matched. Hence, duplicated entries will count as Spurious.
Incorrect matches are reported when START and END values match, but not the type. Partial matches are reported when two intervals [START, END] have a non-empty intersection, such as the case ofv“ías respiratorias” and “respiratorias” in the previous example (and matching LABEL). Notice that a partial phrase will only be matched against a single correct phrase. For exampleti,p“o de cáncer” could be a partial match for botthip“o” and “cáncer”, but it is only counted once as a partial match with the wotripdo“”. The word “cáncer” is counted then as Missing. This aims to discourage a few large text spans that cover most of the document from getting a very high score.
Missing matches are those that appear in the GOLD file but not in the DEV file. Spurious matches are those that appear in the DEV file but not in the gold file.
From these definitions, we compute precision, recall, and a standard F1 measure as follows: = = 1 = 2 ⋅
1 + 2 + + +
1 + 2 + + +
⋅ +
This scenario only evaluates Subtask B. The input is plain text and a corresponding .ann file with the correct entities annotated. The expected output is a .ann file with both entities and relations. For this to happen, the entity annotations from the provided .ann file can be copied with the relation annotations appended.
To compute the scores we define correct, missing, and spurious matches. The expected and actual output files do not need to agree on the ID for each relation (which is ignored) nor on their order. The evaluation matches are based on the start and end of text spans and the corresponding type. A brief description about the metrics follows: Correct: relationships that matched the GOLD file exactly, including the type and the corresponding IDs for each of the participants.
Missing: relationships that are in the GOLD file but not in the DEV file, either because the type is wrong, or because one of the IDs did not match.
Spurious: relationships that are in the DEV file but not in the gold file, either because the type is wrong, or because one of the IDs did not match.
We define standard precision, recall and F1 metrics as follows: =
+
This scenario evaluates a set of 100 sentences from an alternative domain (not health related), to experience with transfer learning techniques. A small development dataset with 100 sentences and their corresponding annotations will be provided when the general test set is released. Participants will need to train their systems in the full eHealth-KD 2020 corpus, and then apply some fine-tuning techniques in the additional 100 sentences from the alternative domain in order to successfully approach this scenario. The input and output format, and evaluation metrics are the same as for Scenario 1.
The purpose of this scenario, which we consider a complex challenge, is to stimulate the development of systems that can generalise to new knowledge domains without too many additional training examples. Hence, we encourage participants to focus not only on ehealthspecific features and techniques, but also consider more generalizable approaches.
The corpus used in this edition of the challenge is composed of several sources reused from previous challenges, as well as new annotated content. The annotation guidelines and procedure followed were as described in Piad-Morfis et al2.][.
A total of1000 training and development sentences are reused from the previous edition of the challenge, which is based on the same annotation model and methodology. For the test corpus, a new set of300 sentences from Medline were manually annotated. An additio2n0a0l sentences were selected from Wikinews, of whic1h00 were provided for development an1d00 for testing in the evaluation Scenario 4. Finally, based on the submissions of the previous edition, an ensemble of3, 000 sentences automatically annotated was constructed by aggregating the annotations produced by previous participants. These sentences have not been manually revised, hence they are provided as an additional resource for fine-tuning but should be used with care when training a new system. The general statistics of the corpus are summarised in Ta1.ble
This section briefly describes the eight systems that were submitted to the challenge. In contrast with previous editions, there was a high degree of uniformity among participants, in the sense that most approaches involve the use of deep learning architectures with contextual or static embeddings. Nevertheless, there are interesting diferences among the approaches which proved significant with respect to the results obtained. The participant teams and their corresponding systems are described next:
Sentences Entities - Concept - Action - Predicate - Reference Relations - target - subject - in-context - is-a - in-place - causes - domain - argument - entails - in-time - has-property - same-as - part-of
Total
3400
25225
16207
6431
1902
685
20504
6376
3156
2503
2013
1250
890
994
857
308
489
1088
346
234
800∗
5012
3112
1319
412
169
4571
1281
674
502
458
304
292
269
254
117
126
134
93
67
Vicomtech [
UH-MAJA-KD [
Test 300 2921 1944 628 299 50 2710 562 438 380 262 237 92 196 185 28 127 91 66 46 3000 14745 9513 3866 963 403 10778 3913 1623 1288 1070 521 405 373 298 109 129 827 137 85
IXA-NER-RE [
SINAI [
HAPLAP [
ExSim [
Baseline is a basic implementation that stores all pairs of entities and labels, and all triplets of tow entities and relation labels found in the training set, and simply outputs for the test set a label if it finds an exact match. The purpose of the baseline is to provide participants with a starting point that already takes care of loading the data, parsing the annotation format, and producing the right output.
By far the most common type of approach corresponds to recurrent deep learning
architectures (e.g., LSTM layers) with contextual embeddings (e.g., BERT). This combination is the basis
of seven out of eight participant systems. This is not surprising given the recent success of
these approaches in several NLP tasks, and in fact it was suggested in the Overview of previous
editions of the eHealth-KD Challeng1e1[] [
For describing each system we define a set of characteristics that group the diferent approaches used by the participants. These characteristics span from abstract concepts as using external knowledge to implementation details such as using transformers or other contextual embeddings. The purpose of these characteristics is to analyse what is common among the systems that perform best in each scenario and possibly identify interesting or unexplored techniques. The characteristics are described below.
NLP: Using classic natural language processing features and strategies, such as TF-IDF encoding, stemming, lemmatization, dependency parsing, etc.
Static embeddings: Using pre-trained word embeddings such as Word2Vec or Glove, trained on standard corpora.
Contextual embeddings: Using contextual embeddings such as BERT or GPT, trained on standard corpora.
Custom embeddings: Using any type of embedding with a custom dataset selected for this task or a fine-tunning process.
Recurrent Network: Using any variant of recurrent neural networks, such as GRU or LSTM, possibly combined with other deep learning architectures.
Knowledge Bases: Using any source of external semantic knowledge either to define features or to enrich the training set.
End to end: Designing a single system that is simultaneously trained on both subtasks and shares at least a part of the features, representation or learning parameters for both entities and relations.
Table2 summarises the results obtained by each participant in each evaluation scenario. Results are sorted b y 1 in Scenario 1 which is considered the main evaluation. The top three results in each scenario are highlighted in bold.
Overall the best performing system was presented bVyicomtech [
Finally, it is interesting to note that the systems that obtained the best results for each individual task (i.e.S,INAI in Scenario 2 andIXA-NER-RE in Scenario 3) do not rank among the top three in the general scenarios. This suggests an interesting trade-of between focusing on solving one specific task or designing a generally well-performing system.
According to the characteristics defined in Sectio3n.1, we performed a qualitative analysis of the most successful strategies in each scenario. Figu2rsehows a box-plot of the ranking obtained by systems with each of the characteristics above defined, per evaluation scenario. The box-plot shows the mean, inter-quartile ranges, and the minimum and maximum score among all systems with a given characteristic.
As observed, the common strategy of using contextual embeddings and recurrent networks
is capable of producing results in the full range of rankings. However, several systems have
deployed and tailored this strategy, producing results with a range of variations. Hence, the use
of BERT or LSTM layers alone does not guarantee a successful strategy. Likewise, as observed
in previous editions, the use of custom embeddings seems to incur a marginal disadvantage,
perhaps given that training high-quality embeddings in domain-specific corpora is dificult.
On the other hand, the use of enternal knowledge bases to enrich semantic representations
seems to be helpful in the entity recognition subtask, as exemplified by the result obtained
by SINAI [
The eHealth-KD 2020 proposed –as with the previous editions eHealth-KD 20119][and
eHealthKD 2018[
In the challenge, eight systems were presented achieving a maximum F1 score0o.6f65. All participants presented algorithms in all scenarios, with the the end-to-end systems obtaining best results. The most used significant change to 2020’s edition with respect to previous ones is the use of contextual embeddings (i.e., transformer architectures, and specifically BERT) as a replacement of static word embeddings. The results indicate that although promising approaches were presented in the challenge, the extraction of general-purpose semantic relations from natural language text is still an open area of research. Moreover, even though modern deep learning approaches are the most successful, we believe there is still a margin for improvement by incorporating knowledge-based components that can exploit the structure of the annotation model.
This research has been partially supported by the University of Alicante and University of Havana, the Generalitat ValencianCaon(selleria d’Educació, Investigació, Cultura i Esport) and the Spanish Government through the projects SIIAPR(OMETEO/2018/089, PROMETEU/2018/089) and LIVING-LANG (RTI2018-094653-B-C22).