The eHealth Knowledge Discovery Challenge, hosted at IberLEF 2019, proposes an evaluation task for the automatic identi cation of key phrases and the semantic relations between them in health-related documents in Spanish language. This paper describes the challenge design, evaluation metrics, participants and main results. The most promising approaches are analyzed and the signi cant challenges are highlighted and discussed. Analysis of the participant systems shows an overall trend of sequence-based deep learning architectures coupled with domain-speci c or domain-agnostic unsupervised language representations. Successful approaches suggest that modeling the problem as an end-to-end learning task rather than separated in two subtasks improves performance. Interesting lines for future development were recognized, such as the option of increasing the corpus size with semi-automated approaches and designing more robust evaluation metrics.
Knowledge discovery is a growing eld in computer science, with applications in
several domains, from databases [
Speci cally in the health domain, there is a growing number of scienti c
publications that are virtually impossible to analyze manually. This surplus of
data encourages the design of knowledge discovery systems that can leverage
the large amount of information available for building, for example, automated
diagnostic systems [
The representation model used in eHealth-KD 2019 [
This paper describes and evaluates the results of the 10 di erent systems designed by the participants in the 2019 edition of the eHealth Knowledge Discovery Challenge. Additional insights on the most promising lines for future research are outlined. Section 2 describes the challenge, evaluation criteria and corpora. Section 3 brie y describes the solutions presented in the challenge. Section 4 presents the main results and additional analysis about the best performing approaches. Finally, Section 5 discusses the main highlights of the challenge, and Section 6 concludes and provides ideas for future development. 2
Even though this challenge is oriented to the health domain, the structure of the
knowledge to be extracted is general-purpose. The semantic structure proposed
models four types of information units. Each one represents a speci c semantic
interpretation, and they make use of thirteen semantic relations among them.
The following sections provide a detailed presentation of each unit and relation
type. Additional details about the annotation model and the exact semantic
de nition of each entity and relation are available in [
Based on previous experience with similar challenges, the process for identifying the entities and relations de ned is divided in two subtasks. The rst subtask deals with identifying the spans of text that de ne entities, and their categories (see Section 2.1). The second subtask deals with identifying the semantic relations that connect the entities previously identi ed (see Section 2.2). 2.1
Given a list of eHealth documents written in Spanish language, the goal of this subtask is to identify all the key phrases per document and characterise them with the concepts (i.e. classes) that represent them. These key phrases are all the relevant terms (single word or multiple words) that represent semantically important elements in a sentence. Figure 1 shows the relevant key phrases that appear in an example set of sentences.
Some key phrases (e.g., \v as respiratorias " and \60 an~os") span more than one word. Key phrases always consist of one or more complete words (i.e., not a pre x or a su x of a word), and never include any surrounding punctuation symbols. There are four categories or classes for key phrases: Concept: a general category that indicates the key phrase is a relevant term, concept, idea, in the knowledge domain of the sentence.
Action: a concept that indicates a process or modi cation of other concepts. It can be indicated by a verb or verbal construction, such as \afecta" (a ects), but also by nouns, such as \exposicion " (exposition), where it denotes the act of being exposed to the Sun, and \dan~os" (damages), where it denotes the act of damaging the skin.
Predicate: used to represent a function or lter of another set of elements, which has a semantic label in the text, such as \mayores " (older), and is applied to a concept, such as \personas " (people) with some additional arguments such as \60 an~os " (60 years). Reference: A textual element that refers to a concept {in the same sentence or in di erent one{, which can be indicated by textual clues such as \esta", \aquel ", and similar.
The input for Subtask A is a text document with a sentence per line. All sentences have been tokenized at the word level (i.e., punctuation signs, parenthesis, etc, are separated from the surrounding text). 2.2
Subtask B bene ts from the output of Subtask A, by linking the key phrases detected and labeled in each sentence. The purpose of this subtask is to recognize all relevant semantic relationships between the entities recognized. Eight of the thirteen semantic relations de ned for this challenge can be identi ed in Figure 2.
The semantic relations are divided into di erent categories: General relations (6): general-purpose relations between two concepts that have a speci c semantic: is-a, same-as, has-property, part-of, causes, and entails.
Contextual relations (3): allow a concept to be re ned by attaching the modi ers: in-time, in-place, and in-context.
Action roles (2): indicate which concepts play a role related to an Action, which can be subject and target.
Predicate roles (2): indicate concepts play a role in relation to a Predicate, which can be the domain and additional arguments. 2.3
The challenge proposed a main evaluation scenario (Scenario 1) where both subtasks, previously described, are performed in sequence. The submission that obtained the highest F1 score for the Scenario 1 was considered the best overall performing system of the challenge. Additionally, participants had have the opportunity to address speci c subtasks by submitting to two optional scenarios, once for each subtask. These two additional scenarios measured the performance in individual subtasks independently of each other.
Scenario 1 is considered more complex than solving each optional scenario separately, since errors that systems generate when facing the subtask A are transmitted to subtask B. For this reason it is considered the main evaluation metric. Additionally, this scenario also provides the possibility of integrating endto-end solutions that solve both subtasks simultaneously. The evaluation metric is a standard F1 where precision and recall are de ned in terms of (C)orrect, (M)issing, (S)purious, (I)ncorrect and (P)artial matches. Incorrect matches are reported when key phrases are correctly identi ed regarding the text span, but they are not assigned to the correct category. Partial matches are reported when key phrases overlap but do not match exactly with the correct text span.
A higher precision means that the number of spurious identi cations is smaller compared to the number of missing identi cations, and a higher recall means the opposite. Partial matches are given half the score of correct matches, while missing and spurious identi cations are given no score. The evaluation formulas for scenario 1 are de ned as follows:
RecallAB = P recisionAB =
F1AB = 2
CA + CB + 21 PA CA + IA + CB + PA + MA + MB
CA + CB + 21 PA CA + IA + CB + PA + SA + SB
P recisionAB RecallAB P recisionAB + RecallAB (1) (2) (3)
Likewise, similar formulas are de ned for scenarios 2 and 3, using respectively only the statistics for subtask A and B. Additional details about the evaluation metrics are available in the eHealth-KD Challenge website4. 2.4
For the purpose of the challenge, a corpus containing 1; 045 sentences was distributed in several collections to participants. A set of 600 sentences for training and 100 for model validation was distributed in the rst stage along with gold annotations. For the test phase, 300 sentences were distributed, 100 per scenario, and gold annotations were kept blind until the end of the challenge. An additional 8,700 unannotated sentences were distributed in the test phase, which can be used for a semi-automatic extension of the corpus via an ensemble of the best performing submissions. All 8; 800 sentences in scenario 1 were shu ed; hence, participants had no information on which were the actual 100 or the 8; 700 additional sentences, and were thus forced to submit responses for all the sentences. 4 https://knowledge-learning.github.io/ehealthkd-2019 This also had the e ect of discouraging a manual annotation or other forms of gaining unfair advantage on the test set.
The corpus annotation process followed closely the methodology proposed in
the previous edition [
Sentences Key phrases - Concept - Action - Predicate - Reference
1,045
In the eHealth-KD challenge 2019, 30 teams were registered from which 10
submitted their approaches successfully. They were characterized by the use of a
variable range of algorithms and techniques. The most common approaches
involved knowledge bases, deep learning and natural language processing
techniques. This section brie y describes each participant system. To simplify the
comparison and better understand the characteristics of each system, we de ne
several tags to describe the kind of techniques used by each team: (C)onditional
(r)andom elds; (P)retrained or (C)ustom word embeddings; (Ch)aracter-level
embeddings; hand-crafted (R)rules; natural language processing (F)eatures;
dealing with the (O)verlapping of entities; (At)tention mechanisms; (Co)nvolutional
layers; dataset (Au)gmentation techniques; and, if they solve both subtasks in a
(J)oint form rather than separated. The 10 systems are subsequently described,
and they are distinguished by the name of the team responsible for their creation.
coin ipper (P-R-F) [
Hulat-TaskA (Cr-P-Ch-Au) [
IxaMed (Cr-Cu-F-At) [
LASTUS-TALN (Cr-Cu-F-At) [
LSI2 UNED (P-Ch-F-Co) [
NLP UNED (P-F-At) [
entities and relations simultaneously using BERT embedded sentences combined with GRUs and Convolutional architectures. Both Subtasks are solved at the same time, modelling the dependency between entity labels and the possible relations between them. They reuse the previous challenge data to improve performance.
architecture, with word embeddings trained in a Wikipedia-based medical
corpus, and additional POS tagging features in Subtask A. For Subtask B,
the model is a Bi-LSTM multiclass classi er that uses the longest path
between keyphrases in the dependency tree as phrase-level features.
VSP (-) [
Baseline (R): A hand-crafted baseline was built by the challenge organizers to provide a minimum working solution for participants and a measuring point. This baseline stores every key phrase and relation tuple seen in the training set, and outputs the exact label when a 100% match is found in the set.
By far the most common approach involves deep learning architectures, speci cally Bi-LSTM layers, which some teams combine with other types of neural network architectures. This is to be expected, since LSTM architectures are commonly used for natural language processing given their ability to learn correlations between elements of a sequence. Several systems use Conditional Random Fields (CRF) to decode the outputs for Subtask A. In contrast with the previous edition, there are no pure rule-based or knowledge-based approaches, although some systems incorporate domain knowledge in the form of custom embeddings. One team (LSI2 UNED) uses Wikidata entities, which can be considered a knowledge-based approach combined with a deep learning architecture. Two teams (IxaMed and UH-Maja-KD) train custom embeddings on external sources with domain knowledge, which can be considered an unsupervised approach. All teams except one (TALP-UPC) solve both subtasks separately, even though some reuse the same architecture in both. 4
The results obtained by each team are summarized in Table 2 and are ranked in order of best performance for Scenario 1. Highlighted in bold are the top three results per scenario, except for Scenario 3 (Subtask B) where four results are highlighted because two of them are very similar.
Overall, the best performing system was presented by TALP-UPC [
In Subtask A, the top three systems obtain very similar results, which can be explained in part by the similarity of their approaches, i.e., LSTM-based architectures with di erent types of embeddings as input features. In Subtask B, a larger margin exists between the top result and the rest, which is an argument in favor of end-to-end solutions. However, since the architectures of di erent submissions have di erent characteristics, it is unclear whether this advantage comes from a better model or actually from the joint training. Further experimentation is necessary to determine the degree to which end-to-end training in uences the overall performance. 4.1
In this section we present an analysis of the performance of participant systems with respect to two qualitative criteria. First, we analyze the characteristics (as de ned by the tags in Section 2) that are correlated with a higher performance in each scenario. Next, we analyze the di culty of recognizing each type of annotation independently, and the impact of having more annotations.
To analyze the most signi cant strategies and approaches, we t a linear regression model on the challenge results. For each participant, this model approximates its score as a weighted average of the tags that describe the corresponding system. For example, for the team coin ipper with description P-R-F and index 2 in the table, the approximation formula is WP +WR +WF +error2 = 0:621 for Scenario 1, and correspondingly for all teams and scenarios (except the baseline). The weights that minimize the approximation error P errori2 are thus considered as the relative impact of a speci c tag. The R2 score for all three scenarios is respectively 0:773, 0:857 and 0:936 which indicate that these tags provides an adequate, if not perfect, description of the evaluated systems. Table 3 shows the weighting adjustment for all tags and all evaluation scenarios.
According to these weightings, one of the most signi cant factors for increasing performance in Scenario 1 is the use of an end-to-end system that solves all tasks jointly. This was expected since the most e ective system created by (TALP) is the only one that exhibits this feature. Other signi cant factors include: using NLP features in addition to word embeddings; employing some form of dataset augmentation; and, adding custom domain rules (e.g., identifying which tokens to merge into a single key phrase, such as done by coin ipper). The use of custom word embeddings (trained on domain-speci c datasets), as opposed to generic word embedding produces a marginally negative e ect. This may be due to the di culty of training embeddings on domain-speci c text, where its hard to obtain a su ciently large corpus.
In Scenario 2 (subtask A), solving the overlapping problem provides a marginal advantage, since it increases the recall of some overlapping key phrases that otherwise would be missing. The use of customized rules to solve the key phrase discontinuities (e.g., as applied by UH-Maja-KD) are also a relevant strategy, since several key phrases are not always formed by continuous tokens. Considering the overlapping issue is key to Scenario 3 (subtask B) also, presumably because otherwise all the relations between unreported overlapping key phrases would be counted as missing. The next most important feature is the use of attention mechanisms, which obtain a negative weighting in previous scenarios, but appear to be favorable in subtask B. Attention mechanisms could aid in identifying complex semantic relations that are far apart in the same sentence, in which LSTM networks alone fail to capture long-range dependencies.
Table 4 shows the cumulative distribution of correct matches for each type of annotation. For each instance of each annotation, we count the number of systems that output that speci c annotation correctly. Then we report the percentage of each type of annotation (key phrase or relation) that is correctly identi ed by at least X systems. Hence, these results are more indicative of recall than precision (without considering partial matches). Given that systems could produce unlimited spurious annotations, measuring a similar distribution with respect to precision is unfeasible.
Since several teams did not participate in Subtask B (relation extraction), it is to be expected that relations have a lower recall than key phrases in general. However, as explained in Section 4, the best performing systems in Subtask B obtained a lower score than in Subtask A. Both these factors indicate that Subtask B is considerably more di cult to solve than Subtask A.
With respect to speci c key phrase labels, Concepts appear to be marginally easier to identify than Actions and the remaining labels. Given that Concepts Annotation Key phrases Relations Concept Action Predicate Reference
1 are considerably more frequent in the dataset than the remaining labels, a larger di erence is to be expected. This may be an indication that low-dimensional features (such as POS-tags) are likely to be su cient to di erentiate key phrases from non key phrases, since a surplus of annotation does not produce a similar improvement in recall.
Regarding relations, the distribution shows that the least common types are also considerably harder to recognize. Given the unbalanced nature of the corpus, some participants e ectively decided not to target all possible labels, and only consider the most common ones. Increasing the number of output predictions can harm a model's performance more than the relative improvement in F1 score, especially when some labels have a marginal impact on the overall score, given their low count. This situation creates a scenario where it is preferable to simply not consider some of the labels. In future challenges we will reconsider the scoring metrics to mitigate this e ect. Key phrases or relations that appear more frequently in the training set are found to be more easily identi able from the semantic perspective. Figure 3 shows a scatter plot of all the annotation types. The horizontal axis measures their relative rank with respect to instances in the training set, i.e, annotation types are ordered from left to right according to frequency. The vertical axis measures the relative rank of annotation type with respect to the average number of systems that identify them; for example, annotation types are ranked in ascending order according to identi cation complexity {IC{. A perfect correlation between the instances in the training set and their IC would be represented by a diagonal arrangement of annotatation types. Annotations above the diagonal (e.g., Reference) are considerably easier to identify even with a lower frequency, whereas annotations below the diagonal (e.g., causes) are more di cult regardless of the higher frequency.
e r o M d e iif t n e d i s e c n a t s n I s s e L
Predicate in-place
Concept
Action target in-context is-a
subject entails argument domain
causes same-as has-property
in-time part-of
Less -- Instances in training set -- More
The correlation coe cient between these two magnitudes (i.e., rank by frequency in corpus and IC) is 0:811, which, as expected, indicates a high relation between the number of annotations of a given type and how easy they are to identify. However, since correlation is not perfect, there is still a factor of variance that needs explanation. For example, References are considerably easier to identify than what their frequency would suggest, since there are only 215 instances in the training set. In contrast, causes annotations have a higher frequency but a much lower recall overall. This is to be expected, since Reference annotations arguably have less syntactic variation than all the patterns in which, for example, a causality can be expressed. These are examples of the general hypothesis that key phrases are consistently easier to identify correctly than relations. 5
The results of the eHealth-KD Challenge 2019 show the task of knowledge discovery in Spanish health-related documents is still challenging. However, important advances have taken place since the previous edition, which indicate that research in this area is active and progressing. Most approaches have converged towards a common factor, i.e., using Bi-LSTM models, possibly coupled with other, more sophisticated, deep learning techniques. Solving both tasks with an end-to-end system appears to be a promising approach, although more experiments are necessary to e ectively measure the impact of this design strategy isolated from other models and training strategies. In contrast with previous challenges, domain-speci c knowledge did not provide a signi cant advantage against black-box deep learning methods. However, some domain-speci c rules for solving key phrase overlapping and discontinuity issues do increase performance. As indicated earlier, the subtask B of relation extraction is considerably more di cult to solve than the key phrase identi cation, although subtask A is still not completely solved, given the large number of di erent annotation types de ned.
The large correlation between identi ed annotations and their relative frequency in the training set suggests that there is still a large space for improvement simply by using more annotations. Since the corpus was not intentionally balanced in terms of the di erent annotation types, the less common patterns (e.g., part-of ) naturally occurred less frequently. A possible suggestion that arises from this analysis is considering oversampling the less frequent patterns during annotation, to ensure a more balanced training set. Likewise, systems that perform dataset augmentation or transfer learning from similar domains will bene t from additional training examples. To this end, we will pursue the construction of a larger, semi-automated corpus, by means of pooling the annotations provided by participants in the 8; 700 raw sentences included in Scenario 1.
An interesting issue that emerges from this analysis is the design of a better evaluation metric. The F1 score de ned, though intuitive, promotes undesirable behaviors when attempting to optimize the score. For example, since all annotation types are micro-averaged, the less frequent ones have a much smaller impact on the overall score. Since adding more outputs to a model usually increases the parameters and harms learning in general, systems optimizing F1 could potentially completely ignore the least frequent relation types and improve their score. On the other hand, it is still unclear how to balance the relative importance of subtask A and subtask B in a single metric, especially since mistakes in subtask A necessary translate to mistakes in subtask B. However, small mistakes in subtask A can have a large impact on subtask B, since a single missing or spurious key phrase can participate in many relations.
Finally, the F1 score fails to capture the essence of the problem at hand, which is extracting the semantic meaning of a sentence. Since the F1 score measures each decision independently, two systems can obtain the same score even though one makes a \small" mistake by missing, for example, an argument, while the other may leave the sentence completely disconnected by failing to recognize an entailment between two main ideas. This suggests the need to design a more robust metric that promotes systems which attempt to solve both subtasks effectively and correctly captures the relative importance of the di erent semantic elements to be identi ed.
The eHealth-KD Challenge 2019 presented a problem of key phrase identi cation and relation extraction in Spanish health-related texts. A total of 10 teams presented a variety of approaches, with a common factor involving the use of Bi-LSTM networks and embedding-based representations. An analysis of the most successful approaches indicates that some domain-speci c rules are helpful, even though most of the progress has been achieved with domain-agnostic representations and generic NLP features. An interesting open issue is the use of end-to-end systems that solve both subtasks simultaneously versus a more classic pipeline with a speci c design tailored for each subtask.
The most immediate e orts will focus on using the 8; 700 automatically
annotated sentences to build a semi-automatic corpus by pooling the predictions
of the most e ective systems. This corpus will then be used to train the most
promising models and con rm the impact of additional data. Given that most
approaches are domain-agnostic, in future challenges we will introduce
crossdomain tasks that require generalizable models. We are also interested in the
design of alternative evaluation metrics that capture the semantic nature of the
task. Finally, given the variety of models proposed, we will investigate the use of
ensembles and Automatic Machine Learning (AutoML) techniques [
Funding: This research has been supported by a Carolina Foundation grant in agreement with University of Alicante and University of Havana. Moreover, it has also been partially funded by both aforementioned universities, the Spanish Government( Ministerio de Econom a y Competitividad) and the Generalitat Valenciana (Conselleria d'Educacio, Investigacio, Cultura i Esport) through the projects PROMETEU/2018/089, RTI2018-094653-B-C22 and RTI2018-094649B-I00.
The authors would like to thank the team of annotators from the School of Math and Computer Science, at the University of Havana.