Most eHealth entity recognition and relation extraction models tackle the identi cation of entities and relations with independent specialized models. In this article, we show how a single combined model can exploit the correlation between these two tasks to improve the evaluation score of both, while reducing training and execution time. Our model uses both traditional part-of-speech tagging and dependency-parsing of the documents and state-of-the-art pre-trained Contextual Embeddings as input features. Furthermore, Long-Short Term Memory units are used to model close relationships between words while convolution lters are applied for farther dependencies. Our model was able to get the highest score in all three tasks of IberLEF2019's eHealth-KD competition[7]. This advantage was specially promising in the relation extraction tasks, in which it outperformed the second best model by a margin of 9.3% in F1 Score.
This article describes the model presented by the TALP team for IberLEF2019's
eHealth-KD [
IberLEF2019's eHealth-KD shared task supersedes and extends previous
year's Taller de Analisis Semantico en la SEPLN 2018's eHealth-KD
(TASS2018's eHealth-KD)[
The models presented for the aforementioned related task incorporate
combinations of several techniques such as Convolutional or Recurrent Neural
Networks, Support Vector Machines, Conditional Random Fields and even
rulebased systems. Our team concurred to the key-phrase classi cation and
relation extraction sub-tasks of TASS-2018's eHealth-KD with a joint CNN -based
model[
Given the similarity to the aforementioned tasks, we decided to rst try a
model for TASS-2018's eHealth-KD data set and then used the model weights as
an starting point. This idea of transferring some of the weights of a model trained
for a di erent task (transfer-learning ), has been extensively used in low-resource
machine learning tasks such as image classi cation, text analysis, question
answering and more[
The model takes a document and a token's index and computes the boundaries
and classes of the shortest key-phrase it belongs to, and the relations of every
other entities' tokens to it. Hence, the model should be run for each token of
the input document. This approach is inspired by attention-based translation
models such as Transformer [
The joint identi cation model's structure is visually described in Figure 1, and is composed of a set of shared layers and two independent output layers. Both output layers share the same structure, a fully connected layer and CRF; which respectively predict the target token's smallest entity sequence and each other token's relation to it. The core of the shared layer contains a recurrent layer composed by multiple bidirectional memory units (Either Gated Recurrent Units or Long-Short Term Memory units) followed by a convolution layer. The RNN and CNN's outputs are then fed to a fully connected layer with output dropout.
The recurrent and convolution layers allow for looking to both the local and global contexts of each input token. The local context is captured by the RNN Layer's output and the non-pooled convolution layer's output, which are concatenated for each time-step. The global context is captured by the maxpooled convolution layer's output. The global context information and the target token's local context information are added to all time-steps before being fed to the fully connected shared layer. 1 Two additional key-phrase classes (Predicate and Reference) with their related relation classes (in-time, in-place, in-context, domain and arg) were added. Moreover, one relation class was removed (property-of ) and others were added (same-as, hasproperty, causes and entails) Ycn Yrn Ycn+1 Yrn+1 Ycn+2 Yrn+2 Ycn+3 Yrn+3
CRF (Relations) CRF (Concepts)
FC FC FC On+1 On+2 +Concatenate + m 0.10 saa í 0.15 sva o a r 0.10 is-a saitrirspe
The nal outputs are then generated by a Conditional Random Field (CRF)
layer. Output CRF layers have proven to improve the capabilities of GRU and
LSTM networks in low-resource sequence tagging tasks[
Fuly-connected
M Cn+3 Max-pooling 0.45 0.10 0.85 0.25 0.00 0.00 0.45 caea ft 0.10 0.45 0.25 0.90 0.15 Subject 0.75 Target As described in Section 2, the model receives the sequence of tokens of a document and a token's index and outputs the bounds of the innermost key-phrase to which the token belongs. These bounds are encoded and decoded by assigning a Begin, Inside, Unitary and End tag to each token included in that key-phrase and Out to every other token (BIOUE-tag ). Note that just one key-phrase is decoded for each token index. Consequently, in order to identify all key-phrases, the model has to be evaluated for every token.
We approach relation extraction at the level of tokens. Given a token, the list of relations' probabilities between the innermost entity to which the token belongs and each one of the tokens in the document is predicted. Note that for the source token, we only consider the innermost entity whereas for the target tokens we consider all parent entities. This restriction is imposed so that the encoded sequence is not ambiguous. Other alternatives such as source and target encoding were also considered but ultimately discarded as the increased decoding complexity did not yield improved results.
A visual representation of relations' probability predictions is shown in Figure 2. Relations are predicted from the target key-phrase if the aggregated score inside a key-phrase span surpasses a threshold. Only the key-phrase with the highest score is selected if multiple key-phrases overlap. A pseudo-code of the relation decoder is listed below. Algorithm 1 Relation decoding algorithm 1: procedure decode(is; pc; E; R)
of class c 2: es innermost entity at index is from E 3: while not done do 4: fet; ptg f ; 0g 5: for e E do 6: p aggregate pci 8i 2 bounds of e
aggregation function 7: if p > pt ^ p > pth then 8: fet; ptg fe; pg then Rc S ffes; et; cgg 0 8i 2 bounds of et . Relations from token is with probabilities pc
. Pre-de ned probability . If probability is above the threshold . Probabilities in the span are set to 0 . Add relation to the relation set 2.2
In section 2 we describe how the model looks at sentences at the token level. The sentences are rst tokenized, tagged and dependency-parsed by FreeLing.
We represent each token by a vector, which results from the concatenation of the features listed below: { One-Hot encoding of the category and type elds of the token's Part-of
Speech Tag from FreeLing's tag-set.
{ Normalized vector encoding the dependencies found in the path between the
token and the target token (the one that is being decoded). It is computed
by adding the one-hot encoding representation of the dependency class for
each hop in the dependency path and normalizing the resulting vector, not
considering its direction. For instance, the representation of the token "I" in
"I eat sh" when the target token is " sh" would be a vector with p2 in the
positions corresponding to "subj" (subject) and "cd" (direct complement);
whereas for "eat" it would be a vector with a single 1 in the "cd" position.
{ One-Hot encoding of the distance between the token and the target token.
{ Contextualized word embedding of the token, computed by extracting the
weights of the last layer of a multi-language, general-purpose pre-trained2
2 We used the BERT-Base, Multilingual Cased model (104 languages, 12-layer,
768-hidden, 12-heads, 110M parameters) from the authors' repository (https://
github.com/google-research/bert)
Bidirectional Encoder Representations from Transformer model (BERT)[
No ne-tuning of the BERT model is done. 2.3
IberLEF2019's eHealth-KD ' training data-set is arguably small considering the
number of classes and the variability in the examples (3818 concepts and 3503
relations). In order to prevent over- tting and consequently increasing recall, we
opted for using previous years' TASS-2018's eHealth-KD [
The presented model was evaluated for tuning of hyper-parameters for using the provided development data-set. For this, we used the evaluation metrics and scripts provided by the challenge's organizers3. For more information about the data-set please refer to the tasks' overview paper.
In addition to this evaluation, we also evaluate the model learned in the pretraining step using TASS-2018's eHealth-KD 's testing data-set. For the sake of comparison, we also use the author's evaluation metrics and scripts and contrast the results to the challenge's contestants4.
Table 1 shows the relevant evaluation metrics of TASS-2018's eHealth-KD 's best-performing models for the three scenarios, compared to the presented model (Joint-RCNN ). The joint identi cation model clearly outperforms our previous model (talp) in all metrics, also beating the rest of the participants in sub-tasks B and C. In line with the original task's results, rriveraz 's model shows impressive results and surpasses our model in black-boxed sub-task A by a margin of 4:1% in F1 score, which ranks in second position.
Table 2 shows the shared task's nal evaluation results. Our model ranked in rst position for all three evaluation scenarios. The largest advantage resides in the relation extraction task, as it outscores the second runner by a margin of 9:6% in F1 score. After the competition was closed, we found that our model was outputting invalid combinations of key-phrase classes and relation classes. x-relations shows the evaluation of the xed model. 3 IberLEF2019's eHealth-KD data-set and evaluation scripts were downloaded from https://github.com/knowledge-learning/ehealthkd-2019. 4 TASS-2018's eHealth-KD data-set and evaluation scripts were downloaded from https://github.com/TASS18-Task3/data. In this article we have presented a joint concept and relation identi cation and classi cation model that exploits the mutual information between the entities and their relations by using a single network that looks at both local and global textual features. This newly presented model signi cantly outperforms all other competing models in both TASS-2018's eHealth-KD and IberLEF2019's eHealthKD shared tasks.
We hypothesize that the fact that the model shares both structure and weights allows the full model to more accurately capture the synergy between the two tasks and hence provide better precision and recall than traditional stepby-step models. In spite of this, the task is still not fully gured out, and we argue that further experimentation should be done in this line of research. We identify three noteworthy challenges: { Entities and relations are formally very di erent. The rst is usually encoded as a sequence and the second as a set of one-to-one labels. These two representations are di cult to combine, so more appropriate encoding is required to take full advantage of joint models. { Optimization functions are also hard to de ne, as they have to balance both entity recognition and relation extraction, taking into account the di erent amounts and di culty of instances of both tasks. { Depending on the model's structure, the optimization of the model's hidden layer's parameters for di erent outputs may be mutually opposite. These hidden layers should be designed to promote reciprocal collaboration between both objective functions. 5
Funding: This contribution has been partially funded by the Spanish Ministry of Economy (MINECO) and the European Union (TIN2016-77820-C3-3-R and AEI/ FEDER,UE).