In this paper we present the procedure followed to complete the run submitted by the UPF-UPC team to the TASS 2018 Task 3 challenge. Such procedure may be classi ed, according the organization's codes, as H-KB-S as it takes pro t from a knowledge based methodology as well as some supervised methods. Our pipeline includes: i) A standard pre-process of the documents using Freeling tool suite (POS tagging and dependency parsing); ii) Use of a CRF sequence labelling tool for completing both subtasks A (key phrase identi cation) and B (key phrase classi cation), and iii) Facing the subtask C (setting semantic relationships) by using a hybrid approach that uses two Logistic Regression classi ers, followed by lexical shallow relation extractors for entity/entity pairs related by is-a and same-as relations.
Text mining and natural language processing (NLP) techniques have been applied to the biomedical domain for a long time. Automatic identi cation of relevant terms in medical texts (research and educational material as well as medical reports) and how they relate each other represent a major improvement for indexing and for search tools. Its results are useful for research as well as clinical and educational purposes.
In this paper we present two tools for facing the tasks proposed in the TASS-2018Task 3 challenge: eHealth Knowledge Discovery. The rst one is a term extraction tool for nding terminologically relevant substrings (key phrases) and classify them in one of the two classes proposed by the organization: Concept or Action. The second one is dedicated to recognize those semantic relationships chosen by the Organization between the recognized entities.
Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes. 2 2.1
TASS 2018 Task 3
The corpus for this competition was compiled by sampling XML les produced by the National Library of Medicine, the world's largest medical library. It brings information about diseases, conditions, and wellness issues in understandable language. The full collection is available at https://medlineplus.gov/xml.html.
Given a collection of eHealth documents written in Spanish, TASS 2018 Task 3 has been conceived as a three task for three different scenarios. Each task may be described as follows: A) To identify all the key phrases per document; B) To assign a label (Concept or Action) to each of the key phrases; C) to link the entities detected and labelled in each document through the following semantic relationships: (a) Concept-Concept: is-a,
property-of and same-as; (b) Action-Concept: subject and target. part-of, The output of each task is the input of the next one. Proceeding in this way the organization has considered the following three evaluation scenarios: 1. Only plain text is given (Subtasks A, B and C must be completed); 2. Plain text and manually annotated key phrase boundaries are given (Subtasks B and C must be completed); 3. Plain text with manually annotated key phrases and their types are given (only Subtask C must be completed).
More details about the tasks and scenarios
may be obtained through the web site of the
TASS-2018 competition1 and the overview
paper
After downloading the documents we have
processed them using Freeling tool suite2,
2.2
3/
1http://www.sepln.org/workshops/tass/2018/task
For subtasks A and B we proceeded jointly
using a single tool that is able to select the
set of nominal term candidates, TC, included
in the text under analysis. This approach
is based in YATE, an in house term
extractor that has been tuned for treating medical
text. See
Term extraction can be seen as semantic annotation task because it provides machinereadable information based on meaning. The way to attack the problem varies according the available resources for each language. Some languages (mainly English) disposes of lexical resources (like ontologies and/or term repositories) that can be used for reference while other languages have to identify term within text using other procedures that include linguistic/statistical strategies.
3Freeling can provide the set of possible synsets
for each token when a WordNet is available. In the
case of Spanish EuroWordNet is used.
YATE is a hybrid system whose
keypoints are: i) the combination of
heterogeneous detection strategies and ii) the use
semantic knowledge in such strategies.
Initially, it used the lexical ontology
EuroWordNet
After applying standard linguistic analysis procedures, it starts by extracting a basic list of TCs. Such candidates are then analysed using a collection of heterogeneous methods.
A rst method to evaluate the termhood of any candidate is its domain coe cient. It is calculated using the above mentioned domain borders and indicates in which degree a given TC belong to the domain of interest (medicine in this case).
Other methods included in YATE are based on statistical information, context information and the result to decompose a TC in its graeco-latin components. In the original tool all these informations were combined using a voting scheme or a boosting algorithm. For this competition, we decided to add information from Snomed-CT 5 and also, due to its speci c requirements (analysing every mention of each nominal or verbal TC), we modify the original term extraction tool and chose to use a CRF classi er6 for combining all the available information and to predict the BIO tag7 for each token. In order to select which pieces of information are given as an input to the model a template has to be built for each TC. It allows to take into account both the features associated to the target token as well as the ones of its neighbours. The set of features chosen for each word in this task was the following: • The lemmas of the target token and of the tokens appearing in a size-2 window around it; 4A domain border is de ned as an EWN synset likely belonging itself and its descendants to such domain (eg. disease, bodypart and medical procedure among many others).
5A comprehensive and well known clinical health terminology available for several languages including Spanish, https://www.snomed.org/ 6CRF++, https://taku910.github.io/crfpp/ 7BIO is a popular way of tagging tokens for detecting useful sequences, B stands for beginning of a sequence, I for inside it and O for out of it. • The reduced POS tag8 of the target token and the tokens appearing in a size-2 window around it; • The domain border to each of the tokens of the string detected as a TC ; • The main class for those TC included in Snomed-CT9. This information is applied to all the tokens of the string detected as a TC ; • The rst/last three letters of each token. 2.4
For facing the subtask C we learned two multi-class classi ers, one for action/concept relations and the other for concept/concept relations. We used a simple LR Logistic Regression model10 for both tasks with the same set of features, detailed below, but for learning two di erent classi ers. For de ning the feature set we performed an initial learning process from the training documents consisting on the following steps: • Collecting the whole set of correct entities. Computing the tf *idf and sorting the collection by descending tf *idf weight. • Decomposing the multi-word terms in the previous collection into atomic components. Computing also their tf *idf weight and sorting them accordingly. • Extracting the shapes of all the correct entities. We consider two types of shapes, long and short, the long shape simply maps the characters occurring in the term into a set of tags11 while the short shape groups together sequences of identical tags. For instance, for the term "DM-2" the long shape is "AA*0" and the short one is "A*0". We also obtained an histogram of the length of entities in tokens, in order to constraint the length of the generated candidates. 8The rst character of the label.
9Snomed-ct is organized as a tangled taxonomy with 19 top classes. We have used these top classes as tagset.
10using Scikit-learn package.
11"A" stands for upper case letter, "a" for lower case, "0" for number, " " for the space, and "*" for other characters. • Collecting all the labels occurring in the dependency trees of all the sentences in the training documents. • Collecting histograms of the POS appearing as initial, middle, and ending tokens in valid multi-word terms, and those appearing in single-word terms. From these collections those POS occurring under a threshold were removed. For instance, in the initial set (resulting in 26 POS) the most frequent POS was "NCMS000"12, that appears 90 times, in the middle set (18 POS) "DA0FS0"13 occurred 20 times, in the single set (65 POS) "NCFS000" occurred 316 times, and in the ending set (10 POS), "NCFS000"14 occurred 77 times.
It is worth noting that all these collections have been extracted independently for the two settings, Concept-Concept and ActionConcept. Once obtained these collections we performed a feature selection process using the development corpus, looking for di erent set of features (which we name "con gurations"). The most reliable con guration was the following: • A vector of the 1,000 most relevant lemmas (using tf *idf weigth) for both entities in the relation15. • A vector of the 200 most relevant atomic words (using tf *idf weigth) for both entities in the relation. • A vector of the labels of the path between the two entities (is existing) in the dependency tree of the involved sentence. 32 labels occurred in the training set in the Concept-Concept case. • A vector of the long word shapes for both entities. 51 di erent long word shapes were detected in the training set in the Concept-Concept case. • A vector of the short word shapes for both entities. 13 di erent short word shapes were detected in the training set in the Concept-Concept case. 12Common noun, masculine, singular 13Determiner article, feminine, singular 14Common noun, feminine, singular 15All the feature vectors are used as Boolean indicators. • The distance in tokens between the two entities. • The distance in characters between the two entities. • The length in tokens of both entities up to a maximum of 5 tokens. • Whether the entities are simple or complex.
As samples for learning and testing we consider as positive examples all the pairs of entities from subtask B occurring in the same sentence and the text between them. For the rst classi er one of the entities has to be an action and the other a concept, for the second one both entities have to be concepts. For negative examples we used pairs of entities so that only one element of the pair occurred as an entity in subtask B and the other should satisfy the POS constraints obtained above.
Besides these two classi ers we have
applied two other simpler lexical relation
extractors for some speci c relations:
• We looked in each sentence for the
cooccurrence of two entities being one of
them an acronym and the other its long
form. If it was the case the two
entities are tagged as related by a same as
relation,
In order to adjust the behaviour of our system to the speci cities of this competition we use the package develop also provided by the organization. In this context we adjust some parameters to the characteristics of the text to be processed. In particular, for subtasks A and B, we reviewed the set of domain borders already de ned adding three new ones.
For the task C a summary of the features used by the Concept-Concept logistic classier is shown in Table 1. Each candidate is represented as a vector of oat in a 2,578 dimensional space. Note that some of the features, as the dependency labels or the distances are applied to the pair, while others to both entities in the pair.
In our setting we generated 1,700 examples, from which 994 positives. Figures for the Action-Concept classi er are similar.
We also performed some experimentation in obtaining some is-a relation for multiword terms. Consider for example the TC s ndrome de Marfan, if the nucleus of this term (s ndrome) has been validated for YATE it is reasonable to consider that the following semantic relation exists: s ndrome de Marfan →is-a →s ndrome. This procedure reach a precision of 66 %. 4
The evaluation results obtained by the above described system and delivered to the organization for the evaluation are quanti ed in Table 2. This table shows both of our results as well as the baseline and best scores16 17. As can be observed in Table 2 our results for sub-tasks A and B are at least acceptable as they are above the baseline. Overall we were ranged on the third position, with a global score of 0.446 (0.464 and 0.461 for the rst and second teams). Unfortunately such results for sub-task C in two of the scenarios are bellow such baseline and this fact does not 16Note that the best score for each scenario correspond to the team with the best global result for each scenario. It means that it considers all subtasks for such scenario. This explains, for instance, that for the task C in scenario 1 the best scores are set to zero.
17Detailed results may be checked at the web site of the TASS-2018 competition. See footnote 1. satisfy our expectation. For such reason we decided to revise in depth our procedures and algorithm for such task and we found a malfunction for this task. After solving such bug and performing some minor improvement for sub-tasks A and B, we run again all our system and evaluate the new results but using in this case the script provided by the Organization. The results obtained after correcting our procedures are shown in Table 3. Such table shows a clear improvement in the results obtained for Task C. 5
In examining the text to be analysed we found some sentences whose inclusion in a health related corpus is not clear as they seems to be quite out of the domain. Example 1 shows a clear example of this kind of sentences.
(1) El CO se encuentra en el humo de la combustion, como lo es el expulsado por automoviles y camiones, candelabros, estufas, fogones de gas y sistemas de calefaccion.
In this example, the annotator tags Concepts like "CO", "humo" and "fogones" among others. Also it was tagged an is-a relation involving "humo de la combustion" and "humo". It is not clear the reason that such Concepts and Relations are considered relevant in a Spanish health document. The consequence is that our term extractor identify such units but does not validated them as relevant in the domain. As a matter of fact, most of the evaluated as missing have similar characteristics (such as: "proveedor", "insecticidas", "pintura", ...).
Training phase has been completed using only those les provided by the organization. As YATE only obtain nominal terms. We rely in the training phase for obtaining Action terms.
In order to show the behaviour of the two main parts of system (YATE and semantic relationship detector) we present some additional information about such modules.
Table 4 shows the amount of di erent term candidates that has been validated and discarded for the text provided for scenario 1. It also shows the details for each term candidate analyser. It should be noted that a single candidate may be successfully evaluated by more than one analyser (ex. "botulismo", "diureticos" and "psoriasis" among others).
Classi er LR action-concept LR action-concept LR concept-concept LR concept-concept LR concept-concept LR concept-concept lexical classi er lexical classi er Total LR Total Lexical
Label subject target same-as is-a part-of property-of same-as is-a future work: • Improve YATE accuracy when dealing with Actions. • Improve the integration of SNOMED
CT in YATE architecture • Analyse why there is no clear improvement when moving from scenario 1 to 3. • Analyse the contribution of the di erent features to the tasks A, B, and C. • Analyse the poor recall of the two LR classi ers for task C • In the case of task C we have used two linear multi-class classi ers LR. The set of features used was rather big and as the number of training samples was small the learning process was prone to overtting. We plan to test other non linear classi ers and to reduce the number of features, specially the lexical ones, lemmas and atomic forms, probably using embeddings learned on the medical domain. • Analyse the performance of the two LR classi ers separately. Only the overall results have been considered. The fact that basically the same feature set was used for learning the two classi ers was not the better solution. • Taking into account that the number of classes in task C is rather small, moving from multi-class to binary classi cation, i.e learning one classi er per class, should be considered.
This work was partially supported by the projects TERMMED (FFI2017-88100P, MINECO) and GRAPHMED (TIN201677820-C3-3R).