In this paper, we describe our participation in the mesinesp Task of the BioASQ biomedical semantic indexing challenge. The participating system follows an approach based solely on conventional information retrieval tools. We have evaluated various alternatives for extracting index terms from IBECS/LILACS documents in order to be stored in an Apache Lucene index. Those indexed representations are queried using the contents of the article to be annotated and a ranked list of candidate labels is created from the retrieved documents. We also have evaluated a sort of limited Label Powerset approach which creates meta-labels joining pairs of DeCS labels with high co-occurrence scores, and an alternative method based on label pro le matching. Results obtained in o cial runs seem to con rm the suitability of this approach for languages like Spanish.
This article describes the joint participation of the CoLe group 3 from the
University of Vigo and the LYS group 4 from the University of A Corun~a in the
Spanish biomedical semantic indexing task of the 2020 BioASQ challenge [
In our participation we have followed a similarity based strategy, where the
nal list of DeCS descriptors assigned to a given article is created from the set
of most similar IBECS/LILACS articles stored in a textual index created from
the training dataset. This neighbor based strategy was explored in previous
participations in BioASQ challenge [
In the case of text categorization for Spanish written documents in mesinesp Task we have employed this similarity based method using several index term extraction approaches in order to evaluate the e ects of document representation in the overal quality of the predicted labels. We have also tried improving the categorization performance using a sort of limited Label Powerset multi-label categorization approach, where meta-labels created by joining pairs of labels with high co-occurrence scores replace the original document labels. Additionally a similarity based method using synthetic documents to represent "label pro les" was evaluated and integrated into our o cial mesinesp8 runs.
The rest of the paper is organized as follows. Section 2 describes the main ideas behind the proposed similarity based approach for mesinesp8 annotations and also describes the text processing being applied. Section 3 brie y details the use of synthetic meta-labels in our Label Powerset approach and how we use the "label pro les" to annotate mesinesp articles. Finally, section 4 discusses our o cial runs in the BioASQ challenge and details the most relevant conclusions of our participation. 2
Approaches based on k nearest neighbors (k-NN) have been widely used in the
context of large scale multi-label categorization, being employed for MEDLINE
documents [
In this way, our annotation scheme starts by indexing the contents of the mesinesp training articles. For each new article to annotate that index is queried using its contents as query terms. The list of similar articles returned by the indexing engine and their corresponding similarity measures are exploited to determine the following results: { predicted number of descriptors to be assigned 5 Source code available at https://github.com/fribadas/mesinesp8. 6 https://lucene.apache.org/ { ranked list of predicted DeCS descriptors
The rst aspect conforms a regression problem, which aims to predict the number of descriptors to be included in the nal list, depending on the number of descriptors assigned to the most similar articles identi ed by the indexing engine and on their respective similarity scores. The other task is a multi-label classi cation problem, which aims to predict a descriptors list based on the descriptors manually assigned to the most similar mesinesp articles. In both cases, regression and multi-label classi cation, similarity scores calculated by the indexing engine are exploited. These scores are computed during the query processing phase. Query terms employed to retrieve the similar articles are extracted from the original article contents and linked using a global OR operator to conform the nal query sent to the indexing engine.
In our case, the scores provided by the indexing engine are similarity measures resulting from the engine internal computations and the weighting scheme being employed, which do not have an uniform and predictable upper bound. In order for these similarity scores to behave like a real distance metric, we have applied the following normalization procedure: 1. Articles to be annotated are preprocessed in the same way than the training articles indexed by the Lucene engine. 2. In classi cation time, all of the relevant index terms from the article being annotated are joined by an OR operator to create the search query. 3. In the ranking of similar articles returned by the indexing engine the top result will be the same article used to query the index, this result is discarded but its score value (scoremax) is recorded for future normalization. 4. For each element on the remaining articles set, the number of descriptors is recorded and it is also recorded the list of assigned descriptors, linking to each of them an estimated distance to the article being annotated, equals to 1 scsocroermeax , which will be employed in the weighted voting scheme during k-NN classi cation.
With this information the number of descriptors to be assigned to the article being annotated is predicted using a weighted average scheme, where the weight of each similar article is the inverse of the square of the estimated distance to the article being annotated, that is, 1
(1 scsocroermeax )2 .
To create the ranked list of descriptors a distance weighted voting scheme is employed, associating the same weight values (the inverse of squared estimated distances) to the respective similar articles. Since this is actually a multi-label categorization task, there are as many voting tasks as candidate descriptors were extracted from the articles retrieved by the indexing engine. For each candidate label, positive votes come from similar articles annotated with it and negative votes come from articles not including it. In our preliminary experiments we have tested several approaches to extract the set of index terms to represent mesinesp articles in the indexing process.
Regarding article representation we have evaluated four index term extraction approaches. In these experiments and also in the o cial mesinesp8 runs we have worked only with the Pre-processed Training set provided by BioASQ organizers. We have discards the dataset of PubMed abstracts translated into Spanish provided by mesinesp organizers due to format issues regarding part of the available translated abstracts. The nal training dataset comprised 318,658 records with at least one DeCS code. Index terms which occurred in 5 or less articles were discarded and terms which were present in more than 50 % of training documents were also removed.
Our aim with these experiments was to determine whether linguistic motivated index term extraction could help to improve annotation performance in the k-NN based method we have described. We employed the following methods: Stemming based representation. This was the simplest approach which employs stop-word removal, using a standard stop-word list for Spanish, and the default Spanish stemmer from the Snowball project7. 7 http://snowball.tartarus.org Morphosyntactic based representation. In order to deal with the e ects of morphosyntactic variation in Spanish we have employed a lemmatizer to identify lexical roots instead of using word stems and we also replaced stop-word removal with a content-word selection procedure based on partof-speech (PoS) tags.
We have delegated the linguistic processing tasks to the tools provided by the spaCy Natural Language Processing (NLP) toolkit 8. This toolkit o ers a set of state-of-the-art components written in the Python programming language, together with a collection of pretrained models, ready to be used in typical natural language processing tasks like dependency parsing, named entity recognition, PoS tagging and morphological analysis.
In our case we have employed the PoS tagging and lemmatization information provided by spaCy to tokenize and assign PoS tags to the mesinesp abstract contents. We employed the standard Spanish models available on spaCy without using any speci c data for biomedical related contents.
In order to lter the content-words from the processed mesinesp abstracts, we have applied a simple selection criteria based on the employment of the PoS that are considered to carry the sentence meaning. Only tokens tagged as a noun, verb, adjective, adverb or as unknown words are taken into account to constitute the nal article representation.
After PoS ltering, the lemmas (canonical forms of words) corresponding to surviving tokens are employed to normalize the considered word forms in a slightly more consistent way than simple stemming.
Nominal phrases based representation. In order to evaluate the contribution of more powerful NLP techniques, we have employed a surface parsing approach to identify syntactic motivated nominal phrases from which meaningful multi-word index terms could be extracted.
Noun Phrase (NP) chunks identi ed by spaCy are selected and the
lemmas of the constituent tokens are joined together to create a multi-word
index term. In the current version of the system no other syntactical units
of interests like prepositional phrases or verbal phrases are considered, since
nominal phrases use to carry most of the text semantic content.
Dependencies based representation. We have also employed as index terms
triples of dependence-head-modi er extracted by the dependency parser
provided by spaCy. A dependency parser analyzes the grammatical structure of
a sentence, establishing relationships between head words and words which
modify those heads. In our case spaCy provides a dependency parsing model
for Spanish that identify syntactic dependency labels following the Universal
Dependencies(UD) [
Dependence relationships encode information that provides an approximation to high level semantic relationships, giving information regarding the agent of an action (with a nsubj relationship between the main verb and the root of the nominal phrase acting as subject) or the object of that action (by means of a obj relationship), among others. In our system, the complex 8 Available at https://spacy.io/ index terms were extracted from the following UD relationships 9: acl, advcl, advmod, amod, ccomp, compound, conj, csuj, dep, at, iobj, nmod , nsubj, obj, xcomp, dobj and pobj.
UIMA Concept Mapper representation. In addition to those representations, we also have employed the Concept Mapper 10 module from the UIMA (Unstructured Information Management Architecture) framework. This component employs a dictionary with all of the DeCS labels and their corresponding synonyms and searches for exact matches of those DeCS labels into the abstract text. In our case we have added to the document representation as index term each one of those matches in order to maintain its absolute occurrence frequency.
In order to illustrate the index term extraction procedure, gure 1 shows an example of a mesinesp record with the set of representations extracted from the textual contents of its abstract.
Table 1 summarizes the results obtained in our preliminary tests regarding document representation, using as measures MicroPrecision, MicroRecall, MicroF, Example based Precison, Recall and F measure, and Accuracy. The test dataset was created after processing 750 manually indexed records from the Core-descriptors development set provided by BioASQ organizers. We have evaluated the performance of the described index term generation methods (stems, lemmas, nps, deps and using all of them together) for increasing values of k, the number of similar articles to be used (1) in the estimation of the number descriptors to be assigned and (2) in the voting procedure that will construct the nal list of descriptors to attach to a given article.
As can be seen in table 1, the best results were obtained with fairly high values for k ( 30). Regarding the index term representations, the runs which employed index term extracted by means of stemming (both stemming alone and stems mixed with the other index terms) provided the best performance. The representations using complex index terms extracted from noun phrase chunks and dependencies triples o ered poor performance, maybe because of very infrequent index terms that can have the undesired e ect of boosting internal scores in schemes where inverse document frequencies are taken into account. 3
In this section we describe two approaches that try to improve labeling performance taking advantage of the information inherent to DeCS labels. Even DeCS is a fairly large concept hierarchy we have tested the suitability of extending the label space using an approach inspired by the Label Powerset(LP) method employed in multi-label categorization. In this case we create a set of "meta-labels" that replace pairs of DeCS labels which tend to appear together in 9 Detailed list of UD relationships available at https://universaldependencies.org/ u/dep/ 10 http://uima.apache.org/d/uima-addons-current/ConceptMapper/RELEASE NOTES.html id biblio-1000005 db LILACS journal Oncol. (Guayaquil) title Manejo de Tumores de Mediastino, Serie de Casos abstract Introduccion: A pesar del dif cil acceso anatomico para los tumores de mediastino, la reseccion quirurgica sigue siendo el mejor enfoque diagnostico y terapeutico. En la presente serie de casos presentamos la experiencia de un centro oncologico en el abordaje de tumores del mediastino y sus resultados.
Metodos: En el departamento de Jefatura de Cirug a Oncologica del Instituto Oncologico nacional de Solca-Guayaquil, durante los meses de Enero del 2013 a Enero 2017 se realizo un estudio descriptivo, retrospectivo. Se analizaron todos los casos de pacientes derivado ... decs codes 9562,8650,21034,24375,21044,20174,14341,238,9062,21030,23039 stems training dataset. The other aspect regarding DeCS labels that we have explored is to exploit the idea of "label pro les". These pro les represent the concepts behind each DeCS label by means of a synthetic document that aggregates the contents of all of the abstracts annotated with a given label. 3.1
Label Powerset(LP) [
Our approach limits classical LP multi-label categorization to the cases where only combinations of highly correlated pairs of labels are taken into account. To select the pairs of labels to be joined we have computed the Normalized Pointwise Mutual Information (NPMI) between each pair of DeCS labels, li and lj , across rank rank the training dataset employing the following formula: Where P M I is the Pointwise Mutual Information computed by: N P M I (li; lj ) = P M I (lj ; lj ) = log(
P M I (li; lj ) log(P (li; lj ))
P (li; lj ) P (ii) P (lj ) ) And where P (li; li) is computed as jdocs. labeled with li and lj j and P (l) is computed jdocs. in training collectionj as
jdocs. labeled with lj jdocs. in training collectionj
.
The measure N P M I (li; lj ) normalizes the values of PMI in [ 1; 1], resulting in -1 for a pair of labels never occurring together, 0 for independence, and +1 for complete co-occurrence of labels li and lj .
In our experiments we have evaluated three thresholds (0.25, 0.50 and 0.75) to create new "meta-labels" joining pairs of labels whose NPMI scores are over them. Table 2 compares the most frequent codes in the original dataset and when "meta-labels" with NPMI scores above 0.25 replace the original codes.
Once these "meta-labels" are identi ed we create new training documents replacing in the set of DeCS labels associated to each record the two original labels with the corresponding new "meta-label". To annotate the test articles we apply the k-NN procedure described in previous sections over the new training documents where "meta-labels" were placed. 3.2
Another approach that we have tested in order to capture the semantics of the DeCS labels is the use of "label pro les" that try to represent the contents associated to each DeCS label and match incoming test documents to those pro les.
To create those DeCS label pro les we have followed a very simple approach that is easily integrated into our Lucene backed k-NN multi-label categorization scheme. 1. For each DeCS label we collect the index terms extracted from the abstracts of documents annotated with that label. 2. With those lists of index terms we create a synthetic Lucene document concatenating the terms to create a big document that holds the representation of the "label pro le" for the corresponding label. 3. All those synthetic documents representing "label pro les" for every DeCS label are indexed into a Lucene index.
To annotate an incoming article abstract text is processed as described in precedent section to extract its index terms. With those index terms the Lucene index of "label pro les" is queried and the top most similar synthetic documents are recorded to annotate that article with their corresponding labels. The idea behind this approach is to improve the main k-NN annotation procedure, which follows a content-based method, with a complementary method focused on the labels and its semantic aspects. rank
MiF
EBP
EBR
EBF
MaP
MaR
MaF
MiP
MiR
Acc. Although we have tested several alternatives to try to improve the results obtained by the Lucene based k-NN method, only the most simple ones have been submitted to the o cial batches of BioASQ challenge.
In table 3 the o cial performance measures obtained by our runs in the mesinesp8 Task are shown. The o cial runs sent during our participation were created using the following con gurations. iria1. This run created the representation of mesinesp articles using all of the index term extraction methods described in section 2.1. During indexing and querying, terms appearing in 5 or less abstracts and terms used in more than 50% of total documents were discarded. The number of neighbors used by the k-NN classi er is 30 and the predicted number of descriptors to be returned was increased a 10% in order to ensure slightly better values in recall related measures. iria2. For this run the same setup as iria1 was employed, but instead of using the original train dataset this runs employed the limited Label Powerset approach and indexed a new training dataset annotated with "metalabels" created by joining pairs of DeCS labels with a NPMI scores above 0.25. iria3. This run was simply the intersection of the labels predicted by iria1 and iria2. iria4. This run created a set of "label pro les" over the train dataset employed in iria2, that is, documents annotated with "metalabels" created by joining pairs of labels with NPMI scores over 0.25. In this case the number of labels to predict was xed to 10 and the number of neighbors used by the k-NN classi er was 15. iria-mix. This run was based on run iria1, adding the predictions of iria4 and the exact matches provided by UIMA Concept Mapper.
Labels predicted by iria4 but discarded by iria1 were added to the nal list of candidate labels. The same procedure was applied to add the exact matches identi ed by means of Concept Mapper but not predicted by neither of iria1 and iria4.
The results of our participation in the mesinesp8 task of the BioASQ biomedical semantic indexing challenge were not far from the results of the most competitive teams, showing that similarity based methods can still be considered for large scale indexing tasks. As positive aspects of our participation, we have conrmed that k-NN methods backed by conventional textual indexers like Lucene are a viable alternative for this kind of large scale problems, with minimal computational requirements and fairly good results in the case of Spanish biomedical abstracts. We have also conducted a comprehensive evaluation of the performance of several alternatives to index term extraction, ranging from simple ones, based on stemming rules, to more complex ones were natural language processing was required.
The future lines of work are related with the improvement of natural language processing. In this participation we have employed general domain NLP models. Biomedical documents have many speci c characteristics that suggest that custom NLP models trained with text from this domain will help to improve the performance of our classi er. Likewise, the use of "meta-tags" in this work opens a future line of research on the exploitation of the semantics inherent to the co-occurrence of tags.
F.J. Ribadas-Pena and S. Cao have been supported by the Spanish Ministry of Economy, Industry and Competitiveness (MINECO) through the ANSWERASAP project (TIN2017-85160-C2-2-R), and by the Galician Regional Government (Xunta de Galicia) under project ED431D 2017/12.
E. Kuriyozov received funding from the ANSWER-ASAP project (TIN201785160-C2-1-R) from MINECO, and from Xunta de Galicia (ED431B 2017/01, ED431G 2019/01). He is also funded for his PhD by El-Yurt-Umidi Foundation under the Cabinet of Ministers of the Republic of Uzbekistan.