CUNI at the CLEF eHealth 2015 Task 2

Abstract

We present our participation as the team of the Charles University in Prague at the CLEF eHealth 2015 Task 2. We investigate performance of di erent retrieval models, linear interpolation of multiple models, and our own implementation of blind relevance feedback for query expansion. We employ MetaMap as an external resource for annotating the collection and the queries, then conduct retrieval at concept level rather than word level. We use MetaMap for query expansion. We also participate in the multilingual task where queries were given in several languages. We use Khresmoi medical translation system to translate the queries from Czech, French, and German into English. For the other languages we use translation by Google Translate and Bing Translator.

Keywords

• multilingual information retrieval
• language models
• UMLS
• blind relevance feedback
• linear interpolation

Introduction

Can we use the current web search engines to look for medical information? Authors in [ 17 ] showed that when users pose queries describing speci c symptoms or general health information, the current search engines can not e ectively retrieve relevant documents. This can lead to dangerous consequences if users try to apply the results they obtain for self-treatment. The biggest challenge in medical information retrieval is that users do not have enough medical knowledge so they cannot choose the correct medical terms which described their information needs. This often leads to "circumlocution" when the query contains more and vague words instead of less but speci c medical terms. Modern information retrieval systems have started to move in the direction of concepts rather than terms which helps to solve the "circumlocution" problem [ 14 ].

Task description

The goal of CLEF eHealth 2015 Task 2 [ 6, 10 ] is to design an IR system which returns a ranked list of medical documents (English web pages) from the provided test collection as a response to patients' queries. The task is de ned as a standard TREC-style text IR task1.

1 http://trec.nist.gov/

Data

Document Collection

The collection for Task 2 contains about one million documents provided by the Khresmoi project2. It contains automatically crawled web pages from popular medical websites. Non-HTML documents (e.g., pdf, rtf, ppt, and doc) which were found in the collection were excluded. The HTML documents were cleaned using the simple HTML-Strip3 Perl module. Other more advanced tools and statistical approaches for cleaning HTML documents did not bring any improvement in our previous experiments [ 12 ]. After cleaning the documents, we used MetaMap [ 1 ] to annotate the data with concept identi ers from the UMLS Metathesaurus [ 15, 2 ] version 2014AA. The UMLS Metathesaurus is a large vocabulary database containing information about biomedical and health-related concepts, their names and relationship between them. Terms are linked to others by various relationship such as synonymy, hypernymy, hyponymy, lexical variations, and many others. The Metathesaurus is organized by concept, which symbolize a semantic concept or a meaning. Each concept or meaning in the Metathesaurus has a unique and permanent concept identi er (CUI). We utilize MetaMap's highly con gurable options in our annotation process. We use the -I option so that the concept IDs are shown, and -y option to enable word sense disambiguation. The text is broken down into the components that include sentences, phrases, lexical elements and tokens. The disambiguation module then process the variants and output a nal mapping. We put this concept annotations into an additional XML eld in the document and query les. An example of cleaned and annotated document is given in Figure 1.

2 http://khresmoi.eu/ 3 http://search.cpan.org/dist/HTML-Strip/Strip.pm

We use the test queries from the CLEF eHealth 2014 Task 3 [ 5 ] to train our system. The experiments are evaluated using the newly created CLEF eHealth 2015 Task 2 test queries. We have annotated both sets of queries by MetaMap the same way as the documents. Some basic statistics associated with the query sets are shown in Table 1.

System description

Retrieval model

We use Terrier [ 9 ] to index the collection and to conduct the retrieval, we examine several weighting models and based on comparing P@10 performance for those models, we choose the following: { Bayesian smoothing with Dirichlet prior weighting model (DIR) This retrieval model is based on language modelling. The documents are scored by calculating the product of each term's probability in the query using language model for that document. Term probabilities in a document are estimated by maximum likelihood estimation. This might cause zero probabilities when a query term does not appear in the document. To avoid this problem the estimated probability distribution can be smoothed by various methods [ 16 ]. This retrieval model employs Bayesian smoothing with Dirichlet prior which uses di erent amount of smoothing based on the length of the document, for longer documents the smoothing will be less. The smoothing parameter is set by default to 2500. For more details and comparison with another smoothing methods see [ 13 ]. { Per- eld normalisation weighting model (PL2F) This model extends Poisson model with Laplace after-e ect and normalisation 2 (PL2) model. PL2 is based on Divergence from Randomness (DFR) document weighting models. The basic idea behind the DFR models is that the term frequency of a term in a document carries more information when it divergences from its distribution in the collection. In PL2F, each term in the document is assigned to one eld and the frequency of that term is normalised and weighted independently of other elds [ 8 ]. { LGD weighting model In this model, DFR approach is used together with log-logistic distribution, see [ 3 ]. 4.2

Query Expansion using the UMLS Metathesaurus

In UMLS, a concept can be represented by many di erent names. In other words, a concept structure represents synonymy relationships, i.e. terms assigned to the same concept are synonymous. For example, concept C0010054 corresponds to the term coronary arteriosclerosis. This term is synonymous with the term coronary heart disease, which also has the same CUI in the Metathesaurus. There are 113 other terms that are assigned with this CUI. In our experiment, we utilize this relationship to pick the candidate for query expansion.

As mentioned in the previous section, the queries are annotated with concept identi ers. For every concept in a query, we generate a list of synonymous terms under that concept. We keep the original query terms, and the added candidate terms that are not yet in the query terms. We do not add all the synonymous terms, but only up to ve words that have the highest inverse document-frequency in the collection. In one of our runs, we further lter this words by doing an initial document retrieval using the original query, and then only use the synonyms that also appeared in top n relevant documents. We utilize Terrier's query language4 to experiment on eld weighting with the query. Terrier query language has several operators with di erent functions. In our experiment, we used the ^operator that is used to assign weights to words. term1^2 means that the weight of term1 is multiplied by 2. There are three kinds of elds that we utilize: the original query terms, the expanded query terms, and the concept identi ers from the original terms. We assign di erent weights to di erent elds. We tune our system using the CLEF 2014 test set to get the best weights con guration. Query language is only available in Terrier for single line query format, so we have to rst convert the provided TREC format to single line format. Figure 2 shows some samples of weighted single line queries, where original terms, expansion terms, and concept IDs are given weight 4:1, 0:6, and 0:1 respectively. 4.3

Blind relevance feedback

Blind relevance feedback (BRF) automates user's part of relevance feedback by expanding user's query using extra information from the collection [ 4 ]. In BRF, an initial retrieval steps is performed to nd a set of n highly-ranked documents. These documents are assumed to be relevant and a set of m terms from these documents is extracted and added to the original query and the nal retrieval step is conducted using the expanded query. In our experiments, the term selection is based on IDF scores extracted from the entire collection. We tune both n and m parameters using the CLEF 2014 test set. We found that adding just one term from top 25 documents gives the highest P@10. 4.4

Linear interpolation

In some of our experiments, we perform linear interpolation of scores from multiple retrieval models.The following equation generates a new (interpolated) score 4 http://terrier/org/docs/v4.0/querylanguage.html We tune lambda value using the CLEF 2014 test set to get highest P @10. Figure 3 shows the curves for the CLEF 2014 test set and the CLEF 2015 test set 4.5

Term weighting

In the multilingual task, we use our term weighting algorithm for languages Czech, French and German to expand queries. First we use Khresmoi translation system to translate the queries into English and return the n-best-list translations. These translations form a translations pool. Each term in the translations pool is assigned a weight, which is the score of its translation hypothesis given by the translation model, say T M (term). Then, we use the BRF algorithm as described in Section 4.3 to retrieve the n highly-ranked documents, where the queries are taken only from the best translation. For each term in the translations pool, we calculate the IDF score in the entire collection and the term frequency (TF) in the translations pool. We normalise all of these scores and then each term is weighted as follows:

Score(term) = T M (term) 1 T F (term) 2 IDF (term)(10 0 0.2 0.4 0.6 0.8

1

Lambda Where lambda values sum up to 10. After scoring terms in the translations pool, we sort them descending by their score and add top m terms into original query. Lambda values, n and m are trained using the CLEF 2014 test set.

Experiments and results

Monolingual Task

We submitted to this task 10 runs, summarised in Table 2, as follows: { Run1, Run2, and Run3 employ Terrier's implementation of DIR, LGD, and PL2F retrieval models, respectively. Indexing and retrieval are conducted at term level. { Run4 is a linear combination of Run1, Run2, and Run3 with parameters set to 0.56, 0.32, and 0.12, respectively. The values were tuned on the CLEF eHealth 2014 test queries. { Run5 employs query expansion based on the UMLS Metathesaurus. For each UMLS concept ID in each query, we retrieved all entries with that concept ID and then expanded that query with 5 words that have the highest IDF in the collection, excluding words that already occur in the original query. We used original terms, concept id, and expansion terms for the retrieval process using Terrier's implementation of PL2F. Each eld is weighted di erently, see Section 4.2. { Run6 uses the same settings at Run5 but employs the LGD retrieval model. { Run7 interpolates Run5 and Run6 with parameters 0.71 and 0.29, respectively. { Run8 interpolates Run1 with a system that only uses concept IDs for retrieval (using PL2F model), with parameters 0.98 and 0.01, respectively. query ID original query expanded term clef2015.test.1 many red marks on legs after trav- striaestretch

eling from us clef2015.test.2 lump with blood spots on nose mixx clef2015.test.3 dry red and scaly feet in children scaleness clef2015.test.4 itchy lumps skin healthdental clef2015.test.5 whistling noise and cough during ringining sleeping + children { Run9 is similar to Run5, but the expansion terms are further ltered by relevance feedback, i.e. we only use the terms that also appeared in the initial retrieval. { Run10 employs our own implementation of blind relevance feedback. We do initial retrieval using Run1, then from top 25 ranked documents. We add one term with the highest IDF in the collection into original query, then we do the retrieval again using Run1, see Section 4.3.

Table 4 shows our system performance on the CLEF 2014 test set. The di erence between numbers in italics and bold is not statistically signi cant using the Wilcoxon test [ 7 ]. Run4 which is an interpolation between the rst 3 runs gives the best P@10. Run8 also brings some improvement to the baseline system. Run5, which uses the same model with Run3, brings a slight improvement with the query expansion. However, Run6 decreases slightly in P@10 compared to unexpanded Run2. Run7 brings some improvement over the two other runs that it interpolated. In case of Run9, combining our query expansion with relevance feedback decreases the performance compared to Run5.

The results of our submitted runs are shown in Table 5. In terms of P@10 metric, Run1 and Run2 have the same P@10, but Run1 outperforms Run2 in terms of MAP, NDCG@10 and the number of relevant retrieved documents. We have improvement in Run4, which linearly interpolates Run1, Run2 and Run3. As in the training set, Run5 improves the P@10 when compared to Run3, and Run6 does not bring any improvement over Run2. Run7, which is the best performing run, brings improvement over Run5 and Run6. Run3 has 3 unjudged documents in the rst 10 ranked documents among 66 queries, so the shown metrics may not be accurate. BRF in Run10 does not bring overall improvement in P@10, but it does in some queries. BRF still does not guarantee to choose the best term to expand the query with (see Table 3). 5.2

Multilingual Task

In this task, we are given parallel queries in Arabic, Czech, Farsi, French, German, Italian and Portuguese and the goal is to design a retrieval system to nd relevant documents to these queries from the English collection. For queries in Czech, French and German, we submitted 10 runs as follows (see also Table 7): { Run1, Run2 and Run3 runs, we translate the queries using Khresmoi [ 11 ] then use Terrier's implementation of DIR, PL2F and LGD retrieval models respectively. { Run4 interpolates Run1, Run2, and Run3 with parameters (0.57, 0.40, 0.03) respectively, tuned on the CLEF eHealth 2014 test set. { Run5 uses Khresmoi to translate the queries into English, queries are expanded using term weighting algorithm as described in Section 4.5. After model tuning, we use as parameters 1 = 4, 2 = 2, n = 1, and m = 30. { In Run6, we also use Khresmoi, then we use BRF to expand queries by adding one term from the top 25 documents, which has the highest IDF in the collection into original query. Both the initial and nal retrieval steps use the DIR model. { Run7, In this run we translate the queries using Google Translate, and use

Terrier's implementation of DIR model { Run8 is similar to Run7, but queries are translated using Bing translator. { Run9, we use Google Translate to translate the queries into English then use

Terrier's implementation of PL2F. { Run10 interpolates Run7 and Run8, for Czech we use the parameters 0.92 and 0.08 respectively, for French 0.90, and 0.10 and for German 0.7, and 0.3.

For AR, FA, IT and PT, we submitted the following runs (see also Table 6): { Run1, Run2 and Run3, we translate the queries using Google Translate and then use Terrier's implementation of DIR, PL2F, and LGD respectively. { Run4 interpolates Run1, Run2 and Run3 with the parameters 0.56, 0.32, and 0.12 respectively.

Highlights of the red spots on the nose bulge with blood stains on his nose tumor with bloody spots on the nose clot with blood stains on his nose The mass highlighted with red spots on nose bump with blood stains on the nose Tumor with bloody points on the nose lump with bloodstains on the nose { Run5, Run6 and Run7, queries are translated using Bing translator, then retrieval is conducted using PL2F, DIR, and LGD respectively. { Run8 interpolates Run5, Run6 and Run7 with the parameters 0.57, 0.40, and 0.03. { Run9 uses Google Translate and BRF to expand queries using 25 document and 1 term and DIR model for both initial and nal retrieval. { Run10 interpolates Run1 and Run6 with the parameters 0.84 and 0.16.

The results for multilingual submission are shown in Table 9. The table shows P@10 values for all languages and the last column shows the result of our monolingual task for comparison. Linear interpolation between runs use Google Translate and Bing translator together with DIR model improved the results for Farsi, French, German and Italian. The Baseline in Italian is identical to monolingual one, other Italian runs are very close to monolingual runs and sometimes higher, anyway we have many unjudged documents in our Italian submission so results may di er after full assessment. Example of how Google and Bing translate one query in di erent languages is shown in Table 8. In this paper, we have described our participation in the CLEF eHealth 2015 Task 2. We used the Terrier IR platform to index the collection and conduct retrieval using di erent retrieval models and our own implementation of blind relevance feedback. We used MetaMap to annotate the documents in the collection and the queries with concepts and then built systems on concept levels, which improved the performance measured by P@10. Linear interpolation of runs conducted using di erent approaches, it improved P@10 when interpolating models use di erent retrieval models. Although query expansions did not bring improvement over the baseline, we got improvement for some individual queries. In some cases it also improved the performance compared to a system with the same retrieval model that only used the original query terms. We also submitted runs to multilingual task. We translated queries in the given languages into English and performed several experiments. The most promising results were obtained by linear interpolation of runs using di erent translation system for languages Farsi, French, German and Italian which use DIR model. We also used information from translation variants provided by the Khresmoi translation system (n-best lists). This did not bring overall improvement but it did for some individual queries. We believe that more thorough investigation should be done on terms selection algorithms to re ne query expansion based approaches, so it will lead to better performance.

Acknowledgments

This research was supported by the Charles University Grant Agency (grant n. 920913), Czech Science Foundation (grant n. P103/12/G084), and the EU H2020 project KConnect (contract n. 644753).

References

1. Aronson , A.R.: E ective mapping of biomedical text to the umls metathesaurus: the metamap program . Proc AMIA Symp pp. 17 { 21 ( 2001 )

2. Bodenreider , O. : The uni ed medical language system (umls): Integrating biomedical terminology ( 2004 )

3. Clinchant , S. , Gaussier , E.: Bridging language modeling and divergence from randomness models: A log-logistic model for ir . In: Azzopardi, L. , Kazai , G. , Robertson , S. , Rger , S. , Shokouhi , M. , Song , D. , Yilmaz , E. (eds.) Advances in Information Retrieval Theory, Lecture Notes in Computer Science , vol. 5766 , pp. 54 { 65 . Springer Berlin Heidelberg ( 2009 )

4. Efthimiadis , E.N. : Query expansion . Annual review of information science and technology 31 , 121 { 187 ( 1996 )

5. Goeuriot , L. , Kelly , L. , Li , W. , Palotti , J. , Pecina , P. , Zuccon , G. , Hanbury , A. , Jones , G. , Mueller , H.: Share/clef ehealth evaluation lab 2014, task 3: User-centred health information retrieval . In: Proceedings of CLEF 2014 ( 2014 )

6. Goeuriot , L. , Kelly , L. , Suominen , H. , Hanlen , L. , Nvol , A. , Grouin , C. , Palotti , J. , Zuccon , G. : Overview of the clef ehealth evaluation lab 2015 . In: CLEF 2015 - 6th Conference and Labs of the Evaluation Forum. Lecture Notes in Computer Science (LNCS) , Springer ( September 2015 )

7. Hull , D. : Using statistical testing in the evaluation of retrieval experiments . In: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval . pp. 329 { 338 . SIGIR '93, ACM , New York, NY, USA ( 1993 )

8. Macdonald , C. , Plachouras , V. , He , B. , Lioma , C. , Ounis , I.: University of glasgow at webclef 2005: Experiments in per- eld normalisation and language speci c stemming . In: Peters, C. , Gey , F. , Gonzalo , J. , Mller , H. , Jones , G. , Kluck , M. , Magnini , B. , de Rijke , M. (eds.) Accessing Multilingual Information Repositories, Lecture Notes in Computer Science , vol. 4022 , pp. 898 { 907 . Springer Berlin Heidelberg ( 2006 )

9. Ounis , I. , Amati , G. , Plachouras , V. , He , B. , Macdonald , C. , Lioma , C. : Terrier: A high performance and scalable information retrieval platform . In: Proceedings of ACM SIGIR'06 Workshop on Open Source Information Retrieval (OSIR 2006 ) ( 2006 )

10. Palotti , J. , Zuccon , G. , Goeuriot , L. , Kelly , L. , Hanbury , A. , Jones , G.J. , Lupu , M. , Pecina , P. : Clef ehealth evaluation lab 2015, task 2: Retrieving information about medical symptoms . In: CLEF 2015 Online Working Notes. CEUR-WS ( 2015 )

11. Pecina , P. , Dusek , O. , Goeuriot , L. , Hajic , J. , Hlavacova , J. , Jones , G. , Kelly , L. , Leveling , J. , Marecek , D. , Novak , M. , Popel , M. , Rosa , R. , Tamchyna , A. , Uresova , Z. : Adaptation of machine translation for multilingual information retrieval in the medical domain . Arti cial Intelligence in Medicine ( 2014 )

12. Saleh , S. , Pecina , P. : Cuni at the share/clef ehealth evaluation lab 2014 . Proceedings of the ShARe/CLEF eHealth Evaluation Lab 1180 , 226 { 235 ( 2014 )

13. Smucker , M.D. , Allan , J.: An investigation of dirichlet prior smoothings performance advantage . Tech. rep., Tech. Rep. IR-445 , University of Massachusetts ( 2005 )

14. Stanton , I. , Ieong , S. , Mishra , N.: Circumlocution in diagnostic medical queries . In: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval . pp. 133 { 142 . ACM ( 2014 )

15. U.S. National Library of Medicine: UMLS reference manual (2009), metathesaurus . Bethesda, MD , USA

16. Zhai , C. , La erty , J.: A study of smoothing methods for language models applied to information retrieval . ACM Transactions on Information Systems (TOIS) 22(2) , 179 { 214 ( 2004 )

17. Zuccon , G. , Koopman , B. , Palotti , J.: Diagnose this if you can: On the e ectiveness of search engines in nding medical self-diagnosis information . In: Advances in Information Retrieval (ECIR 2015 ). pp. 562 { 567 . Springer ( 2015 )