MayoNLPTeam at the 2016 CLEF eHealth Information Retrieval Task 1

Abstract

This paper presents the participation of MayoNLPTeam in the 2016 CLEF eHealth Information Retrieval Task (IR Task 1: ad-hoc search). We explored a Part-of-Speech (POS) based query term weighting approach which assigns di erent weights to the query terms according to their POS categories. The weights are learned by de ning an objective function based on the mean average precision. We applied the proposed approach with the optimal weights obtained from TREC 2011 and 2012 Medical Records Track into the Query Likelihood model (Run 2) and Markov Random Field (MRF) models (Run 3). The conventional Query Likelihood model was implemented as the baseline (Run 1).

Keywords

• information retrieval
• Part-of-Speech
• language model
• Markov Random Field model

-

The amount of health information on the web has increased tremendously during the last decades. People access these contents to nd information or answers regarding their health concerns. According to a late 2013 survey by the Pew Research Center's Internet and American Life Project, 85% of US adults use the Internet and 72% of them have looked for health information online [ 1 ]. However, it is di cult to nd the information related precisely to their concerns. To tackle this issue, the 2016 CLEF eHealth Information Retrieval (IR) Task 1 [ 3,13 ] focuses on the retrieval of health contents on the web for the health queries generated by exploring real consumer posts from health forums. The goal of this task is to explore possible IR systems that people could use to search for information or answers to their health questions instead of posting those questions on health forums and waiting for answers.

As a participant in IR Task 1, we introduce a Part-of-Speech (POS) based query term weighting approach. The POS property re ects whether the term is informative or not. Intuitively, a noun is more informative than a preposition, and in the medical domain a proper noun is more important than a noun to understand semantics of a query. Therefore, we hypothesize that leveraging POS information to weight the query terms would improve the performance of IR systems. The experiments on the Electric Health Records (EHRs) retrieval provided by the Text REtrieval Conference (TREC) 2011 and 2012 Medical Records tracks [ 10 ] have veri ed our hypothesis. We would like to examine whether this approach is viable for the internet health contents retrieval.

We submitted three o cial runs to CLEF eHealth. In Run 1, we utilized plain text of the test topics as the input queries and the Query Likelihood model with Dirichlet smoothing [11] as the retrieval model. This run was served as a baseline. In Run 2, we utilized the POS-based query term weighting method to improve the Query Likelihood model. Di erent from Run 2 where we assumed that the terms were independent, Run 3 applied Markov Random Field (MRF) model [ 5 ] and incorporated the POS-based query term weighting method.

The rest of this paper is organized as follows. Section 2 describes the dataset and queries of the 2016 CLEF eHealth IR Task. Section 3 presents the our framework in detail including the system details and how we trained the weights. The experiments and experimental results are shown in Section 4. Section 5 concludes our study. 2

Dataset and Queries of CLEF eHealth IR Task 1

The ClueWeb12 B13 dataset is used in the 2016 CLEF eHealth IR Task. This dataset is a small portion of the crawled web documents in ClueWeb12 dataset as part of the lemur project1. It contains over 52 million documents. Those web documents in this corpus include health and non-health contents. Each document includes a title eld in the \WARC- TREC-ID" eld of the document's WARC header and a title and a heading eld. The goal of this challenge is to nd those relevant health contents given an input query.

The queries provided by the task are extracted from the posts in the askDocs health web forum2. Therefore, this set of queries re ects the real information needs of health consumers. Each of six query creators with di erent medical expertise was given 50 initial posts from the forum to generate the queries with a total of 300 queries created. Each query has an id eld and a title eld. The id eld is used to distinguish the queries while the title eld represents the queries. Task 1 of the challenge requires us to treat each query individually and submit up to 3 ranked runs with up to 1000 documents per query. 3

Method

In this section, we present the IR system and the proposed approach, and then detail the submitted runs.

1 http://lemurproject.org/

2 https://www.reddit.com/r/AskDocs/

3.1

System Overview

In our previous submission of the 2013 ShARe/CLEF eHealth Evaluation Lab [12], we focused on semantiIcnstroadnudctiuontilized multiple external sources, such as Mayo Clinic clinical notes collection and the Uni ed Medical Language System Objectives the POS pwrhoicpheisrtkynotwon aassEsilegcntrodniic HeeraeltnhtRweceoirdg h(EtHsRt)osyqstuemer.y terms. (UMLS)¦,f Colriniqcaulerercyoredxspanadnhseioalnth.inTfohr misatyioenaarrewsetofreodcuinsdimgitoalrfeoromn, syntac¦ti Pcsroaponsde alePvOeSrawgeeighted MRF model for clinical information retrieval, POS-MRF, to improve clinical notes retrieval accurac organizersbfyoGroldtehnSeecttiaonsLkin.e TSeharceh Adlgaortitahms)e, tandaµnj+1d st0.3 andard indexes were available in the

=µj +yjdj. If j<τ,replace j by j+1,and repeat Step1. 0.2 Azure instOathnecrweis.e, Sif pj=τe, cgoi tocSatelply2., the preproces0.2s5ing included stopwords removal and

0.2 0.15 Krovetz Øs t2e: mLemtλk+i1n=gµτ.+1.ITf||λhk+e1−nλk||t<hε,ethetnitsltoep. and heading elds were indexed using Indri rOetpheearwtSistee,ple1t.µ1 =λk+1, let j=1,replace k byk+1, and 0.15 0.1

Experimental Results ¦  Data Ø  TREC 2011 & 2012 Medical

Records Track ¦  Measures Ø  Mean Average Precision

(MAP) 0.3 0.05 Ø  3: return λ’s.

0.1 0.05

Acknowledgements

This study is supported by grant 1R01LM011934-01A1 from the National Institute of Health.

The POS categories for queries were obtained using Apache cTAKES 3, an open source software package. The POS model in Apache cTAKES was derived using multiple POS annotated corpora including a corpus of a collection of clinical notes, GENIA corpus [ 9 ] and Penn Treebank [ 4 ], and thus favorable for medical text annotation. In this subsection, we describe the POS-based query term weighting approach and how the Indri queries are generated.

Given the bag-of-words assumption, the conventional Query Likelihood model ranks the documents according to the following ranking function: r(Q; D) = X f (qi; D);

qi where Q represents the query, D the document, qi the ith query term, and f (qi; D) the potential function over qi and D. f (qi; D) could be de ned by various methods, such as tf-idf [ 7 ], BM25 [ 6 ], Jelinck-Mercer method [ 2 ], or Dirichlet smoothing method [11]. The conventional Query Likelihood model treats each query term equally. However, in the medical domain we observe that the query terms are not always equally important. For example, in the query Alzheimer's disease, the term Alzheimer is more important and speci c than disease to understand the query semantics where the semantics can be partially inferred from syntactics. As in the previous example, Alzheimer is a proper noun and disease is a noun. The observation motivates us to assign di erent weights to query terms according to the POS categories. By doing so, the ranking function can be written as: r(Q; D) = q1 f (q1; D) + q2 f (q2; D) + + qn f (qn; D); (2) where qi is the weight corresponding to the POS category of query term qi.

Beyond the bag-of-words assumption, the MRF model considers the relations between terms by using the Markov property [ 5 ]. The ranking function is de ned as: r(Q; D) = X f (c; qi; D) + X f (c; qi; D) + X f (c; qi; D);

c2T c2O c2U where c is the clique set, qi is the ith query term in that clique, and T , O, U denote dependency types full independence, sequential dependence, full dependence, respectively. We can incorporate the POS-based query term weighting approach and de ne the new ranking function as:

( ( ) r(Q; D) = X

)

X qi f (c; qi; D) + X c2T

qi + X c2U ( c2O

) X qi f (c; qi; D) : qi

3 http://ctakes.apache.org/

X qi f (c; qi; D) qi (1) (3) (4) 3.4

Weight Training

The weights can be trained by de ning an objective function based on retrieval performance metrics, e.g., mean average precision (MAP). Our goal is to maximize the performance metrics. Since there are over 30 POS categories according to the Penn Treebank Project 4, this is a multidimensional optimization problem.

We adopted MAP as the performance metric for weight training and considered seven POS categories: (singular or mass nouns (NN), plural nouns (NNS), past participle verbs (VBN), past tense verbs (VBD), adjectives (JJ), adverbs (RB), singular proper nouns (NNP)) and marked all other categories as others. Then we utilized a cyclic coordinate method to solve this optimization problem. The dataset of the TREC 2011 and 2012 Medical Records tracks [ 10 ] was used to train the weights. It contained over 93 thousand de-identi ed clinical reports and 34 test queries for the TREC 2011 and 47 for the TREC 2012. We rst trained on the TREC 2011 data and tested on the TREC 2012 data, and then trained on the TREC 2012 data and tested on the TREC 2011 data. Finally the average weight for each POS category was used to generate the Indri queries for the 2016 CLEF eHealth IR Task. 4

Experiments and Results

4 https://www.cis.upenn.edu/ treebank/

In this paper we presents our participation to the 2016 CLEF eHealth Information Retrieval Task 1. We explored a Part-of-Speech (POS) based query term weighting approach which assigns di erent weights to the query terms according to their POS categories. In the future work, we would like to explore how to utilize the external resources for query expansion in the proposed method.

Acknowledgments

The authors gratefully acknowledge the support from the National Library of Medicine (NLM) grant R01LM11934. 11. Zhai, C., La erty, J.: A study of smoothing methods for language models applied to ad hoc information retrieval. In: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 334{342. ACM (2001) 12. Zhu, D., Wu, S.T.I., Masanz, J.J., Carterette, B., Liu, H.: Using discharge summaries to improve information retrieval in clinical domain. In: Proceedings of the ShARe/CLEF eHealth Evaluation Lab (2013) 13. Zuccon, G., Palotti, J., Goeuriot, L., Kelly, L., Lupu, M., Pecina, P., Mueller, H., Budaher, J., Deacon, A.: The ir task at the clef ehealth evaluation lab 2016: User-centred health information retrieval. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS (2016)

References

1. Pew Internet and American Life Project: Health Fact Sheet . Tech. rep. , Pew Research Center ( 2013 ), http://www.pewinternet.org/fact-sheets/ health-fact-sheet/

2. Jelinek , F. : Interpolated estimation of markov source parameters from sparse data. Pattern recognition in practice pp . 381 { 402 ( 1980 )

3. Kelly , L. , Goeuriot , L. , Suominen , H. , Nvol , A. , Palotti , J. , Zuccon , G. : Overview of the clef ehealth evaluation lab 2016 . clef 2016 - 7th conference and labs of the evaluation forum . In: Lecture Notes in Computer Science (LNCS) . Springer ( 2016 )

4. Marcus , M.P. , Marcinkiewicz , M.A. , Santorini , B. : Building a large annotated corpus of english: The penn treebank . Computational linguistics 19(2) , 313 { 330 ( 1993 )

5. Metzler , D. , Croft , W.B.: A Markov random eld model for term dependencies . In: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval . pp. 472 { 479 . ACM ( 2005 )

6. Robertson , S.E. , Walker , S. , Jones , S. , Hancock-Beaulieu , M.M. , Gatford , M. , et al.: Okapi at trec-3 . NIST SPECIAL PUBLICATION SP pp. 109 { 109 ( 1995 )

7. Salton , G. , McGill , M.J. : Introduction to modern information retrieval . McGrawHill , Inc. ( 1986 )

8. Strohman , T. , Metzler , D. , Turtle , H. , Croft , W.B.: Indri: A language model-based search engine for complex queries . In: Proceedings of the International Conference on Intelligent Analysis . vol. 2 , pp. 2 { 6 . Citeseer ( 2005 )

9. Tsuruoka , Y. , Tateishi , Y. , Kim , J.D. , Ohta , T. , McNaught , J. , Ananiadou , S. , Tsujii , J.: Developing a robust part-of-speech tagger for biomedical text . Advances in informatics pp. 382 { 392 ( 2005 )

10. Voorhees , E.M. , Hersh , W. : Overview of the trec 2012 medical records track . In: The Twenty- rst Text REtrieval Conference proceedings (TREC) ( 2012 )