This paper describes the participation of the SNUMedinfo team at the CLEFeHealth2014 task 3. We submitted 7 runs to Task3a (monolingual information retrieval): 1 baseline run using query likelihood model in Indri search engine; 3 runs applying UMLS based lexical query expansion utilizing discharge summary as an expansion term filter; 3 runs applying learning to rank technique utilizing various document features. We submitted 4 runs to Task3b (multilingual information retrieval): 1 baseline run using Google Translate for the English translation; 3 runs applying learning to rank technique on the translated query.
In this paper, we describe the methods used for our participation of the
CLEFeHealth2014 Task 3 User-centered health information retrieval. For detailed task
description, please see the overview paper of task3 [
We submitted 1 baseline run (SNUMEDINFO_EN_Run.1) using unigram
language model with Dirichlet smoothing [
In task3b, queries are expressed in French, German and Czech. We used Google
Translate [
We submitted 3 runs applying UMLS based lexical query expansion, utilizing discharge summary as an expansion term filter. In this method, document content of discharge summary is assumed as user context information.
Firstly, MetaMap [
Original query part is weighted by 0.9 and expansion query part is weight by 0.1. e.g., Indri query example #weight ( 0.9 #combine ( convalescence open pelvic fracture right superior rami fracture ) 0.1 #combine ( fractures open pelvis pelvic right superior ) ) We applied different parameter settings per each run. In Run Number 2 (which corresponds to SNUMEDINFO_EN_Run.2), we used title field as a query. In Run Number 3, we used title, description and narrative field as a query. In Run Number 4, we used title field as a query, and Dirichlet prior parameter is changed to 1,000. 2.3
We applied learning to rank to incorporate various document features into the ranking model.
The following features are extracted per each top 1,000 documents. 1. Relevance score of query likelihood model from fulltext index 2. Rank of query likelihood model from fulltext index 3. Relevance score of query likelihood model from title+url index 4. Rank of query likelihood model from title+url index 5. Document quality features Feature 1 and 2 refers to the relevance score and rank acquired from baseline retrieval model on fulltext index.
Feature 3 and 4 refers to the retrieval result acquired from the index built from keyword content of document. We assumed that terms occurring in the title (e.g., <title> Community-acquired pneumonia </title>) and the url field (e.g., http://www.merckmanuals.com/home/lung_and_airway_disorders/pneumonia/commu nity-acquired_pneumonia.html) as keywords of document.
With regard to the feature 5, we hypothesized a certain notion of document content quality of medical web documents, which defines ideal characteristics that relevant documents could have in common across different types of queries. For example, content reliability, comprehensiveness, whether writing is well structured and so on.
For general web search task, there are several prior studies trying to assess web
page quality [
In the medical domain, several prior studies tried to evaluate methodological
quality of medical literatures [
In [
In this study, we tried to combine medical web page quality assessment with topical relevance ranking. We used only textual features for medical webpage quality assessment. We tried to identify terms possibly relevant to the medical document quality evaluation. Each term’s document term frequency is used as a feature value. First author arbitrarily collected about 80 words which considered to be relevant to the document quality (Table 5). These terms are punctuation removed, case-folded, tokenized and stemmed, so finally 82 unique terms are used. We included terms such as ‘etiology’, ‘prognosis’, and ‘treatment’. When we write any text, we often asked to write in terms of 6W such as what, why, where. We thought that these ‘etiology’, ‘prognosis’ terms could be necessary condition to be well-organized medical document, analogous to the 6W. We also included terms like ‘md’. When medical doctors wrote document content, in many cases they write their name and qualification at the end of text. We thought that these information could give some assurance about the quality of document content.
We used CLEFeHealth 2013’ Task 3 [
We used random forest algorithm [
We applied different parameter settings per each run. In its default setting, minimum leaf support parameter is 1, and number of bags parameter is 300. In Run Number 5, we set minimum leaf support parameter to 10. In Run Number 6, we set the number of bags parameter to 50. In Run Number 7, we set the minimum leaf support parameter to 50 (For French, German and Czech language queries in task 3b, parameter setting is same as English query). 3 3.1
Experimental results are described in Table 1, Table 2, Table 3 and Table 4. In Task3a (Table1), primary evaluation metric was P@10 and secondary evaluation metric was NDCG@10. Our baseline P@10 was 0.738.
SNUMEDINFO_EN_Run.2 (UMLS query expansion method) and SNUMEDINFO_EN_Run.5 (Learning to rank method)’s performance is slightly improved than SNUMEDINFO_EN_Run.1 (Baseline). The amount of performance gain was not large enough: on average, 1~2% improved in terms of P@10 compared to the baseline. If we compare SNUMEDINFO_EN_Run.2 to the baseline performance in terms of P@10, 5 queries improved, 41 queries unaffected, 4 queries harmed. If we compare SNUMEDINFO_EN_Run.5 to the baseline performance in terms of P@10, 14 queries improved, 32 queries unaffected, 4 queries harmed. But both of them failed to attain significant improvement against baseline method when we performed twotailed paired t-test. Nevertheless, we think that both of our methods; (1) lexical query expansion using discharge summary as context filter; (2) learning to rank approach utilizing medical web page quality features; showed good potentials of improvement against tough baseline. We will try to improve our methods in our future research.
Regarding Task3b (Table2, 3 and 4), we just used Google Translate to translate French, German, and Czech query. These translated queries showed comparable performance compared to the original English query. 4
In CLEFeHealth 2014’ Task 3, we tried to test various retrieval technique. We tried lexical query expansion methods and learning to rank approach utilizing various features for medical web document. Evaluation results shows potentially promising results. We will try to improve our methods with more experiments in the future study.
This study was supported by a grant of the Korean Health Technology R&D Project, Ministry of Health & Welfare, Republic of Korea. (No. HI11C1947). 5 ethnicity etiology evidence follow up followup frequency frequent gender guide guideline history hospital illness introduction journal literature management md medical doctor medication medicine morbidity mortality ms occurrence op operation overview pathology pathophysiology phd physiology prevent prevention problem prognosis proof publication publish race radiology recommend recommendation research article risk factor sex sign statistic study sx symptom test therapy treat treatment tx update w/u work up workup