This paper presents our second participation in user-centred health information retrieval task at the CLEFeHealth 2015. This task has as objective to retrieve pertinent documents that answer circumlocutory queries posed by users when they are faced with symptoms and signs of a disease. We have submitted ve runs, the rst is the baseline system and the other runs use the pseudo relevance feedback technique to expand the original query in di erent ways. This year, result pools are built from receiving submissions for all participants considering the top 10 documents ranked by every three highest priority runs (1, 2, 3). The obtained results are motivating, but we can be improved with P@10=0:3212 for the baseline system and p@10=0:2939 for the best P@10 in the other runs.
With the proliferation of documents on the web, searching pertinent documents to user's need becomes a di cult task, especially if the queries are not well expressed.
This process of searching documents in a corpus in order to meet a need is called
Information Retrieval (IR). Many information retrieval systems are developed
in order to provide all necessary functions to seek relevant information.
Evaluating these systems becomes important to know the quality of returned
results. For that, International campaigns for assessment have intervened in a
competitive context, such as TREC 1 and CLEF, in order to evaluate several
research systems of the various participants in these competitions.
This paper presents our participation at clef ehealth 2015: User-Centred Health
1 http://trec.nist.gov/
Information Retrieval task [
The goal of this task is to evaluate the e ectiveness of information retrieval
systems when searching health content on the web. The objective is to promote
development of search engines and research adapted to health information.
This task is the further of previous CLEF eHealth Task 3 that held in 2013
and 2014, and adopts the TREC-style evaluation process, with a collection of
documents and queries shared between participants [
In this year's, we study queries which are di erent from queries of the previous
years. In fact, Queries are expressed by not medical expert people whose are
confronted with a symptom or a sign and try to nd more about the disease they
may have [
In our participation, we used Vector Space Model that allows to compute a degree of similarity between documents and queries; also it returns ranking documents according to their relevance. We submitted ve runs, the rst was the baseline system and the others presents obtained results using query expansion techniques based on semantic relationships between terms. 2 2.1
The data set is composed of a crawl of about one million documents, which have
been made available to CLEF eHealth task through the Khresmoi project [
The topic set contains 67 circumlocutory queries that users may pose when
faced with symptoms and signs of a medical condition [
{ Query: longer description given by the user. 2.2
Our work was to index and search the top-1000 relevant medical documents for
each topic. We used for the rst run a very traditional information retrieval (IR)
system based on the terrier platform [
This platform used by MIRACL team for the 2014 Clef Ehealth task [
It is an open source, and a comprehensive and transparent platform for research and experimentation in text retrieval. It is developed at the School of Computing Science, University of Glasgow. 3
This section presents the di erent models developed for evaluation: * Mantadory baseline run. * Baseline runs with automatic query expansion with di erent manners of adding related terms to the original query. 3.1
This model has been designed to be the simplest model to the task. It is
based on Vector Space Model (VSM) developed by Salton [
The most widely used numeric similarity measures to calculate the relevance of a document is the cosine of the angle between the vector of the query and the document vector. 3.2
In this section, we present submitted runs based on query expansion and
using the same model (VSM) employed to mandatory submitted run.
The idea is to use the blind relevance feedback technique; called also pseudo
relevance feedback; to automatically expand the original query without any
user interaction [
In this method based on local analysis , the top k documents returned by
the baseline system are assumed as pertinent (relevant). Then we calculate
semantic relationships between terms of these selected documents as
referring to the statistical least square method (LSM) [
Indeed, we calculate for each term (ti) its relation with other terms in a linear way (see equation 1).
ti 1t1 + 2t2 + + i 1ti 1 + i+1ti+1 + + ntn + (1) Where are the real values of the model and present degrees of relationships between terms and represents the minimum associated error of the relation. In fact, we added to the original query only terms which have that are above a certain threshold. We have submitted four runs that di er by the number of chosen documents (k) and by the xed threshold. 4
For this rst year, the Share/CLEF eHealth 2015 TASK 2 built result pools from receiving submissions for all participants considering the top 10 documents ranked by every baseline system (run 1), and the two highest priority runs (run 2 and 3). This was because, unlike in last year, all submissions greatly di ered between each other, and thus lead to very large (and di erent) pools. Accordingly, principal measures proposed by the Share/CLEF eHealth 2015 TASK 2 to evaluate systems are: cut 10).
Metrics such as MAP (Mean Average Precision) may not be reliable for evaluation due to the limited pool depth.
In table 1, we show obtained results of the ve submitted runs.
In the baseline system (run 1), we have obtained 0:32 of P@10 and 0:27 of ndgc@10. These values decrease slightly in the four other runs that uses a pseudo relevance feedback technique to expand the original query.
In fact, in the second run, we proposed that the top 250 documents returned by
the baseline system as relevant. Then we calculated relations between terms that
appear in these selected documents using the statistical method: least square [
Although this method has given results that are slightly lower compared to results of the rst run, it allows to improve results of some queries. Take as an example the topic (query) 9: < top > < num > clef 2015:test:9 < =num > < query >red itchy eyes < =query > < =top > we took the top 250 documents returned by the baseline system as relevant. This set form a list of 8685 terms. Then we calculated relations between terms in the query with this set of terms and we expanded the original query by adding terms which have positive values of . we have obtained this new query: < top > < num > clef 2015:test:9 < =num > < query > red itchy eye see conjunct allerg nose drop gei zilink zaditor eyelid keratoconjunct lastacaft visin allergi relief antihistamin dose spray allergen tablet pollen claritin hydrochlorid olopatadin levocabastin bottl naphazolin liquid benzalkonium rxnorm formtitl in amm aki stye eyeexamin bedinghaus hive ige swollen pink bruis bump tear eyelash hay springtim slit dermat prescript rash sensit pharmaceut cream zyrtec blister section ketotifen western amerisourc fcab scream dismiss mart wal fbab mite are urticaria ahref erysipela vital health zon tnss ectoin bhcsite mast overreact diphenhydramin azelastin cetirizin seborrh thrush hrcolor uticason iannelli eczemadermat chkd prd beauti skindiseas outgrow respivert msfphover idiopath ledum vitamin tagid zpack eyebright emolli walgreen zyr butterbur hcl botrulelrulerrul allergicchild injector paradis euphrasia pheniramin nit mrhd enclos kmart ayala disk dolgencorp dryl personalhealthzon dbdc spoon teeter allina morri hydrochloridetablet bactrban < =query > < =top > Where the three rst terms present the original query and the other terms present the added terms in their root form.
We notice that almost all added terms are in strong relationship with the terms of the initial query. For example terms like conjunct, allerg, eyelid, in amm, pink, keratoconjunct, dermat present the conjunctivitis disease also known as pink eye. It is an in ammation of the conjunctiva (the outer layer of the eye and the inner surface of the eyelids) due to an allergic reaction.
Terms like olopatadin, eyebright, zaditor, azelastin, levocabastin present some medication names for this disease. Some other terms present forums, sites and library of medicine such as : - rxnorm: National Library of Medicine. - bhcsite: Better Health Channel which provides health and medical information.
- mast: Medical Academy for Science and Technology This expanded query achieves the best performance on p@10 measure in this run (p@10 = 0:5) compared to the rst run (p@10 = 0:1), see gures (1 and 2). This same process is used in (run 3) but relations are calculated just for the top 100 documents.
In (runs 4,5), we calculated relations for the top 100 documents and we added only terms that have above a certain threshold with all terms in the original query. In these latter runs, we tried to keep the context of the initial query to not have a query drift problem.
We notice that measures of P@10 et ndgc@10 in (run 5), that expand query with strong related terms ( > 0:3), are increased compared to runs 2 and 3. Some queries in this run have been improved compared to the baseline system based on p@10 measure. For example query number 5: < top > < num > clef 2015:test:5 < =num > < query >whistling noise and cough during sleeping + children< =query > < =top > In the rst run, we obtained 0:1 of p@10 by against this value becomes positive 0:1 in (run 5) due to the new added terms to the initial query which are in strong relationship with initial query terms. Indeed, the new request is the following: < top > < num > clef 2015:test:5 < =num > < query > whistl nois cough sleep children ear wheez maud depress sound plug scream vitalhealthzon deafness research allergen chkd decibel otiti himend istictac pertussi toclevel maincat audiogram mcug wim opic ufa < =query > < =top >
The plots below in gures (1,2,3,4,5) compares our runs against the median and best performance (p@10) across all systems submitted to Ehealth task for each query.
In particular, for each query, the height of a bar represents the gain/loss of submitted system and the best system (for that query) over the median system. The height of a bar in then given by:
medianp@10(q) medianp@10(q) Fig. 3. Comparison between run 3 and the other systems against the median and best performance Fig. 4. Comparison between run 4 and the other systems against the median and best performance We can see that (run 1) has one query that achieve the best performance, but in runs(2,3) we have ameliorated this number of achieved queries of best performance, we have obtained at respectively 8 and 3 (like query number 9 in run 2). In (run 5), we have obtained just one query that achieved this best, although it achieves 21 queries perform better than the median line, while 12 queries were worse than the median.
In comparison with (run 2) which have 16 queries perform better than the median line and 28 queries were worse than the median, we can conclude that proposed method to expand the original query taking into account the context of this latter car improve results for some queries but we need to improve general result of system. 5
In our second participation in User-Centred Health Information Retrieval Task at the CLEF eHealth 2015 in order to evaluate our proposed expansion query method based on semantic relationships between terms de ned with a new statistical method, we obtained motivating results when searching in a large collection of medical documents to answer circumlocutory queries that users may pose when faced with symptoms and signs of a medical condition. For future work, we will try to improve these obtained results by modifying parameters of k and and by using reduction dimensionality techniques to cope with problems of sparse and large dimension matrix.