In this paper, we reported some experiments conducted by our members in the SIG team at the IRIT laboratory in the University of Toulouse within the context of the medical information retrieval (IR) task. As in our previous participation in ImageCLEF, in 2011, our research focuses on the case-based retrieval task. We compared the performance of different state-of-the-art term weighting models for retrieving patient cases that might best suit the clinical information need. Furthermore, we also combined term scores obtained by two state-of-the-art weighting models using a particular data fusion technique. Finally, a state-of-the-art query expansion (QE) technique is used for improving biomedical IR performance.
This paper describes the contribution of the SIG team (Generalized Information
Systems) at the IRIT1 (Institute for Research in Informatics of Toulouse)
laboratory in its second year participation at the medical retrieval track. We focused
in particular on the case-based retrieval task, where patient demographics,
limited symptoms and test results are provided to answer the medical professionals’
information need [
We first investigate the effectiveness of two different state-of-the-art term
weighting models that have been shown to work well in the past: LGD (a log
logistic model) [
The rest of this paper is organized as follows: Section 2 describes our indexing and retrieval framework. Experimental results will be presented and discussed in section 3. We conclude the paper in section 4 and outline some perspectives for our future work. 1 http://www.irit.fr
The indexing aims to organize, structure and store statistical and/or linguistic
information about terms and documents in the collection allowing a rapid and
efficient search. We use the Terrier IR platform for indexing documents [
The document retrieval aims to match the user query and document
representations in order to retrieve a list of results that may satisfy the user information
need. In our work, a document D containing terms used for formulating query
Q is weighted by summing the score of each term figuring in document D:
RSV (D, Q) = X score(t ∈ D)
t∈Q
(1)
where score(t ∈ D) is the query term weight calculated using a particular term
weighting model. In this section, we first describe two different term weighting
models used in our experiments, namely LGD [
In the LGD model, query terms are weighted using the log logistic distribution
[
Nt + tfn) − log2( Nt scoreLGD(t ∈ D) = qtf × log2( N N ) (2) where – t is a query term occurring in document D, – Nt is the document frequency (i.e., number of documents containing term t), – N is the total number of documents in the collection, – qtf is the query term frequency, – tfn is the normalised within-document term frequency, given by: tfn = tf × log2(1 + c × where avg dl is the average document length (in tokens), dl is the document length (in tokens) and c is a multiplying factor or tuning parameter.
For the In expB2 model, query terms are weighted using the Inverse Expected
Document Frequency model with Bernoulli after-effect and term frequency
normalisation [
scoreIn expB2(t ∈ D) = where – t is a query term occurring in document D, – Nt is the document frequency, – N is the total number of documents in the collection, – qtf is the query term frequency, – tf is the within-document term frequency, – tf n2 is the normalised within-document term frequency, given by: qtf × (tf + 1) × tf n2 ×log2 Nt × (tf n2 + 1) × ln2
N + 1 N × (1 − e −Ntf ) + 0.5 (4) tf ln2 tf n2 = × log2 1 + c × avg dl dl (5) 2.3
In the context of information retrieval, data fusion refers to the task of combining
the output of multiple ranking strategies into a single list of objects (documents,
concepts, etc.) [
With the development of information technology and communication,
especially in the IR field, a large number of IR models have been developed and
integrated into the Terrier IR platform [
N N
) + qtf ln2 × (tf + 1) × tf n2 × log2 N × (1 − e−tf/N ) + 0.5
N + 1 Nt × (tf n2 + 1) (6) (7) (8) (9) 2.4
The DFR framework employs a query expansion (QE) mechanism that is a
generalisation of Rocchio’s method [
Inf oDF R M axInf o where – qtf n is the normalised within-query term frequency, – M axInf o = argt∈q∗ max Inf oDF R, – Inf oDF R is the term frequency in the expanded query induced by using a DFR model, that is:
Inf oDF R = − log2 P rob(F req(w|K)|F req(w|C))
where Prob is the probability of obtaining a given within-query term
frequency from the top-ranked documents retrieved in the first stage. In the
DFR framework, several measures are used to compute this probability such
as: Bose-Einstein (Bo) statistics and Kullback-Leibler (KL) measure [
InfoBo = − log2 Prob(Freq(w|K)|Freq(w|C))
= − log2( 1+1λ ) − Freq(w|K) ∗ log2( 1+1λ ) where • F req(w|K) (resp. F req(w|C)) is the the term frequency within the top ranked documents (resp. the collection) • λBo1 = Freq(w|C) and λBo2 = TotalFreq(K)∗Freq(w|C)
N TotalFreq(C) while the latter gives the following term frequency normalisation: InfoKL = T otalF req(K) ∗ log2 FF rreeqq((ww||CK))∗∗TToottaallFFrreeqq((KC))
F req(w|K) (10)
Some statistical characteristics of the Case-based 2011 collection is depicted in table 1. The histogram in figure 1 shows the variation of document length in the collection.
Number of documents Average document length Total number of tokens Size of the vocabulary
Number of queries Average query length Retrieval performance was evaluated using standard measures: P@10, P@20, and MAP. P@10, P@20 represent respectively the mean precisions at the top 10, 20 returned documents. MAP (Mean Average Precision) is the average precision of a query which is computed by averaging the precision values computed for each relevant retrieved document of rank x ∈ (1..K), where K = 1000 is the number of retrieved documents. Our results are generated by the trec eval standard tool2 used by the TREC community for evaluating ad hoc retrieval runs. 3.3
We submitted ten official runs to the case-based medical retrieval track [
According to the results presented in table 2, we see that ignoring low IDF terms
does not help and even harms the IR performance. Normally, low IDF terms are
not useful for describing the semantics of the document and can be ignored from
the document index [
Here, we compare the performance of the two mentioned state-of-the-art models LGD and In expB2. We notice that the LGD model is better than the In expB2 model in terms of MAP with an improvement of +17.36%. In terms of P@10, the two models yield the same performance, but in terms of P@20, the former is better than the latter with an improvement of +22.2%. For this reason, we chose the IRIT LGDc1.0 run as our strong baseline to compare to other runs.
The CombSU M method combining term scores obtained by those models outperforms the In expB2 model and similar to the LGD model (baseline) in terms of MAP. In terms of P@10, the CombSU M fusion technique outperforms the baseline with an improvement of +29.97%. However, in terms of P@20, the 2 http://trec.nist.gov/trec_eval/
Includes terms with low IDF 1 IRIT In expB2c1.0 1* 0.0743 0.1111 0.1000 2 IRIT LGDc1.0* (baseline) 0.0872 0.1111 0.1222 3 IRIT CombSUMc1.0 3* 0.0859 0.1444 0.1000 4 IRIT CombSUMc1.0 KLbfree d 20 t 20 2 0.0874 0.1111 0.1000 5 IRIT LGDc1.0 KLbfree d 20 t 20 1 0.1030 0.1556 0.1278 6 IRIT In expB2c1.0 KLbfree d 20 t 20 0 0.0772 0.1000 0.1000
Ignores terms with low IDF from index 7 IRIT CombSUMc1.0 KLbfree d 20 t 20 2 0.0874 0.1111 0.1000 8 IRIT In expB2c1.0 KLbfree d 20 t 20 0 ignore low idf 0.0793 0.1444 0.0889 9 IRIT LGDc1.0 KLbfree d 20 t 20 1 ignore low idf 0.0937 0.1111 0.0889 10 IRIT CombSUMc1.0 2 ignore low idf* 0.0721 0.1333 0.0778 CombSU M technique gives the same performance as the In expB2, which is lower than the baseline. We conclude that combining term scores at the level of weighting models can be useful for improving the search precision (P@10), without losing the MAP performance.
At this level, we present the results of submitted runs obtained using query expansion. The In expB2 model in combination with the KL QE method shows a small improvement in terms of MAP (+03.90%), a decrease in terms of precision P@10 (-10.00%) and no effect in terms P@20 compared to the In expB2 model without QE. This is probably due to the fact that the clinical query length is long (about 30 in average) and the number of extracted terms for QE is smaller than or equal to 20; therefore extracted terms may be observed as in the original terms or also they can be different from the latter but the top-ranked documents do not or slightly change after expansion. The LGD model in combination with the KL QE method (run IRIT LGDc1.0 KLbf ree d 20 t 20 1) outperforms the baseline with an improvement rate of +17.85% in terms of MAP, +40.05% in terms of P@10 and +4.58% in terms of P@20. This proves that query expansion is only effective if it is based on an underlying effective ranking model. Indeed, the In expB2 model in combination with the KL QE method performs worse than the baseline. This also explains why the CombSU M method in combination with the KL QE gives no improvement compared to the baseline. 4
In this work, we have compared and evaluated the IR performance of two stateof-the-art term weighting models, a state-of-the-art query expansion approach. In our empirical studies, we proposed to combine term scores obtained by different term weighting models to improve the retrieval performance, especially the search precision.
Within the case-based retrieval task, we noticed that low IDF terms are
also useful for indexing and retrieval because they can be used to mention or
distinguish medical concepts. The LGD model proposed by [
Since documents in the case-based collection contains a lot of medical concepts, in our future work, we aim to extract concepts from documents for better representing the document’s semantics. In addition, we’ll also focus on adjusting query by expanding the query with related terms denoting concepts in ontologies or removing non informative terms from the query.