This paper presents our approach to medical information retrieval and experimental results of participating in eHealth Task 3-A of CLEF 2014. The task is to retrieve relevant documents from a medical collection given a query generated from a discharge summary. The key idea of our method is to compute accurate similarity scores via multiple stages of re-ranking documents from initial documents retrieved by a search engine.
Health-related content is one of the most searched-for topics on the internet. This became an important domain for research in information retrieval (IR). Recently, medical IR is actively researched to tackle diverse medical information sources including the general web, journal articles, social media, and hospital records. However, medical IR is still challenging because it should consider various information needs from a wide range of users including patients and their care givers, researchers, clinicians, practitioners, etc. Moreover, it is highly co-related with those users’ background medical knowledge and language skills.
eHealth Task 3-A of Conference and Labs of the Evaluation Forum (CLEF) 2014
[
This paper presents a multiple-stage re-ranking method which focuses on utilizing various retrieval techniques rather than exploiting utilizing external resources and NLP techniques.
In particular, our proposed method passes through multiple re-ranking stages to elevate the ranked position of most relevant documents. Basically, we first perform query expansion with abbreviations, and pseudo relevance model in the end. In the middle of the re-ranking, query expansion with discharge summary, clustering-based document scoring, and centrality-based document scoring can be combined selectively or sequentially.
The rest of this paper is organized as follows. Section 2 delivers related researches of medical information retrieval. Section 3 presents our re-ranking method in detail. The experimental results are described in Section 4. Section 5 concludes with short summary. 2
Recently, many IR researches have been performed with different types of medical
collections. TREC held medical track in 2011 and 2012. A research [
CLEF held eHealth Lab in 2013. A research [
MedSearch system [
The key idea of our method is to re-rank top-k documents via multiple stages for computing more accurate similarity scores with respect to a query.
Figure 1 shows the overview of our multiple stages re-ranking method. For a given
query Q, a set of documents, , , … , , are retrieved from a collection C
using a search engine. In our implementation, initial documents are retrieved by
Lucene1 using a query-likelihood method with Dirichlet smoothing [
(a) Searching Initial documents (b) Re-ranking initial documents
Throughout re-ranking, KL-divergence method is utilized to compute a similarity
score between a query and a document [
To avoid zero probabilities and improve retrieval performance, a document model is
estimated using Dirichlet smoothing [
The first stage aims at expanding a query with abbreviations. In numerous numbers
of medical documents, abbreviations are widely used to represent important
meanings. Unfortunately, the clear interpretation of abbreviations is quite difficult due to
the existences of several different meanings for a same abbreviated expression.
Similarly, medical queries generated by users may also contain abbreviations. If we
submit a query including abbreviations, it may not match relevant documents due to term
mismatch problem or may match documents with abbreviations implying different
meanings. To deal with this problem, query expansion considering abbreviations is
considered. To do that, we extract pairs of abbreviation and corresponding full
representation with an occurrence count using simple rule-based extraction method [
,
The second stage is to reflect information from a discharge summary. A query used
in CLEF eHealth Task-3 is generated by a human expert after reading a discharge
summary corresponds to the query. Therefore, it has hidden but useful information
not captured by a query. The use of a discharge summary can improve retrieval
performance by utilizing such hidden information. To do that, a query model is expanded
by combining a random-walk based discharge summary model. First, we should
compute word-to-word transition matrix to measure the associations among words in a
discharge summary. A simple solution is to use a co-occurrence count between two
words among all sentences [
⋅ where | | is the number of unique words in a discharge summary DS and damping factor .
We approximate the resulting as a discharge summary model and update the query model with it: 1 ⋅ ∑∈ ,
, ⋅ ∈
| ⋅ | where 1, , ,
| | log _ Then, a transition probability is computed: is a distance between words w and u, N is a window size, wt n
is a co-occurrence count of w and u within k-distance, and where is a discharge summary document Based on the translation matrix | random-walk: , word centralities are computed using where is a control parameter and | is a discharge summary model.
The third stage is to incorporate cluster information of documents. Namely, a score
for a document is computed by incorporating the membership of the document to a
cluster we constructed. Bottom-up hierarchical agglomerative clustering [
A new score is computed by combining the initial search score and the cluster score: exp ||
Based on those results, a similarity matrix with the initial documents and corresponding α documents is constructed. Then, random-walk is executed on this matrix to produces centrality scores for the initial documents. This score is multiplied with the previous score: , , ⋅ , The fifth stage is pseudo relevance feedback. A popular way of query expansion is to update a query based on pseudo-relevance feedback (PRF). Updating a query with PRF assumes that top-ranked documents , , … , | | in an initial search results relevant to a given query and terms in F are useful to modify a query for a better representation. Relevance model (RM) is to estimate a multinomial distribution | that is the likelihood of a term w given a query q. The first version of relevance model (RM1) is defined as follows:
RM1 is composed with three components: document prior , document weight | , and term weight in a document | . In general, is assumed to be a uniform distribution without the knowledge of a document D.
| ∏∈ | , indicates the query-likelihood score. | can be estimated using various smoothing methods such as Dirichlet-smoothing. Various strategies are applicable to estimate these components.
To improve the retrieval performance, a new query model can be estimate by combing a relevance model and an original query model. RM3 [16] is a variant of a relevance model to estimate a new query models with RM1: where model.
Based on this query model, final scores for documents are computed.
is a control parameter between the original query model and the feedback
As mentioned, initial documents are retrieved by Lucene using a query-likelihood method with Dirichlet smoothing. We limited the size of the initial documents to 100. Based on the initial documents, we submitted 7 runs by differentiating the components of our re-ranking method.
Table 1 shows the parameter and corresponding values for each component in the experiments. Table 2 describes involving components at each run and evaluation results from corresponding runs. Basically, component 5 which indicate the use of PRF is applied to all runs thus regarded as baseline of our experiments. Except Run01, all runs utilize component 1. The distinction between Run02-04 and Run05-07 is that the former uses discharge summary while the latter doesn’t. Precision and normalized discounted cumulative gain (NDCG) are used to measure the performance of top-10 ranked documents from 100 initial documents. They are denoted as P@10 and NDCG@10, respectively.
Our baseline achieved 0.7300 and 0.7235 in P@10 and NDCG@10, respectively. It shows that PRF is an effective solution to find correct medical documents. For precision, the best performance, 0.7400 is obtained from Run02 which utilizes abbreviations and discharge summary. For NDCG, the best performance, 0.7333, is obtained from Run04 which uses all components in the re-ranking method. This shows that sequentially combining the proposed components is contributed to achieve the best performance in NDCG measure. However, clustering and centrality-based document scoring were not effective in enhancing precision measure.
KISTI_EN_RUN04 KISTI_EN_RUN05 KISTI_EN_RUN06 KISTI_EN_RUN07
O O O O O
O O
O O O O
O O
O O O O O
Due to quite high baseline (i.e., Run01) obtained by PRF with relevance model and lack of in-depth study on the provided healthcare dataset, our experiments fail in showing drastic improvements in evaluation measures. Meanwhile, the moderate performances observed in our multi-stage approach to re-ranking documents (i.e., Run04) may arise from synergistic effects between involved components. The detailed analysis on the involved components in terms of causal and sequential effects is remained as our future work. 5
This paper shows a multiple stage approach to re-ranking medical documents. Our method focuses on utilizing various retrieval techniques rather than utilizing medical dependent external resources and natural language processing to understand medical meanings. We found that using abbreviations and discharge summary play an important role to find correct medical documents. Our future works include further development of two components and in-depth error analysis based on standard assessment dataset. 16. Abdul-Jaleel, N., Allan, J., Croft, W., Diaz, F., Larkey, L.: UMass at TREC 2004: Novelty and HARD. (2004).