In our participation in the CLEF 2014 eHealth task 3a, we investigate the e ectiveness of concept-based retrieval techniques on medical IR. Concepts are determined using the existing resources and tools: UMLS Metathesaurus and MetaMap. We tested several methods based on concepts. Although some of these methods lead to slight improvements in retrieval e ectiveness over a traditional bag-of-words method, the impact of the rich domain ressource is lower than we expected. So the whole question on whether and how such a resource can help improve medical IR e ectiveness remains open. In this report, we describe the methods tested as well as their results.
Our experiments on CLEF 2014 eHealth Task 3 [1, 2] aim to investigate the e ectiveness of concept-based approaches in Medical IR. Medicine is possibly the area in which there are the best manually constructed resources for identifying concepts. Metathesaurus [24] is a large thesaurus in medicine, gathering resources such as MeSH [25], Snomed [26], etc. Tools for identifying and disambiguating concepts in texts, such as MetaMap [27] have also be developed. In Metathesaurus a term is linked to a large number of other terms, denoting his synonyms, lexical variants, abbreviations and hypernyms, hyponyms, etc. Intuitively, the availability of those resources and tools should result in better IR e ectiveness than the traditional bag-of-words approaches. However, the previous experimental results have been disappointing. For example, [3] did not observe any improvement using concepts recognized from texts. [4] exploited a statistical thesaurus and obtained 2.2% improvement. [5] used MetaMap to recognize concepts from texts, and used the concepts in query expansion. This led to an improvement of 4.4% over the bag-of-words approach. A number of other studies [6{21] have also used di erent resources and tools. However, the global conclusions are similar: In some cases, slight improvements are obtained, in other cases, no improvements or even degradations are observed. Overall, the experimental results using medical resources and tools for IR have been lower than one expects. The whole question remains: can we really bene t from the rich resources and tools in the medical area to improve IR e ectiveness? Are they related to the way that the resources and tools are used?
In our experiments in CLEF 2014, we would like to examine a few more possible approaches to take advantage of medical concepts. In our experiments, we use MetaMap to recognize medical concepts from documents and queries. MetaMap identi es concepts from a text (document or query). From the concept IDs (CUI - Concept Uni ed Identi er) identi ed, we can further identify the concept word sequence (SUI - String Uni ed Identi er). Our experiments will test several ways to exploit either CUI or SUI. In prticular, we will focus on query expansion using concepts, as query expansion has been shown to be relatively e ective in the previous experiments on medical IR. 2
Let us rst describe the bag-of-words baseline method to which our methods will be compared. Then we will describe how concepts are determined and used in our approaches. 2.1
Baseline As baseline, we use a traditional approach based on language modeling, with Dirichlet smoothing[23]. We use Indri as the basic experimental platform for all the methods. For the baseline method, the score of a document D for a query Q is determined as follows: where n is the length of query and P (qijD) is adjusted by Dirichlet smoothing, S(Q; D) = 1 Xn log P (qijD) n
i=1
P (qijD) =
tfqi;D +
jDj +
tfqi;C
jCj
(
Concept-based IR Concept identi cation We use UMLS Metathesaurus Release2012AB as our resource. A concept is de ned as a \meaning"1. Each meaning is given a CUI (Concept Uni ed Identi er). The di erent synonyms and abbreviations of this concept is called a Term which is identi ed by LUI (Lexical Uni ed Identifer). Each of their lexical variant will be further subdivided into di erent String. SUI (String Uni ed Identi er) is their ID. For example, concept C0004238 corresponds to the meaning atrial fibrillation. While atrial fibrillation and auricular fibrillation are two synonyms, they are identi ed by two different LUIs L0004238 and L0004237. These two terms have both their singular and plural forms, with and without s. So in UMLS concept C0004238 corresponds to 4 di erent SUIs representing its 4 di erent expression strings, called SUIname in Metathesaurus.
MetaMap is a tool that identi es concepts from a text. Among other functionalities, MetaMap can identify the CUI corresponding to the concept string. It can also nd all di erent string expressions (i.e. SUI names) for this concept. CUI and SUI names are the two di erent concept expressions that we used in our experiments. An example is shown in the gure below. Retrieval on concept ID space We can view the whole set of concepts IDs as de ning a concept space. Both document and query can then be represented as a set of CUI that MetaMap has recognized. The ranking score of a document can be determined by the matching score based on the concept IDs using the language model.
S(Q; D) = S(QCUI ; DCUI ) = n1 Xi=n1 log P (qCUIi jDCUI ) It is possible that some of the concepts in documents and queries cannot be correctly identi ed by MetaMap. In this case, a more reasonable approach is to combine the concept-based retrieval with the traditional word-based retrieval. We implement it as follows:
S(QjD) = S(Qorig; Dorig) + (
S(QjD) = S(Qsuiname; Dorig) Again, the above method can be combined with the word-based approach as follows:
S(QjD) = S(Qorig; Dorig) + (
cilastatin dose mass glomerulosclerosis intercapillary concord enterica entericon salmonella ser subsp
adrenergic nerve aeromonadaceae family organism injection mitomycin
murexide blood entity uidity
gene kdm4b cystic disease medullary uremic entire pelvis renal crotalarias bougardirey hemoglobin mali substance abrasive point factor gamma interferon necrosis tumor hazebrouck hemoglobin blanche grange hemoglobin immunosuppressant macrolide hemoglobin henri mondor substance dibromopropamidine product hemoglobin maputo substance abnormal blood nd urea hemoglobin ibadan k hemoglobin vaasa gard hemoglobin ty phosphomannan
In our experiment, the original query is expanded by the top mutual information concepts. In addition, the query is further expanded by the suiNames of the concepts.
S(Q; D) =
1S(Qorig; Dorig) + 2S(Qsuiname; Dorig) + 3S(Qmi; Dorig) (
jCj
jDj +
(
The third component is an unordered model, in which the words within a concept can appear in any order within a text window.
P (qunorderedConceptjD) =
tf#uwk+1(q1;q2;:::qk);D +
jDj +
tf#uwk+1(q1;q2;:::qk);C
jCj
(
Based on the above probabilities, we can de ne S(qorderedConcept; D) and
S(qunorderedConcept; D). The nal score is a combination of these three models,
S(Q; D) = 1S(Qword; D) + 2S(qorderedConcept; D) + 3S(qunorderedConcept; D)
(
The data set for task 3 consists of a set of documents in the mdeical domain, provided by the Khresmoi project. Each document contains #Uid,#date,#url and #content elds. We convert the collection into TREC style. In the content part, we eliminate all commend, css and JavaScript part and all HTML tags. Only the remaining textual contents are indexed. Each query contains <title>, <desc>, <discharge_summary>. We use the short title queries.
The following 12 methods (runs) are tested: 1. baseline (Submitted as GRIUM_EN_Run1) 2. SUIname query, groupped by #1() oprator. 3. SUIname query expansion, groupped by #1() oprator. 4. SUIname query expansion, groupped by #uwN() oprator. 5. SUIname query expansion, groupped by #uwN+1() oprator.(Submitted as
GRIUM_EN_Run5) 6. SUIname query expansion, groupped by #combine() oprator. 7. manual SUIname query expansion, groupped by #combine() oprator. Concepts are identi ed manually. 8. Pure CUI query retrieved in CUI document 9. CUI query expansion, document also contain <original> and <cui> two elds.(Submitted as GRIUM_EN_Run7) 10. Top mutual information and SUI name query expansion.
(Submitted as GRIUM_EN_Run6 )
also be used 11. Markov Random Field baseline with bigram and biterm. 12. Markov Random Field with concept dependence. Only 4 of them (those with the run IDs) have been submitted. 4
The experimental results are summarized in Fig. 4.3
Run1 Run 1
Run a Run b
Run c Run5 Run 5
Run e Run f
Run g Run7 Run 7 Run6 Run 6
Run h Run i
Baseline 0.3945 0.7180 0.4201 #1(SUIname) query 0.2717 0.5680 0.3042 #1(SUIname) query expansion 0.3916 0.6900 0.4217 #uwN(SUIname) query expansion 0.4055 0.7500 0.4279 #uwN+1(SUIname) query expansion 0.4069 0.7420 0.4283 #combine(SUIname) query expansion 0.4112 0.7140 0.4286 #combine(manual SUIname) query expan- 0.4185 0.7540 0.4306 sion CUI query 0.2276 0.4920 0.2692 CUI expansion 0.3495 0.6540 0.3862 #uwN+1(SUIname) expansion + Mutual- 0.4007 0.7120 0.4156
Markov random eld baseline 0.3999 0.7320 0.4175
First of all, we observe that the method using only strict concept space is less e ective than the traditional word-based method. Run g, which use CUI query leads to a degradation of 42.3% compared to the baseline. If we simply compare the \bag-of-words" and \bag-of-concepts" methods, bag-of-words approach is certainly more exible as a retrieve framework.
The result is far from what was expected. That means concept mapping procedure is still the bottleneck of the concept-based approach. Unfortunately, the mapping process is much more complicated than it seems. The de nision of concept itself is not clear. An important hypothesis of \concept" is that \a meaning " should correspend only to one concept. But in fact, in UMLS a meaning can
GRIUM EN Run5 from 5/6 to 1/10. The submitted result was 0.4016 for MAP, 0.7540 for P10. be represented by a single accurate concept or be broken down into smaller concepts. For example, in query 36, for meaning open pelvic fracture, we can have 4 choices: 1. fOpen fracture of pelvisg 2. fFractures, Openg and fPelvisg 3. fOpeng and fFracture of pelvisg 4. fOpeng and fFractureg and fPelvisg
This is not simply an ambiguity, but also a granularity problem. None of them should be judged as de nitly wrong, but their retrieval performance is di erent. In Fig.5, we show the concepts identi ed using di erent strategies:
strategy
superior rami fracture [Convalescence] [Fractures, Open] [Pelvis] [Open] [Frac- 0.4958 ture of pelvis] [Right superior] [Branch of plant] [Fracture]
the concepts identi ed by MetaMap, the broad concepts, narrow concepts and those in the middle level identi ed manually from Metathesaurus, as well as the corresponding MAP score. As we can see, the strategy that group many words into a very speci c concept (Narrow manual) does not produce the best result. On the contrary, the other strategies that break long concepts into parts work signi cantly better. Still, the concepts that we recognize from a text have a large impact on the nal retrieval result. This brings some new challenges for mapping task. [28] reported that MetaMap reached 84% in precision and 70% in recall. However, this evaluation is not done for the purpose of IR. For the 50 test queries, MetaMap identi ed 88 concepts. A rough evaluation indicates that only 66% of them, i.e.58 concepts seem reasonable for IR. We believe that even these concepts may not form the best way to do retrieval.
Knowing that mapping is not always acurate, some compromise solutions have to be used. Our tests show that at least two such strategies can help to reduce the impact of wrong mapping.
First, the most simple way is to also consider the original query. The concept Run name
Run g CUI query 0.2276 -42.3% Run a #1(SUIname) query 0.2717 -31.1% Run b #1(SUIname) expansion 0.3916 -0.7%
pansion +19.4% +72.1% +80.7% +83.9% based synonyms are only treated as a complement to the original query. In our test, Run b, #1(SUIname) expansion brought an improvement of 57.2% over a pure #1(SUIname) query. At Run c, 5, e, f, the combination query brought an improvement.
Second, instead of strict CUI Id, we use SUIname as the expression of concept. As we can see in the result, Run a produced 19.4% less mistake than Run g. In addition, taking into account the fact that concepts IDs can share many words.Using SUIname can further help us retrieving documents on related concepts. That is why, with #combine() operator, Run e achieved the best performance over all 11 automatic runs. Our two MRF runs (Run h and Run i ) showed in another way that naive concept-based dependence does not bring any improvement.
This year in task 3, we tested several di erent ways of integrating concept knowledge. Our results showed that the \bag-of-concepts" is less e ective than \bagof-words" approach. We further discuss about two e ecive ways of reducing the impact of incorrect concept mapping. Original query is indispensable, and SUIname is a more exiable way of using a concept. The mapping performance is still the bottleneck of the concept-based approach. This is a question that we will examine in our future research. 28. PRATT, Wanda et YETISGEN-YILDIZ, Meliha. A study of biomedical concept identi cation: MetaMap vs. people. In : AMIA Annual Symposium Proceedings.
American Medical Informatics Association. p. 529. (2003) 29. STROHMAN, Trevor, METZLER, Donald, TURTLE, Howard, et al. Indri: A language model-based search engine for complex queries. In : Proceedings of the International Conference on Intelligent Analysis. p. 2.6. (2005) 30. MILLER, George A. WordNet: a lexical database for English. Communications of the ACM, vol. 38, no 11, p. 39-41. (1995)