This paper describes the participation of LABERINTO team at the ShARe/CLEF eHealth Evaluation Lab 2014 task 3a. We perform four dierent experiments which consist of a baseline and three variants of the baseline model. The rst was mandatory baseline system with only title and description in the query. Our baseline retrieval system used a Lucene Index scheme with traditional stopping and stemming, no external resources was used. We submitted three additional runs (without the discharge summaries), two from a Lucene-based system with MeSH query expansion and one of which made use of the National Library of Medicine's MetaMap tool to perform term boosting.
ShARe/CLEF (Cross-Language Evaluation Forum) eHealth Evaluation Lab goal
is to evaluate systems that support laypeople in searching for and understanding
their health information [
The LABERINTO group contributed 4 runs to this year’s challenge. Our methods are based on Lucene retrieval engine. It’s the rst time that we are participating in the ShARe/CLEF eHealth Evaluation Lab and, for a rst approximation, our baseline submission uses Lucene’s default standard analyzer to process free-text title and description elds. The remaining submissions build upon this baseline approach. Specically, we consider the contribution to retrieval eectiveness using boosting of Metamap identied terms and two versions of query expansion using MeSH.
This working notes are organised as follows. In Section 2 we describe some issues related to document preprocessing and indexing. Section 3 describes our approaches for task 3a. Section 4 lists the results of our work in comparison to median and best values obtained across all systems. Finally, we make conclusions and state the future work in Section 5. 2
Document collection preprocessing and indexing
The goal of the third task is to provide valuable and relevant documents to
patients, so as to satisfy their health-related information needs. To evaluate systems
that tackle this third task, the lab organizers provide potential patient queries
and a document collection containing various health and biomedical documents
for task participants to create their search system. As is common in evaluation
of information retrieval (IR), the test collection consists of documents, queries,
and corresponding relevance judgements [
This collection consists of web pages covering a broad range of health topics, targeted at both the general public and healthcare professionals. The crawled documents are provided in the dataset in their raw HTML (Hyper Text Markup Language) format along with their uniform resource locators (URL). In order to remove html tags ,raw webpages are preprocessed to extract main content by the Html parser Apache Tika. The Apache Tika toolkit detects and extracts metadata and structured text content from various documents using existing parser libraries including Html. Tika is a project of the Apache Software Foundation and was formerly a subproject of Apache Lucene.
After the data has been cleaned, we indexed all the documents. We used
a very traditional IR system based on the Apache Lucene open-source toolkit
that was essentially a successor to the system used by LABERINTO team for
the 2011 TREC medical track [
Documents and elds are Lucene’s fundamental units of indexing and searching. A document is Lucene’s atomic unit of indexing and searching. It is a container that holds one or more elds, which in turn contain the real content. Each eld has a name to identify it, a text or binary value, and a series of detailed options that describe what Lucene should do with the eld value when you add the document to the index. To index our collection sources, we must rst translate it into Lucene’s documents and elds. Our indexing module takes every clean Html le from preprocessed collection into a single Lucene document.
What we end up after running Lucene is a directory named index, which contains les used by Lucene to associate terms with documents. To accomplish this, a Lucene index was created with a specic analyzer model-dependent. An Analyzer takes a series of terms or tokens and creates the terms to be indexed. A unique kind of Lucene index has been used for all developed models, or in other words, all LABERINTO models for ShARe/CLEF eHealth Evaluation Lab 2014 share the same Lucene index. 3
This section presents the dierent models developed for evaluation. Among the dierent test models developed, four have been selected for submission: Mandatory baseline run using only title and description elds 3. Model with Metamap boost terms 4.
Baseline with Query expansion using MeSH 5.
Baseline with query expansion using MeSH and adding narrative eld 6. All the proposed models no uses the discharge summaries. The dierences between models are described in the following sections. 3.1
This model has been designed to be the simplest approximation to the task. The Baseline model is based on the method of bag of words. In this model, topics text is represented as an unordered collection of words, disregarding grammar and even word order. Therefore, the model matches the words in the topic with the words contained in the index. The usefulness of the model is twofold: provides a basis results for comparing and, on the other hand, its code serves as a basis for implementing more complex models. In order to maintain the simplicity, the baseline model makes match the topics words only with the title and description elds.
To develop this model we used Lucene’s default StandardAnalyzer . The analyzer takes only the text and provides a set of terms to be searched in the index. Our base analyzer discards stops words with little semantic value, such as "the", "a", "an","for",.... Cutting down on the number of terms indexed can save time and space in an index, but it can also limit accuracy.
3.2
Lucene provides the relevance level of matching documents based on the terms found. The higher the boost factor, the more relevant the term will be. Boosting allows you to control the relevance of a document by boosting its term 7.
In this model, we used medical main concepts identication by Metamap.
Queries was built from the title and description elds. Next, UMLS concepts in
queries are recognized using MetaMap [
Because our method don’t need any special score mapping or disambiguation,
Metamap’s default options was used. Only medical term identication
functionality was used. In other words, our method only need medical terms detection in
order to know which topic part needs to be boosted. No overmatches are allowed
and the highers score mappings are selected. For this UMLS detected mappings,
a boost factor hits value is set. In particular, values from 1.25 to 2.25 (carried
out with an interval of 0.25) were applied by an exploratory approach, centred
mainly in the training topics and the ImageCLEFmed 2013 database and topics
[
Term expansion is one possible retrieval technique that can benet from public
accessibility of structured medical vocabularies. Applied at query-time usually
it deals with the problem that real-world concepts are referred to using diferent
terms. An information retrieval system can help users and also automatically
rene their queries by exploiting the semantic relationships between terms [
MeSH (Medical Subject Headings) is a controlled vocabulary, produced and
maintained by the U. S. National Library of Medicine [
In this model, an open source implementation of SKOS-based 8 term
expansion for Solr is used. For every term included in both title and description elds,
term expansion through MeSH is performed with SKOS. In this approach, we
expand the query terms with related terms from MeSH, duplicate related terms
are removed. A default expansion terms weighting of 0.7 is used according to
the previous work of Bernhard, Martins and Magalhªes [
As in the case mentioned above, this model use query expansion with MeSH with SKOS. The only dierence lies in the use of the query expansion in addition of
narrative eld terms. In this case, for every term included in title, description and narrative elds, term expansion through MeSH was performed and terms in narrative eld were added.
It is worth mentioning at this point that narrative eld hasn’t an important relevance for query expansion. Major dierences for information retrieval in this model comes from adding narrative eld terms that MeSH concepts expansion determination.
In relation to this issue, table 1 shows a summary of MeSH concepts detected in topics for query expansion and some data for synonyms extracted from MeSH ontology in the performed expansion. For MeSH concepts, minimum, maximum and average number of MeSH concepts detected per topic has been collected (along with the standard deviation). In the same way, data from the synonyms entries used for expansion are gathered. Two main metrics were taken account for Share/CLEF eHealth 2014 task 3: Precision at 10 (P@10) as primary measure, and Normalised Discounted Cumulative Gain at rank 10 (nDCG@10) as secondary measure. The Share/CLEF eHealth 2014 task 3 built result pools from participants submissions considering the top 10 documents ranked by baseline systems (run 1), and the two highest priority runs that used the discharge summaries (run 2 and 3) and the highest two priority runs that did not used the discharge summaries (run 5 and 6); thus runs 4 and 7 were not sampled to form the assessment pool.
First, we nd that two of our submitted runs outperform the baseline model, one from Metamap boost model (in all metrics) and another one from query expansion model (taken MAP as reference). This shows that these approaches could be more eective than a simple baseline model. In contrast, the concept expansion approach based only in title and description elds decrease the retrieval performance in all metrics. We think that the improper query expansion settings may be the reason of this performance. Our query expansion models dier on elds use, RUN6 model uses the title and description elds, and RUN7 model uses the title, description and narrative elds to select terms to expand. Results show that rening the set of terms used for query expansion often prevents the query drift caused by blind expansion and yields substantial improvements in retrieval eectiveness. Although RUN7 improves map, and taking into account the reference values used at 2014 lab, our query expansion runs have proved that not all query expansion lead to improvements of retrieval.
By other hand, in RUN5, we consider positive the contribution of increase weights of query medical concepts for scoring a document. Empirical results show that considering boost medical concepts along with the original query concepts can improve retrieval eectiveness; which concepts to consider (with Metamap or other tool) and how to weight these is however a challenging issue.
Plots comparing each of our runs against the median and best performance (p@10) across all systems submitted to CLEF for each query topic are shown in gure 1. In particular, for each query the height of a bar represents the gain/loss of our system and the best system (for that query) over the median system.
Per-topic comparison allows observe how performance varies in a important manner by model. Thus, we can see in Fig.1a that baseline has 12 queries perform better than the median, while 26 queries perform worse than the median, and other 12 queries perform in the median line. In g.1b, Metamap based model has 14 queries perform better than the median, while 26 queries perform worse than the median, and other 10 queries perform in the median line. It means that comparing baseline and our best method, the UMLS concept-based method can do something better but heavily needs improvement to surpass baseline.
As regards query expansion models, we can see in Fig.1c that RUN6 has only 8 queries perform better than the median, while 32 queries perform worse than the median, and other 10 queries perform in the median line. In g.1d, RUN7 has just 7 queries perform better than the median, while 33 queries perform worse than the median, and other 10 queries perform in the median line. Though one of the query expansion system has been better than baseline, per topic analysis shows a general poor performance for query expansion. 5
We have presented dierents approachs to medical Information Retrieval from patients’ perspective. Our models were based mainly on concept identication by Metamap and query expansion by MeSH. Both, the concept boosting and query expansion needs to be improved and rened. Some hints to improve, like . ) g i
F + H S e M . terms selection for expansion or tuning boosting parameters, has been exposed. Despite the inconspicuous results, we think that this rst participation provides a platform for further development into medical concept based and query expansion retrieval systems for dealing with medical data from patients’ perspective.
This work has been partially funded by the Andalusian Ministry of Economy, Innovation and Science (Bidamir project, TIC 07629).