It's difficult for laypeople, even clinicians, to understand eHealth contents found in the web medical documents. With the objective to build a health search engine, task 2 of 2015 CLEF eHealth aims to detect levels of accuracy of information retrieval systems when searching for web medical documents. In this task, our approach is to integrate a retrieval of medical concepts into the preprocessing of corpora. This means that all terms in the documents that are not related to medicine are removed before indexing. We also expand queries for searching more effectively. In general, our results are not better than other participants' in doing task 2 except some queries. When using integration of extracting medical concepts and query expansion based on laypeople's queries, searching retrieval is also lower. It can be explained partly that laypeople's queries are not commonly included medical terms or only contain features painting their health situations. In addition, we also give a brief statement of the main points of an estimation of readability which is a significant assessment referred by CLEF eHealth near future.
Laypeople as well as clinicians find it hard to comprehend the eHealth documents which are retrieved from searching for necessary information on the Internet. Their problems are how to understand professional terms more exactly. It’s the third year when CLEF eHealth continues the purpose of promotion in doing research and finding out advanced methods to build a search engine system for meeting users’ requirements of searching for medical information.
CLEF eHealth pointed out two tasks this year1 . Task 1 is a mission
statement of information extraction from clinical text. It is split into two sub-tasks:
task 1a and task 1b. Specifically, task 1a is clinical speech recognition related
to converting verbal nursing handover to written free-text records and task 1b
is named entity recognition in clinical reports. Task 2 is user-centered health
information retrieval [
There are some changes of type of queries and measure methods of relevance assessments this year. One of them is queries that should be made by laypeople do not come from experts in the heath domain because the laypeople want to find out information to help them to be clearer their related medical conditions. Another change is to apply two measures to assess results from participants’ submissions.
As a further matter, CLEF eHealth has also considered a readability-biased
assessment [
In this paper, our approach is to integrate retrieval of medical concepts
from the web medical documents into the preprocessing of corpora which is
based on a list of medical concepts built by [
We obtained a baseline run and other runs after doing experiments in
searching for queries in Index A and B corpus with Dirichlet smooth
coefficients [
We also expanded queries to get more information for searching. Techniques to do this are described in Section 3.
2 https://sites.google.com/site/clefehealth2015/task-2
In general, most of our results are not better than other participants’ in
doing task 2 except some queries (see Figure 2, 3). In addition, when integrating
the method of extracting medical concepts [
The rest of the paper is organized as follows: Section 2 outlines the CLEF eHealth dataset and methods of preprocessing. Section 3 describes the structure of a query and some techniques for query expansion. Section 4 presents relevance assessments. Section 5 demonstrates description of our runs. Section 6 explains experiments done by task participants. Finally, Section 7 concludes the paper. 2
The dataset for Task 2 is provided by Khresmoi project3. It has about one million documents, a set of documents in the HTML (Hyper Text Markup Language) format. All documents are collected from well-known health and medical sites and databases in 2012. The size of the dataset is about 6.3G in compressed status and approximately 43.6 GB after extracting.
Each file in the dataset is in the format of .dat files and contains a set of web pages and metadata where shows the original information of each web page as described below: a unique identifier (#UID) for a web page in this document collection, the date of crawl in the form YYYYMM (#DATE), the URL (#URL) to the original path of a web page, and the raw HTML content (#CONTENT) of the web page 2.1
Majestic-12, Distributed Search Engine (DSearch) projects4, built an opensource tool called HTML parser v3.1.4 for parsing tags in the HTML files. Basing on this tool, we extracted text contents from tags of HTML documents in the dataset. There are some tags in the documents that contain unnecessary
4 http://www.majestic12.co.uk/projects/ texts for building a search engine system in the medical field. Thus we tried to ignore all those. 2.2
The text contents extracted from HTML documents are not always good for building the system. Example, tags contain text of sitemap, update information and some items on the main and popup menu in the HTML documents. Therefore, all of them should be removed.
Next, we had a process of removing stopwords because they were common
words in English language [
We used a list of medical concepts built by [
After this processing, we had a dataset that contains terms related to medical field. We also used Lucene-5.0.0 to index this dataset and named Index B. 3
Because of consideration in building a search engine system for English documents, we only concentrated on English queries. The number of queries for task 2 of 2015 CLEF eHealth includes 66 queries along with their narrative fields. The narrative fields are used to provide information to the assessors when performing relevance evaluations. Here is the typical structure of a query7: 5 http://lucene.apache.org/ 6 http://www.nlm.nih.gov/research/umls/ 7 https://sites.google.com/site/clefehealth2015/task-2 <top> <num>clef2015.training.1</num> <query>loss of hair on scalp in an inch width round</query> <narr>Documents should contain information allowing the user to understand they have alopecia</narr> </top>
With purpose of getting more information for searching, with some tasks
of CLEF eHealth before, participants or teams found out synonym of query’s
terms in the UMLS or MeSH8 (Medical Subject Headings) for expanding their
queries [
Methods of relevance assessments are provided by the Share/CLEF
eHealth 2015 TASK 2 and described as follows:
Result of runs is the top 1000 of relevant documents returned by
searching for 66 queries that based on LM (language model) [
o 2 point scale: non relevant (label 0); relevant (label 1)
─ Evaluation with nDCG: [
o 3 point scale: gain 0 (label 0), gain 1 (label 1), gain 2 (label 2)
─ Readability-biased evaluation: [
9 https://en.wikipedia.org 10 http://trec.nist.gov/trec_eval/ o 4 point scale: very technical and difficult (label 0), somewhat technical and difficult (1), somewhat easy (label 2), very easy (label 3) 5
With given 66 queries, we retrieved the top 1000 of relevant documents for
each query when using LM in Lucene 5.0 for matching the queries with each
document in the Index A or B corpus along with specific values of Dirichlet
smooth coefficients [
Run 2: It is a variant of run 1 in which value of smooth coefficient is 10000. Run 3: We used the corpus of medical concepts (i.e. Index B) for searching with the same smooth coefficient as run 1.
Run 4: It is a variant of run 3 in which value of smooth coefficient is 10000. Run 5: Each query, we took the top 01 of relevant documents at Run 3 for expanding the query. Executing the same run 1, but for this expanded query. Run 6: It is a variant of run 5 in which the top 01 of relevant documents at run 4 was used to expand the query. Executing the same Run 2, but for this expanded query.
Run 7: Each query, we took the top 01 of relevant documents at run 3 for expanding the query. Executing the same run 3, but for this expanded query. Run 8: It is a variant of Run 7 in which the top 01 of relevant documents at run 4 was used for expanding the query. Executing the same run 4, but for this expanded query.
Two primary evaluation parameters for task 2 of 2015 CLEF eHealth are the precision at 10 (P@10) and Normalized Discounted Cumulative Gain (NDCG) at rank 10. Figure 1 shows the results of the submitted eight runs. It can be seen clearly from Figure 1 that run 1 (baseline run) is the best run yielding the highest values followed by run 2 to 5, 7 respectively. Whereas run 8 is the least performing run and following it is run 6. It is observed that the values of nDCG are higher than P@10 in processing with integration of query expansion into the system (run 5 to 8).
1 2 3 4 5 6 7
8
Runs
Demonstration of Figure 1 shows that our approach of integrating a
retrieval of medical concepts [
The Figure 1 also indicates that run 5 to 8 with performance of query expansion techniques is worse than other runs. Thus it can be explained that laypeople’s queries are not commonly included medical terms or only contain terms painting their health situations. In case of that the relevant document for query expansion evaluated by CLEF eHealth is not related to the query, retrieval documents from searching for that query are not also in higher relevance assessments. Another reason for explanation of bad results is that query expansion is only a connection between a query and the relevant document at the top of the searching result of the query. As the result, new queries are too long to apply to LM.
Figure 2 shows the graph structure of the participants’ performance of baseline run (run 1) in task 2 of 2015 CLEF eHealth. For the baseline run, it is observed that our experiment performed most of queries in under the best & median cases of all participated groups. But some queries, we reached outperformance more than other systems as queries: 13, 26, 29, and 65. A few queries are in lags such as 3, 32, 33, and 66 respectively.
Figure 3 shows the performance graph of the participants’ run 2 in the task
2 of 2015 CLEF eHealth. For the run 2, it is observed that our experiment
only out-performs more than other systems in queries 15, 16, and 25
respectively. On the other hand, our run 2 system lags in queries 5, 20, 30, 32, 34,
46, 51, 57, 61, and 65 respectively. Figure 3 points out that increasing value
of smooth coefficient is not efficient.
When applying some techniques as retrieval of medical concepts [
For this year task, it’s the first time CLEF eHealth has considered the
factor of readability (understandability) within the evaluation of the submissions
with assumption that readability is assessed independently of relevance
assessments. To account for readability in the evaluation, we have computed an
understandability biased measure, uRBP [
We used the ubire-v0.1.0 tool11 to point out the values of RBP [
CLEF eHealth also notes that these readability-biased measures are still being experimented and observations that were made with the provided measures may not be conclusive.
1 2 3 4 5 6 7
8
Runs 11 https://github.com/ielab/ubire throughout runs. This is a reductive trend except a fluctuation at run 7. That the line of uRBP is under the uRBPgr line shows that mapping graded understandability scores to different probability values reaches a bit outperformance. 7
It’s necessary to build an efficient retrieval system for searching laypeople’s queries. However, this is not easy and still a challenge for participating groups in CLEF eHealth. This is partly because the features in laypeople’s queries are not utterly medical terms or only terms of description of their health situations. Thus, searching of retrieval systems for laypeople’s queries should be returned.
We carried out some experiments in applying our approach of retrieval of
medical concepts [
In future work, we will keep finding out advanced methods to refine corpora so that they only contain suitable features. Simultaneously, we will consider query expansion techniques, and experiment on various models of matching documents. In addition, we also do research on the results of other participating groups’ systems to make them better.