This paper presents the details of participation of DEMIR (Dokuz Eylül University Multimedia Information Retrieval) research team to the Share/CLEF eHealth 2014. This year, we participated to task 3a: monolingual user-centered health information retrieval. In this task, we focused to apply query expansion techniques selectively to some queries to improve the performance of information retrieval. Thus, we first extracted some statistical features from queries such as length of query, sum and intersect of document frequencies of each query term etc. We develop a system to predict if a query is to be expanded or not. Then, we trained our system with previous year's data. Then, we applied a query expansion method only to the queries, which are selected by the system. The results show that the approach we proposed slightly improves our baseline retrieval performance in terms of P@10.
In this paper, we present the experiments performed by Dokuz Eylül University
Multimedia Information Retrieval (DEMIR) Research Group, in the context of our
participation to the ShARe/CLEF eHealth Evaluation Lab [
Rest of the paper is organized as follow: Section 2 provides an explanation of statistical features of query text and describes query classification process. In the next section 3 we present our experimental results. Section 4 concludes the paper by pointing out the open issues and possible avenues of further research for applying query expansion methods selectively in information retrieval system. 2
Task 3a contains approximately one million medical documents, which are collected different web sources. They are in eight-part folder and each document is taken formatted style in own source file which extension is dat. We processed these dat files and extracted all documents as a single file, which can be indexed and retrieved in IR system.
In original document collection, data structure contains HTML tags in content tag so when we processed them, we stripped out HTML tags and used title, heading and body information in it. We created a new content data and document data structure using these tags information. So we used this new data structure in IR system.
In this work, we tested effect of selective query expansion method in IR system
performance. As in standard IR system, we preprocess document collection and
indexed them. We used Terrier IR Platform API, which is an open source search engine
written in Java and is developed at the School of Computing Science, University of
Glasgow, to generate vector space model [
Before retrieval, we processed the queries and extracted statistics of query terms
from data collection [
We used all these statistical information as attributes for classification process of
queries. To classify the queries as to be expanded and not to be expanded, we used
Naïve Bayes method in WEKA machine learning software [
We performed the training and test sets in the following way. We retrieved two groups of result for 2013 data. We applied baseline method in one of them is described in section 3.2 and is shown as 1 in equation 1. In addition to the first group, only we used KL method as query expansion model in another one and it is shown as 2 in equation 1. This expansion model was better model than others, we experiment it for 2013 data and we explained it in section 3.1. During creation of training set, we calculated difference of map score in retrieval result between these two groups for each query.
_ _ _ = 2 − 1
We used _ _ _ value in equation 1 when we decided to query’s class. If this value was positive, we labelled query as positive and so we applied query expansion model in this query when we retrieved. If difference is zero or negative, we did not apply any model. We formulated this function in equation 2.
, ( ) = , _ _ _ _ > 0 ≤ 0
In 2013 data, we labelled 21 queries as positive, 29 queries as negative. It means that using query expansion improved retrieval performance in 21 queries. For evaluation of our labeling function is shown in Eq. 2, we used Naïve Bayes method as classifier in WEKA as test dataset, and applied 10-fold cross validation. Our classification accuracy is around 70% for 2013 queries. Likewise for 2014 data, we used 2013 queries as training set and 2014 queries as test set and performed a prediction with 2014 queries if a query is to be expanded or not. After classification, we predicted query’s class as positive or negative for query expansion. If the result is positive, we expanded it using KL expansion model in Terrier and did not expand the query if otherwise. (1) (2) 3 3.1
Experiments
In order to assess our proposal, we set up a set of experiments on the 2013 data collection of CLEF eHealth. In experiments, our aim was which weighting-model, query field and expansion method was used in runs.
Between run 1_1 and 1_3, we used title tag as query field and different weighting models such as TF×IDF, BM_25 and DFR_BM25, while retrieving. By the way, we did not use any expansion method. We obtained that TF×IDF is best among them. In run 2_1 and 2_2, we tried to find out which query fields should be used in retrieval. In data collection, each query has title, desc, profile and narr information. We used only title tag, because its performance is the best. In run 3_1 and 3_3, we aimed to choose which query expansion method to be used. In this test, we used only title tag, TF×IDF weighting-model and three different query expansion models such as Bo1, Bo2 and KL which are available in Terrier. We obtained the best result with KL method.
Table 1 shows our experimental results for last year data collection and queries. According to these results, we decided to use title tag as a query field, TF×IDF as a weighting-model, KL as a query expansion method in our submitted runs. • DEMIR_EN_Run.1: This run is our baseline retrieval result. In this run, title and content tag is used to index documents. Term-weighting model is TF×IDF. UTF tokenizer and stopword list were used and we applied porter stemmer. Query expansion model was not used. We used title field in query file for each topic when retrieved. We obtained the best result in CLEF eHealth 2013, using this method so we have it as a baseline for this year. • DEMIR_EN_Run.5: This run is exactly the same with baseline run except for all queries we used KL query expansion method available in Terrier. • DEMIR_EN_Run.6: In this run, we applied similar pre-process and indexing operations on documents like run 1 and 5. For query expansion model, we extracted term statistics such as QE, IntersectDF, MaxDF, MinDF, SumDF, AvgDF, MaxIDF, MinIDF and AvgTF from queries. We used them as attributes and applied Naïve Bayes classification method. As a result of classification, we expect to predict which query to expand. 27 queries are positive and other 23 queries are negative. We expanded positive queries using KL query expansion model, which were selected by classification process. We did not expanded negative queries. • DEMIR_EN_Run.7: In this run, we selected queries manually which were expanded. We called it as blind query expansion. We labelled 16 queries as positive and others were negative. We expanded positive queries.
In this year, we tried to classify queries as to be expanded or not. This is because there is no query expansion methods works for all type of queries. In other say, a query expansion method improves only a set of queries and worsens the rest. Thus, it would be great if we can predict which queries to be expanded. This is the basic idea of our study. Hence, we tried to classify or predict the queries to which an expansion method will work effectively and we apply it on them and expect to improve retrieval performance.
So in this work, we performed query expansion on a selected set of queries instead of the whole queries and expect a performance improvement. The results we obtained showed that the approach we proposed slightly improves our baseline retrieval performance in terms of P@10. It shows that it is promising and needs further studies on this topic.