We used learning-to-rank methods for training ranking model. Due to the limited number of training queries, we split them and conducted 5-fold cross validation. We set the proportion of training and testing as 4 : 6. For the features used in learning the model, a total of 231 features that are from multiple information retrieval models with di erent parameter settings were adopted. For the baseline run, we used Random Forest method to train the models with 5-fold cross validation. Only the binary relevance information were taken into account while training the model. Five trained models from 5-fold cross validation were used on the testing data to predict scores, and then we used equal weights to linearly combine the results given by di erent models. For run #5, we used 8 learning to rank methods to train models separately and linearly combine them together, and the binary relevance judgment was used as well. For run #6 and #7, graded relevance were taken into consideration. The di erence between run 6# and #7 is that for run #6, multiple learning-to-rank methods were used while for run #7, only Random Forest method was used. The best result of the four runs is achieved by run #5, which used multiple models combination based on binary relevance judgment.
These working notes serve to present the experimental method presented by
YorkU in the CLEF eHealth 2014 task 3a [
To achieve the main goal, we used supervised learning-to-rank method based on the provided ve training queries to train the models. Due to the limitedness of training dataset, we used various strategies to combine the trained models and tested on the testing set in order to get balanced results. 2
Learning-to-rank is a new type of method in information retrieval (IR), which
has been merged in the past decade. Di erent from traditional ranking models
in IR, learning-to-rank adopts machine learning approaches to solve the ranking
problem. Similarly to other machine learning methods, learning-to-rank methods
are based on features and in most cases are supervised methods, which means
labeled training data is required. One edge of this type of methods is that it saves
the pain for tuning parameters which is usually time consuming and tedious in
traditional IR models. In the previous study of medical IR, traditional IR models
were used extensively [
In this work, we used an in-house IR platform to do rst pass retrieval for the training dataset. Multiple retrieval models with di erent parameter settings were used to retrieve relevant documents for the training queries. Based on the qrels information (relevance judgment) for the ve training queries provided in the task dataset, and the retrieval results from rst pass retrieval, the candidate training documents were selected. Only those documents appearing in the qrels and has more than m non-zero scores from the n retrieval results were selected. Where n stands for the total number of retrieval models accounting same retrieval model with di erent parameter settings as di erent models. n in this work is 231, and m was chosen as 180.
We only participated in task 3a, which is a standard TREC-style IR task using (a) the 2012 crawl of approximately one million medical documents made available by the EU-FP7 Khresmoi project1 in plain text form which was used in CLEF eHealth 2013's Task 3 and (b) a new 2014 set of English general public queries that individuals may realistically pose based on the content of their discharge summaries. This collection contains documents covering a broad set of medical topics, and does not contain any patient information. The documents in the collection come from several online sources, including the Health On the Net organization certi ed websites, as well as well-known medical sites and databases (e.g. Genetics Home Reference, ClinicalTrial.gov, Diagnosia). Queries are generated from the discharge summaries used in Tasks 2. 3.2
Evaluation will focus on P@5, P@10, NDCG@5, NDCG@10, but other suitable
IR evaluation measures will also be computed for the submitted runs (eg. MAP).
P@N indicates the percentage of relevant documents within the top N results.
NDCG [
As the baseline run, only title and description in the query can be used and
no external resource (including discharge summary, corpora, ontology, etc) can
be used. To keep it simple, we used single learning-to-rank model based on the
binary relevance judgment to train our model. So far there are plenty of methods
in the literature, and we chose to use RankLib2, an open source learning-to-rank
package which implements eight popular algorithms: MART [
For this sake, we conducted ve-fold cross validation using all the eight algorithms on training dataset. Table 2 lists the cross validation setting, where the number represents the id of training queries.
The model that was used as baseline method is Random Forest, which achieved the best average result over the ve folds in terms of precision at 10. 5
As our run #5, multiple models binary relevance (MMBR) run trained models using all eight methods with the same ve-fold cross validation setting as baseline 1 http://www.khresmoi.eu/ 2 http://people.cs.umass.edu/ vdang/ranklib.html
Train Test 1,2 3,4,5 2,3 4,5,1 3,4 5,1,2 4,5 1,2,3 5,1 2,3,4 run and using the binary relevance of training data. The nal result on testing set is gained by combination of equal linear weights of all models. As run #6, multiple models graded relevance (MMGR) run trained models in the same way as MMBR, only di erent in that using graded relevance of training data. As run #7, single model graded relevance run trained model using Random Forest on ve-fold cross validation. For MMBR and MMGR, the combination of multiple models are required. Since the scores range given by di erent models vary, the combination was conducted on top of normalization of the scores. We rescaled ranking scores to the range of [0 1] using Formula 1,
X0 =
X
(1) where X0 is the normalized score, X the original score, Xmin and Xmax the minimal and maximal of the scores given by the model for that particular query.
The performance comparison of the four submitted runs is shown in Figure 1.
As is seen in Figure 1, the best result was achieved by run #5. Notice the di erence between baseline run and run #5 is that baseline run is based on single model, while run #5 uses combination of multiple models. This shows that the single model, even though it achieves the best result in training dataset, is not better than the linear equal weights combination of multiple models in the testing dataset.
Baseline run and run #5 are relatively better than the other two runs. The main di erence is that baseline run and run #5 are trained using binary relevance judgment, while run #6 and run #7 are trained using graded relevance judgment. This somewhat surprises us in that, the graded relevance provides more information to the learning-to-rank model about the ranking of documents, thus naturally should result in a better ranking model. But the result is contrary to this intuition. We attribute this to that the more relevance information confuses the learning models due to the shortage of training queries rather than being bene cial for the model learning.
The comparison of our best result (run #5) with the median results of all submitted runs is shown in Figure 2. It shows that, on approximate half of the queries, our best result is comparable to other systems. 7
In this paper, we describe our methods for medical document retrieval for task 3 in CLEF eHealth 2014. Based on supervised learning-to-rank methods, we have developed four strategies to conduct our experiments. The combination of multiple models using binary relevance judgment is more preferable than others. In the future, we plan to further research learning-to-rank in medical document retrieval, for example, 1) how domain speci c features could bene t the model training, 2) how could unlabeled data be assistant in building ranking model.
This research is supported by the research grant from the Natural Sciences & Engineering Research Council (NSERC) of Canada and the Early Researcher Award/Premier's Research Excellence Award.