), with di erent elds of the topics (title, query). Our incrementally trained model is a support vector machine trained on a TF-IDF representation of title and abstract of the documents. The results of our approach are promising, reaching 0.658 normalized cumulative gain in the top 10 ranked documents in the simple evaluation setting and 0.846 in the cost-e ective evaluation setting, the latter assuming feedback can be obtained from an intermediate user/oracle instead of the end-user.
Evidence-Based Medicine (EBM) is an approach to medical practice that makes use of the current best clinical evidence in making decisions about the care and treatment of individual patients [13]. Researchers in the medical domain conduct systematic research to nd the best available evidence and form review articles summarizing their discoveries on a certain topic. These systematic reviews usually include three stages: 1. Document retrieval. Experts build a Boolean query and submit it to a medical database, which returns a set of possibly relevant documents. Boolean queries typically have very complicated syntax and consist of multiple lines. Such a query can be found for reference in Listing 1.1. 2. Title and abstract screening. Experts go through the title and abstract of the set of documents retrieved by the previous stage and perform a rst level of screening. 3. Document screening. Experts go through the full text of each document that passes the screening of the previous stage to decide whether it will be included in their systematic review.
Considering the rapid pace with which libraries of medical articles are expanding, Systematic Review can be a very di cult and time-consuming task.
Task II [
Listing 1.1. Example of query in data set
It is the rst time this task take place and very little research is previously done on the topic. Previous approaches on this problem use an ensemble of Support Vector Machines (SVM), built over di erent feature spaces (documents' titles, text, etc.). [15] Other approaches use Active Learning techniques to improve results' relevance by utilizing domain experts' knowledge. [14] Finally, Learning to Rank (LTR) approaches have also been tested on biomedical data and have shown promising results. [12]
Our approaches on the task are based on binary classi cation methods combined with existing Learning to Rank techniques. We experimented with di erent classi ers and we also introduce a hybrid classi cation mechanism which consists of two parts: an inter-topic classi er, based on features computed on the training set and an intra-topic classi er, which is trained upon the test set documents. 2
In CLEF eHealth 2017 Task II, participants were given a total of 20 topics with the corresponding document IDs. An example of such topic can be nd in Listing 1.1. Summarizing the topics' structure, they all contain:
2. A topic title, 3. An Ovid MEDLINE query and 4. a set of documents' PIDs that are returned from the query.
2. A title, 3. The abstract text and 4. Mesh headings, based on their taxonomy The test set comprised of topics in similar structure, summing up to a total of 30 topics.
For both the training and the test set, participants were also provided with the corresponding document relevance sheet, in which relevance was provided in the format shown in Listing 1.2, where 0 denotes negative relevance and 1 denotes positive one.
Listing 1.2. Example of query/document relevance. 1 CD010438 2 CD010438 3 CD010438 4 CD010438 5 CD010438 6 CD010438 7 ... 8 9 CD011984 10 CD011984 11 CD011984 12 ... 1. Area under the recall-precision curve, i.e. Average Precision (metric in task's evaluation script: ap) 2. Minimum number of documents returned to retrieve all R relevant documents (metric in task's evaluation script: last rel) a measure for optimistic thresholding 3. Work Saved over Sampling @ Recall (metric in task's evaluation script: wss 100, and wss 95)
T N + F N N {(1 Recall) 4. Area under the cumulative recall curve normalized by the optimal area (metric in task's evaluation script: norm area) 5. Normalized cumulative gain @ 0% to 100% of documents shown (metric in task's evaluation script: NCG@0 to NCG@100) 6. Total cost uniform (metric in task's evaluation script: total cost uniform) optimal area = R
N {
R2 2 m R (N n)
Cp where: { N is the total number of documents in the collection { n is the number of documents shown to the user { (N n) is the number of documents not shown to the user { m is the number of missing relevant documents { Ca is the cost paid for experts/users reviewing returned documents' abstracts to determine their relevance, and { Cp = 2 Ca 7. Total cost weighted (metric in task's evaluation script: total cost weighted) Xm 1
2i (N i=1 n)
Cp
8. Reliability (metric in task's evaluation script: loss er) [
Reliability = lossr + losse where { lossr = (1
recall)2 (metric in task's evaluation script: loss r)
{ losse = R+n100 1N00 )2 (metric in task's evaluation script: loss e)
{ recall = nRr (metric in task's evaluation script: r) and
{ nr is the number of relevant document found and R the total number of
relevant documents
The architecture of our approach, which comprises two models, is depicted in
Figures 1 to 3. The rst model is a learning-to-rank binary classi er that
considers a topic-document pair as input and whether the document is relevant for the
topic or not as output (Figure 1). This inter-topic model is used at a rst stage
of our approach in order to obtain an initial ranking of all documents returned
by the Boolean query of an unseen test topic. The second model is a standard
binary classi er that considers a document of the given test topic as input and
whether this document is relevant to the test topic as output. This intra-topic
model is incrementally trained based on relevance feedback that it requests after
returning one or more documents to the user. The rst version of this model
is trained based on feedback obtained from the top k ranked documents by the
inter-topic model (Figure 2). The re-ranking of subsequent documents is from
then on based solely on the intra-topic model (Figure 3).
For each htopic; documenti pair, we extracted a number of features, following
the paradigm of [12]. The majority of the features were computed by considering
the similarity of di erent elds of the document (title, abstract), with di erent
elds of the topic (title, query), using a variety of similarity metrics, such as the
number of common terms between the topic and the document parts, Levenshtein
distance, cosine similarity or OKAPI BM25 [
In order to use the rich information available in the query eld of the topics, we used Polyglot1, a JavaScript tool that can parse and produce a full syntactical 1 https://github.com/CREBP/sra-polyglot tree of Ovid MEDLINE queries. In particular, we extracted those medical subject headings (MeSH) that should characterize the retrieved documents, avoiding the ones that are negated in the query syntax. As an example, according to Polyglot, the MeSH terms found in the Ovid MEDLINE query of Listing 1.1 are the following:
We eventually settled to the 24 features that can be found in Table 1, after extensive investigation of the performance of our model with additional variations of these features. Two of these features are only topic-dependent, denoted with T in the Category column of Table 1, as opposed to the rest 22 of the features that dependent on both the topic and the document, denoted with T D. The notation used in the Description column of Table 1 is explained here: { t represents the title of each topic, consisting of tokens ti. { m represent the MeSH terms extracted from the query of each topic. { d represents the title or abstract of a document, consisting of jdj tokens dj . { c(x; d) denotes the number of occurrences of title token or MeSH term x of the topic in document d.
We have experimented with a variety of di erent classi ers, including
Support Vector Machines [
Intra-topic model The rst version of the intra-topic model is trained based on the top k documents as ranked by the inter-topic model. We then iteratively re-rank the rest of the documents, expanding the training set of the intra-topic model with the topranked document, until the whole list has been added to the training set or a certain threshold is reached. This iterative feedback and reranking mechanism is described in detail in Algorithm 1. For the local classi er, a standard TF-IDF vectorization was used, enhanced with English stop words removal. 4
Task II of CLEF eHealth 2017 supported two experimental setups: one for simple evaluation and one for cost e ective.
In the simple evaluation, our aim was to utilize relevance feedback as much as possible without any cap or limitation, so as to experiment with di erent techniques for boosting ranking metrics. In the cost-e ective evaluation, we have implemented thresholding by limiting the amount of documents (column Threshold in Table 2) that we request feedback for and by not showing to users documents for which negative relevance was received.
In Table 3 you can nd the o cial results for the simple evaluation setup. In Table 4 you can nd the results for the cost e ective evaluation, as they derive from the evaluation script provided by the task's organizers. Please note, that because of undergoing software enhancements in the script some metrics in the cost-e ective evaluation might be inaccurate, e.g. the total cost metrics, as they have not be adjusted to di erent run outputs for that setup. Each of the runs have a parameterized version of HybridRankSVM and thresholding points, which are listed in Table 2. 4 k0 k; 5 while not f inalRanking contains both relevant and irrelevant documents do 6 k0 k0 + 1; 7 f inalRankingk0 = Rk0 ; 8 while not length(f inalRanking) == n OR length(f inalRanking) == tfinal do 9 train(f inalRanking) ; // Train a local classifier by asking for abstract or document relevance for these documents localRanking = rerank(R f inalRanking) ; // Rerank the rest of the initial list R from the predictions of the local classifier if length(f inalRanking) < tstep then
step = stepinit; 10
In conclusion, in this paper we introduced a hybrid classi cation approach for medical document ranking. Our approach constructs a global classi cation model based on LTR features of the training documents, produces an initial ranking for the test documents and then iteratively asks for feedback and rerank them based on the acquired relevance.
As future work, we believe that experimentation with more features, such as
semantic representations (e.g. word2vec [11], LDA [
This work has been partially funded by Atypon Systems Inc. 10. Liu, X.Y., Wu, J., Zhou, Z.H.: Exploratory under-sampling for class-imbalance learning. In: Proceedings - IEEE International Conference on Data Mining, ICDM. pp. 965{969 (2006) 11. Mikolov, T., Chen, K., Corrado, G., Dean, J.: E cient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013) 12. Qin, T., Liu, T.Y., Xu, J., Li, H.: LETOR: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval 13(4), 346{374 (2010) 13. Sackett, D.L.: Evidence-based medicine. In: Seminars in perinatology. vol. 21, pp.
3{5. Elsevier (1997) 14. Wallace, B.C., Small, K., Brodley, C.E., Trikalinos, T.A.: Active learning for biomedical citation screening. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 173{182. ACM (2010) 15. Wallace, B.C., Trikalinos, T.A., Lau, J., Brodley, C., Schmid, C.H.: Semiautomated screening of biomedical citations for systematic reviews. BMC bioinformatics 11(1), 55 (2010)