This paper describes the participation of Technical University of Vienna (TUW)
at CLEF eHealth 2017 Task 3 [
In 2016, fifty topics were extracted from real user posts/interactions in the AskDocs section of Reddit1. Each topic was presented to six query creators with different medical expertise. Their job was to read a post (usually with a medical question) and formulate a query using their medical background knowledge, if any. In total 300 queries were created.
This year, this track has 4 subtasks (named IRTasks, see [
The experiments conducted by TUW aim to investigate two research directions: 1. IRTask 1 : Can understandability metrics be used to improve retrieval? 2. IRTask 2 : How to personalize retrieval in a learning to rank setting, according to different reading profiles and user expertise?
For IRTask 1, a previous study conducted in the context of CLEF eHealth
2014 and 2015 ([
In this section we describe our learning to rank approach, the feature set devised, and our functions to map topical and understandability assessments into a single relevance label. 2.1
Learning to Rank Our learning to rank approach is based on 3 items: (1) a set of features, (2) a set of <document, label> pairs, and (3) a learning to rank algorithm. The set of features is described in Section 2.2. We consider in this work three different functions to label documents: for Subtask 1, we only use the pure topical relevance as judged in 2016; for Subtask 2, we define two understandability-biased function (named boost and float). Given a document with topical relevance T and understandability score U , and an user with a reading goal G, we define boost and float as: boost(T, U ) = (2 ∗ T
T if |G − U | ≤ 0.2 if |G − U | > 0.2 f loat(T, U ) = T ∗ (1.0 − |G − U |) (1) (2)
As topical relevance scores are either 0, 1 or 2, and the understandability
scores are float numbers from 0.0 to 1.0, the possible values for function boost
are the integers 4, 2, 1 and 0, while the possible values for function float are any
float precision number between 0.0 and 2.0. All experiments used the pairwise
learning to rank algorithm based on gradient boosting implemented in XGboost2
with NDCG@20 as goal to be optimized. Differently from past work [
Features
We devised 91 features from 3 distinct groups: information retrieval traditional
features, understandability related features, and the modified output of
regression algorithms made to estimate the understandability of a document.
Elaborated features based on recent advance on semantic similarity, as made in [
Feature Category Feature Name
BM25
PL2 Common IR Models (7) LTDeFirmicIuDhrlTFetFLMIDF
DFRee
Hiemstra LM Query Independ. (3) DDDooocccuuummmeeennnttt SLPpeanagmgetRhSacnokres Doc. Score Modifier (2) DMiavrekrogvenRceanfrdoommRFaienlddomness
ARI Index Coleman Liau Index
Dale-Chall Score Traditional Formulas (8) FFlleesscchh RKeinacdainidgGEraasdee
Gunning Fog Index LIX Index SMOG Index # Characters ♦† # Sentences ♦ # Syllables ♦† Surface Measures (25) ## (W| oSrydllsa†bles(Word) | > 3) ♦† # (| Word | > 4) ♦† # (| Word | > 6) ♦† # (| Word | > 10) ♦† # (| Word | > 13) ♦† RGeleanteerdalFVeaotcuarbeusla(1ry2) sENDtnuaoglpmelw-ibsCoherhrdsDasl♦ilc♦†Lt†iiosnta♦r†y ♦†
Acronyms ♦† Mesh ♦†
DrugBank ♦† RMeleadtiecdalFVeaotcuarbeusla(2ry7) IMMCeeDdd1iicc0aall(ISPnurteeffifirnxxeaestsi♦o♦n††al classification of Diseases) ♦†
Consumer Health Vocabulary ♦† Sum(chv Score) ♦† Mean(chv Score) ♦† Ada Boosting Regressor
Extra Tree Regressor Modified Regression Scores (7) KG-rNadeiaernetstBNooesigtihnbgorRRegergersessosror
Linear Regression
Support Vector Machine Regressor
Random Forest Regressor
IR Features: Regularly used information retrieval features are considered in
this work. This list includes many commonly used retrieval models and document
specific values, such as Spam scores[
Understandability Features: All HTML pages were preprocessed with
Boilerpipe4 to remove the undesirable boilerplate content as suggested in [
Regression Features: We adapted the output of regression algorithms to create personalized features. The 2016’s judgements were used as labels for a num3 http://www.lemurproject.org/clueweb12/PageRank.php 4 https://pypi.python.org/pypi/boilerpipe ber of regression algorithms (the list of algorithms used is shown in Table 1). Models were trained on a Latent Semantic Analysis (LSA) applied on words from 3.549 documents marked as topical relevant in the QRels from 2016, which understandability label varied from 0 (easy to understand) to 100 (hard to understand). LSA dimensions vary from 40 to 240 according to the best result of a 10-fold cross validation experiment. In order to avoid interference from the training set in the learning to rank algorithm, scores for the documents in the training set were predicted also in a 10-fold cross validation fashion. The personalization step consisted in calculating the absolute difference between the estimated score and the goal score, which is defined by user. For example, if the score estimated by a regression algorithm for a document D was 0.45 and the reading goal of a user U was 0.80, we used as feature the value 0.35 (the absolute difference between 80 and 45). We want to evaluate if features like these ones can help the learning to rank model to adapt according to the reading skills of a user. 3 3.1
Evaluation Metrics We consider a large number of evaluation metrics in this work. As topical relevance centred evaluation metrics, we consider Precision at 10 (P@10) and Rank Biased Precision with μ parameter set to 0.8 (RBP(0.8)). Due to the fact that a learning to rank algorithm has the potential to bring many unjudged documents to the top of the ranking list, we consider also a modified version of P@10, Only Judged P@10, which will calculate P@10 considering only the first 10 judged documents of each topic.
As modified metrics that take into account understandability scores, we
consider understandability-biased Rank Biased Precision, also with μ parameter
set to 0.8 (uRBP(0.8)) as proposed by [
The first personalization-aware metric is a specialization of uRBP, auRBP, which takes advantage of an α parameter to model the kind of documents a user wants to read. We assume that α is a parameter that models understandability profiles of an entity. A low α is assigned to items/documents/users that are experts, while a high α are the opposite. We assume that a user with a low α wants to read specialized documents to the detriment of easy and introductory documents, while laypeople want the opposite. We model in auRBP a penalty for the case in which a low α document is presented to a user that wants high α documents and vice versa. While we are still investigating which one is the best function to model this penalty, we assume a normal penalty. Figure 1 shows an example in which a user is looking forward to reading documents with α=20 and other values for α would have a penalty associated to them according to this normal curve with mean 20 and standard deviation of 30. We use the standard deviation of 30 in all of our experiments, but it is left as future work ways to estimate a right value for it.
The second and third personalization-aware metrics are simple modifications of Precision at depth X. For the relevant documents found in the top X, we inspect how far is the understandability label of each document to the expected value required by a user. We could penalize the absolute difference linearly (LinUndP@X) or using the same Gaussian curve as in auRBP (GaussianUndP@10). Note that lower values are better for LinUndP@10, meaning that the distance from the required understandability value is small, and higher values are better for GaussianUndP@10, as a value of 100 is the best value one could reach. 100 80 60 40 20 0 0 20 40 60 80 100 Seven runs were submitted to IRTasks 1 and another seven were submitted to IRTask 2. Tables 2 and 3 present a summary of each approach, submissions for IRTask 1 and 2, respectively, and the results using 2016 Qrels. 4
As shown in Table 2 and 3, we based our runs on the BM25 implementation from a Terrier 4.2 index of ClueWeb 12-B. The results of using relevance feedback are high because the judged as relevant documents appear at the top of the ranking list of each topic, but it does not necessarily means that these approach will be much better than a plain BM25 for 2017, as the already judged documents will be discarded by the organizers.
Results on CLEF eHealth 2016 QRels
Results on CLEF eHealth 2016 QRels 17.34 25.76 that higher RBP(0.8) are followed by higher uRBP(0.8) and auRBP(0.8), while This means that our efforts to retrieve more topical relevant documents also increases uRBP and auRBP, but does not affect LinUnd. and GausianUnd. metrics. better) and 56.96 for GausianUnd.P@10 (the higher the better).
We are looking forward to evaluating our results with 2017 QRels, but as the 2017’s assessments are still being conducted, an analysis of the official results will be posted online at https://github.com/joaopalotti/tuw_at_ clef_ehealth_2017. CLEF eHealth Evaluation Lab 2016: User-centred Health Information Retrieval. In CLEF 2016 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS, September 2016.