The Precision Medicine (PM) track of the Text REtrieval Conference (TREC) focuses on providing useful precision medicine information to clinicians treating cancer patients. The PM track gives the unique opportunity to evaluate medical IR systems on two di erent collections: scienti c literature and clinical trials. In this paper, we evaluate several state-of-the-art query expansion and reduction methods to see whether a particular approach can be helpful in clinical trials retrieval. We present those approaches that are consistently e ective in all three TREC PM editions and we compare them to the results obtained by the research groups who participated in all three editions.
Medical Information Retrieval (IR) helps a wide variety of users to access and
search medical information archives and data [
This work combines and discusses the methodology and the results originally
presented at SIGIR 2019 [
The rest of the paper is organized as follows: Section 2 describes the approach used to evaluate di erent query reformulation methods. Section 3 presents the experimental setup and Section 4 compares the results obtained using our approach with those obtained by the other research groups who participated in TREC PM 2017, 2018 and 2019. Finally, Section 5 reports some nal remarks. 2
The approach we propose consists of four steps: (i) indexing, (ii) query reformulation, (iii) retrieval and (iv) ltering.
Indexing. We create the following elds to index clinical trials collections: <docid>, <text>, <max_age>, <min_age> and <gender>. Fields <max_age>, <min_age> and <gender> contain information extracted from the eligibility section of clinical trials and are used in the ltering step. The <text> eld contains the entire content of each clinical trial.
Query Reformulation. The approach relies on two types of query reformulation methods: query expansion and query reduction.
Query expansion: We perform a knowledge-based a priori query expansion.
First, we rely on MetaMap [
Query reduction: We reduce original queries by removing, whenever present,
gene mutations from the <gene> eld. To clarify: consider a topic where the
<gene> eld mentions \BRAF (V600E)". After reduction, the <gene> eld
becomes \BRAF". The reduction aims at mitigating the over-speci city of topics,
since the information contained in a topic is too speci c compared to those
contained in the target documents [
Retrieval. We use BM25 [
Filtering. The eligibility section in clinical trials comprises three important demographic aspects that a patient needs to satisfy to be considered eligible for
the trial, namely: minimum age, maximum age and gender; where minimum age and maximum age are the minimum and the maximum age, respectively, required for a patient to be considered eligible for the trial, while gender is the required gender. After the retrieval step, we lter out from the list of candidate trials those for which a patient is not eligible { i.e. his/her demographic data (age and gender) does not satisfy the three aforementioned eligibility criteria. In those cases where part of the demographic data is not speci ed, a clinical trial is kept or discarded on the basis of the remaining demographic information. For instance, if the clinical trial does not specify a required minimum age, then it is kept or discarded based on its maximum age and gender required values. 3
This section describes the experimental collections and the setup used to apply and evaluate our approach.
Experimental Collections. We report the main information related to topics and document collections below.
Topics consist of 30, 50, and 40 synthetic cases created by precision oncologists in 2017, 2018, and 2019, respectively. In 2017, each topic contained four key elements in a semi-structured format: (1) disease (e.g. a type of cancer), (2) genetic variants (primarily present in tumors), (3) demographic information (e.g. age, gender), and (4) other factors (which could impact certain treatment options). In 2018 and 2019, each topic had three of the four key elements used in 2017: (1) disease, (2) genetic variants, and (3) demographic information. Furthermore, the 2019 topics contain ten non-cancer-related topics.
Clinical Trials consist of a set of 241,006 clinical trial descriptions, for both 2017 and 2018, and of an updated version of 306,238 descriptions for 2019. The collections are derived from ClinicalTrials.gov5 { a database of privately and publicly funded clinical studies conducted around the world. When none of the available treatments are e ective on oncology patients, the common recourse is to determine if any potential treatments are undergoing evaluation in a clinical trial. Therefore, it would be helpful to automatically identify the most relevant clinical trials for an individual patient. Precision oncology trials typically use a certain treatment for a certain disease with a speci c genetic variant. Such trials can have complex inclusion and/or exclusion criteria that are challenging to match with automated systems.
Experimental Procedure. We use Whoosh,6 a Python search engine library,
for indexing, retrieval, and ltering. For BM25, we keep the default values k1 =
1:2 and b = 0:75 provided { as we found them to be a good combination [
{ Indexing
Index clinical trials using the following created elds: <docid>, <text>, <max_age>, <min_age> and <gender>. { Query reformulation
Use MetaMap to extract from each query eld the UMLS concepts restricted to the following semantic types: neop for <disease>, gngm/comd for <gene>, and all for <other>; Extract from concepts all name variants belonging to NCI, MeSH, SNOMED CT and MTH knowledge sources; Expand (or not) topics that do not mention \lymphoma" or \leukemia" with the term solid ; Reduce (or not) queries by removing, whenever present, gene mutations from the <gene> eld;
Remove (or not) the <other> eld. { Retrieval 2017/2018 PM track: Adopt any combination of the reformulation strategies; 2019 PM track: Adopt the three best reformulation strategies from 2017/2018 PM tracks; Weigh expanded terms with a value k = 0:1;
Perform a search using expanded queries with BM25. { Filtering
Filter out clinical trials for which the patient is not eligible.
Evaluation Measures. We use the measures adopted in the TREC PM tracks, that are: inferred nDCG (infNDCG), R-precision (Rprec) and P@10. 4
In Table 1, we report the results of our experiments on query reformulation (Part A) and compare them with the results obtained by the research groups that participated at TREC PM 2017, 2018 and 2019 (Part B). Given the large number of experiments we performed, we decided to only present the 5 runs with the highest P@10 for each year. Each line shows a particular combination (yes or no values) of semantic types (neop, comd, gngm), usage and expansion of <other> eld (oth, oth exp), query reduction (orig), and expansion using weighted solid (tumor) keyword. We use the symbol ` ' to indicate that the features oth, oth exp are not applicable for the years 2018 and 2019 due to the absence of the <other> eld in 2018 and 2019 topics. We highlight in bold the top 3 scores for each measure, and we use the symbol z to indicate the combination that performs well in all three years. For the TREC PM participants, we select those participants who submitted runs in all three years and reached the top 10 performing runs in at least one edition for each measure [11{13]. The results achieved using the ve most e ective query reformulations for each year. (z) indicates a particular query reformulation e ective in all three years. Part B (bottom) reports the results obtained by participant runs, along with the lowest score required to enter the top 10 TREC results list and the score obtained by the best combination of our approach. Further details are reported in Section 4. reported in part B of Table 1 indicate the best score obtained by a particular run for a speci c measure; the best results of a participant are often related to di erent runs. The symbol ` ' means that the measure is not available, while `<' indicates that none of the runs submitted by the participant achieved the top 10 performing runs. For the sake of comparison, we add for each measure the lowest score required to enter the top 10 TREC results list, and the score obtained by the best combination of our approach { indicated by the line number { as if we were participants of these tracks. Analysis of Query Reformulations. The results from Table 1 (Part A) highlight that the use of solid expansions with weight 0.1 as well as query gene reductions (orig = n) seems to improve performance consistently in 2017 { two of the three best runs in terms of P@10 (lines 1 and 2) applying both techniques. Regarding knowledge-based expansions, the semantic type gngm (lines 1 and 5) seems more e ective than neop (line 3), while comd does not seem to have any positive e ect at all. All ve runs do not consider the other eld (oth = n) nor its expansion (oth exp = n) { con rming our intuition that it might represent a potential source of noise in retrieving precise information for patients. Similarly to 2017, two of the best three runs of 2018 use no knowledge-based expansions and rely on the solid (tumor) expansion with weight 0.1 (lines 7 and 9). In particular, the runs combining query gene reductions and solid expansions (marked as z) provide the best performances for all the measures considered, both in 2017 and 2018. This suggests that removing highly specialized information (i.e. the gene mutation) or adding general terms (e.g. solid) bene ts the retrieval. A possible reason is related to the nature of the document collections, since clinical trials often contain general requirements to allow patients to enroll. The results obtained in 2019 with the top three query reformulations from 2017 and 2018 con rm this trend. The run combining query gene reductions and solid expansions (line 13z) is one of the top three runs of 2019, however two query reformulations from 2017 (line 14) and 2018 (line 11) provide better performance. This result shows how di cult the task is. In fact, even though we found a particular query reformulation approach (marked as z) to be highly e ective in all three years { especially in 2017 and 2018 { it was not the best approach for 2019. Therefore, this analysis helps researchers to select an e ective subset of query reformulations to build strong baselines for clinical trials retrieval. Comparison with TREC PM Participants. The results from Table 1 (Part B) mark a clear division between the 2017 and 2018 tracks and the 2019 track. In 2017 and 2018, most of the participant runs did not reach the top 10 threshold in any of the considered measures { the only exception is the research group from Poznan University of Technology, whose best runs always belong to the top 10 performing runs for the track. Conversely, in 2019 all the participant groups submitted runs that achieved results higher than the top 10 threshold.
Compared with the results obtained using the query reformulations from Table 1 (Part A), we can see that all runs employing the best query reformulations obtain results higher than the top 10 threshold for all the considered measures in all three years. Furthermore, the runs using the top ve query reformulations achieved consistently better results than participant runs for each measure in all three years. This is an indication of the robustness of our approach across the di erent collections and also of the e ectiveness of the proposed query reformulations for the clinical trials retrieval. In particular, it is worth to mention that the runs using the (z) query reformulation achieve performances that belong to the top three best performing systems of each year PM track [11{13]. Therefore, the analysis of query reformulations made on the 2017 and 2018 PM tracks conrmed its trend in the 2019 PM track and allowed us to identify a speci c set of query reformulations bene cial for the retrieval of clinical trials. 5
In this paper, we further elaborate the results originally presented at SIGIR
2019 [