In this paper, we present our previous and current work about the the methodology and the experimental analysis of a query reformulation, pseudo-relevance feedback, and document filtering approach. In particular, we present a summary of two studies carried out in the context of the TREC Precision Medicine track. The two original papers are [1] and [2]. The Clinical Trial Task1 at the Text REtrieval Conference (TREC)2 is an information retrieval challenge in the medical domain where the query is the a synthetic patient descriptions and the corpus is a large set of clinical trial descriptions. This task is part of a long run challenge related to the evaluation of Clinical Decision Support systems3 started in 2014. These tasks have sought to provide benchmark datasets and evaluate information retrieval systems focused on many of the most important information access problems in biomedicine. The dataset provided by the organizers of the task (in both 2021 and 2022) consists of a set of topics, a brief patient case description, and a set of documents, a snapshot of ClinicalTrials.gov. This collection consists of a snapshot of all the clinical trials available on ClinicalTrials.gov on April 27, 2021. The data is available as XML, with this specific snapshot containing 375,581 clinical trial descriptions. In this extended abstract, we summarize the methodologies and the experimental results that we achieved in the last two editions of the Clinical Trial Task, 2021 and 2022, where the main
focus was the retrieval of clinical trials given a lengthy query that describes the patient case that simulates an admission statement in an electronic health record.
Our participation to these two editions focused on the evaluation of a mixture of query
reformulation, rank fusion, and document filtering approaches optimized on the experimental
analyses of our previous participations to this track [
The methodology we employed in both participations was mainly based on a query reformulation approach, the merging of the ranking lists provided by the diferent retrieval methods using (or not) summarized queries and applying query expansion based on pseudo-relevance feedback. In the following sections, we describe the approach for each element of the retrieval pipeline.
In the 2021 edition, we use either BART [
We used the RM3 model to implement a pseudo-relevance feedback strategy including query
expansion [
For each query, we run the Okapi BM25 retrieval model [
After the retrieval step, we filter out from the list of candidate trials those for which a patient is not eligible based on their demographic data – that is, age and gender. In other words, we automatically extract patient’s age and gender from queries and filter out trials whose eligibility criteria do not allow for the extracted age and gender values. In those cases where part of the demographic data are not specified, a clinical trial is kept or discarded on the basis of the remaining demographic information. For instance, if the clinical trial does not specify a required minimum age, then it is kept or discarded based on its maximum age and gender required values.
Given diferent ranking lists, we used the CombSUM [
The organizers of the TREC tasks usually provide the summary of the results in terms of best, median, and worst value for each topic for three evaluation measures: Normalized Discount Cumulative Gain (NDCG), precision at 10 (P@10), and Reciprocal Rank (RecipRank). In the following sections, we summarize the details of each experiment and the results for each year. 3.1. Results of 2021 In 2021, we submitted five runs: • RM3Filtered: run with RM3 expansion, using BM25 as the first and second stage retrieval model. After both the first and the second retrieval stages, results have been filtered to remove trials with unfeasible age or sex attributes; • T5RM3Filt: Prior to the retrieval, queries are summarized using the T5 summarization algorithm with a summary length - chosen by T5 - between 30 and 130 words. The same model as RM3Filtered is used to retrieve documents; • BARTRM3Filt: Prior to the retrieval, queries are summarized using the BART summarization algorithm with a summary length - chosen by BART - between 30 and 130 words.
The same model as RM3Filtered is used to retrieve documents; • imsFused1: additive fusion of runs obtained with T5 summarizations with exact lengths 20, 50, 100, 150 and a run with T5 summarizations in the range 0-150. BM25 is used as the retrieval model. results with unfeasible values of age or sex have been removed; • imsFused2: CombSUM fusion with min-max normalization of imsFused1, RM3Filtered,
T5RM3Filt, and BARTRM3Filt;
In Table 2, we report the median values of the three measures averaged across topics, as well as the averaged results of the five submitted runs.
The results show that all the runs perform better than median values. In particular, the RM3 Filtered run performs significantly better than median (statistical analyses will be provided in the final version of the paper), followed by the imsFused2 run and the BART RM3 filtered rank. Given these promising results, we plan to investigate the integration of re-ranking components in the retrieval pipeline.
measure infNDCG P@10 RecipRank
In Table 2, we report the median values of the three measures averaged across topics, as well as the averaged results of the five submitted runs.
The results show that the runs have mixed performances compared with median values. Among the diferent approaches, those using keyword-based summaries seem to achieve higher performance. On the other hand, the impact of RM3 to expand queries is not clear, and might hinder the performance – as for the ims_RM3Filtered_s run. Given these mixed results, we plan to deepen the investigation on manual summarization to understand what is the proper tradeof between NLS and KS summaries.
In this extended abstract, we have presented a brief overview of the experimental results obtained in the last two editions of the TREC Clinical Trial task. The results of the proposed approach were well above the median results of the participants in both editions. For some evaluation measures, our results were included in the top 5 performing systems.