This paper details materials, methods, results, and analyses of the Consumer Health Search Task of the CLEF eHealth 2021 Evaluation Lab. This task investigates the efectiveness of information retrieval (IR) approaches in providing access to medical information to laypeople. For this a TREC-style evaluation methodology was applied: a shared collection of documents and queries is distributed, participants' runs received, relevance assessments generated, and participants' submissions evaluated. The task generated a new representative web corpus including web pages acquired from a 2021 CommonCrawl and social media content from Twitter and Reddit, along with a new collection of 55 manually generated layperson medical queries and their respective credibility, understandability, and topicality assessments for returned documents. This year's task focused on three subtask: (i) ad-hoc IR, (ii) weakly supervised IR, and (iii) document credibility prediction. In total, 15 runs were submitted to the three subtasks: eight addressed the ad-hoc IR task, three the weakly supervised IR challenge, and 4 the document credibility prediction challenge. As in previous years, the organizers have made data and tools associated with the task available for future research and development.
In today’s information overloaded society, using a web search engine to find information
related to health and medicine that is credible, easy to understand, and relevant to a given
information need is increasingly dificult, thereby hindering patient and public involvement in
healthcare [
In 2021, CLEF eHealth initiative has organised a CHS task with the following three IR subtasks: 1. Adhoc IR, 2. Weakly Supervised IR, and 3. Document Credibility Prediction.
The task has challenged researchers, scientists, engineers, analysts, and graduate students to develop better IR systems to support creating web-based search tools that return webpages that are better suited to laypeople’s information needs from perspectives of 1. information topicality (i.e., how relevant are the contents to the search topic), 2. information understandability (i.e., how easily can a layperson understand the contents), and 3. information credibility (i.e., should the contents be trusted).
As a continuation of the previous CLEF eHealth IR tasks that ran in 2013–2018, and 2020 [
The main contributions of the CLEF eHealth 2021 task on CHS are as follows: 1. generating a novel representative web corpus, 2. collecting layperson medical queries, 3. attracting new submissions from participants, 4. contributing to IR evaluation metrics relevant to the three dimensions of document relevance, and 5. evaluating IR systems (i.e., runs submitted by the task participants or from organizers’ baseline systems) on newly conducted assessments.
The remainder of this paper is structured as follows: First, Section 2 details the task. Then, Section 3 introduces the document collection, topics, baselines, pooling strategy, and evaluation metrics. After this, Section 4 presents the participants and their approaches while Section 5 addresses their results. Finally, Section 6 concludes the paper.
In this section, we provide a description of the three subtasks ofered in this year’s CHS task, namely: Subtask 1 on ad-hoc IR; Subtask 2 on weakly supervised IR; and Subtask 3 on document credibility prediction.
Similar to previous years of the CHS task, this was a standard ad-hoc IR task. A document collection was provided to task participants along with realistic layperson medical information need use cases, both described in Section 3. The purpose of the task was to evaluate IR systems’ abilities to provide users with credible, understandable, and topical documents. As such, participating teams submitted their runs, which were pooled together with baseline runs and manual relevance assessments conducted, also described in Section 3. These systems’ performance was assessed on multiple dimensions of relevance — credibility, understandability, and topicality. Evaluation metrics employed for this task are described in Section 3.6.
This task aimed to evaluate the ability of machine learning-based ad-hoc IR models, trained with weak supervision, to retrieve relevant documents in the health domain. In order to train neural models to address the search task, a large collection of real-world health related queries extracted from commercial search engine query logs and synthetic (weak) relevance scores computed with a competitive IR system were considered. The submissions were evaluated against the same test set as Subtask 1 submissions, in order to allow a full comparison of traditional vs neural approaches. Details on the methods and materials associated with this task are described in Section 3.
The purpose of this task was the automatic assessment of the credibility of information that is disseminated online, through the Web and social media. Using the dataset related to Subtask 1, this task aimed at comparing approaches estimating documents credibility. The ground truth for this classification task is based on the credibility labels assigned to documents during the relevance assessment process. Evaluation of the runs includes classical classification metrics and investigates the ability of the participants systems to perform well on both web documents and social media content.
In this section, we will describe the materials and methods used in the CHS task of the CLEF eHealth evaluation lab 2021. After introducing our new document collection and novel topics, we will describe our baseline systems and pooling methodology. Finally, we will address our relevance assessments and evaluation metrics.
The 2021 CLEF eHealth Consumer Health Search document collection consisted of two separate crawls of documents: web documents acquired from the CommonCrawl and social media documents composed by Reddit and Twitter submissions.
First, we acquired web pages from the CommonCrawl. We extracted an initial list of websites
from the 2018 CHS task of CLEF eHealth. This list was built by submitting a set of medical
queries to the Microsoft Bing Application Programming Interfaces (through the Azure Cognitive
Services) repeatedly over a period of a few weeks, and acquiring the uniform resource Locators
(URL) of the retrieved results [
Second, we complemented the collection with social media documents from Reddit and Twitter. As the first step, we selected a list of 150 health topics related to various health conditions. Then, we generated search queries manually from those topics and submitted them to Reddit to retrieve posts and comments. After this, we applied the same process on Twitter to get related tweets from the platform.
Please note that for the purposes of the 2021 CHS task, we defined a social media document as a text obtained by a single interaction. This implied that • on Reddit, a document is composed by a post, one comment of the post, and associated meta-information and • on Twitter, a document is a single tweet with its associated meta-information.
The set of topics of CLEF eHealth IR task aimed at being representative of laypersons’ medical information needs in various scenarios. This year, the set of topics was collected from two sources as follows: • The first part was based on our insights from consulting laypeople with lived experience of multiple sclerosis (MS) or diabetes to motivate, validate, and refine the search scenarios; we, as experts in IR and CHS tasks, captured these layperson-informed insights as the scenarios. • The second part was based on use cases from Reddit health forums; we extracted and manually selected a list of topics from Google trends to best fit each use case.
As a result, we had a new set of 55 queries in English for authoring realistic search scenarios. To describe the scenarios, we enriched each query manually by labels in English either to characterize the search intent (for manually created queries) or to capture the submission text (for social medial queries) (Table 3.2).
126 best apps daily activ- I’m a 15 year old with diabetes. I’m planning to join the school ity exercise diabetes hiking club. What are the best apps to track my daily activity and exercises? multiple sclerosis I read that MS develops with several stages and includes phases stages phases of relapse. I want to know more about how this disease develops over time. birth control sup- So I’ve been on birth control taking only active pills for months pression antral now and I went in to a doctor’s ofice to inquire about egg freezfollicle count ing. She seemed optimistic that my ultrasound would be very reassuring but instead she came away from the ultrasound deeply concerned. I did an AMH test and it came back 0.3 which was consistent with what was seen in the ultrasound. I’m terrified.
Like...suicidal terrified. I don’t have any underlying conditions and while I’m not young I’m not old either. I don’t smoke and I’m not terribly overweight. I don’t have insulin resistance and my reproductive organs looks good other than my ovaries. I’d been taking the combo pill to skip periods for months now. I don’t even take the brown pills. When I first heard that you can skip periods and only have 2-3 a year when taking brown pills I started on them. The first time I tried to have a period on the brown pills nothing happened. No period. This was 2 years ago. I freaked out but they said I just respond strongly to the pills. So I went of of the pills completely and had 2-3 normal periods starting about 2 months later. Could the birth control be suppressing my antral follicle count and amh this much? Please help.
Subtasks 1, 2, and 3 used these 55 queries with 5 released for training and 50 reserved for testing; the test topics contained a balanced sample of the manually constructed and automatically extracted search scenarios.
With respect to both Subtask 1 and Subtask 2, we provided six baseline systems. These systems
applied the Okapi BM25 (BM for Best Matching), Dirichlet Language Model (DirichletLM),
and Term Frequency times Inverse Document Frequency (TF× IDF) algorithms with default
parameters. Each of them was implemented without and with pseudo relevance feedback (RF)
using default parameters (DFR Bo1 model [
Regarding Subtask 3, where it is necessary to evaluate the efectiveness of the approaches with respect to assessing the credibility of information, it was decided to approach the problem as a binary classification (identification of credible versus non-credible information). For this reason, simple baselines were developed based on supervised classifiers that act on a set of features that can be extracted from the documents under consideration. Dealing with documents that are both Web pages and social content, it was necessary to consider, in baseline development, only those features that are directly extractable from the text of the documents. For social content, it would also be possible to consider other metadata related to the social network of the authors of the posts, but for consistency with the evaluation of Web pages, such metadata have not been considered.
The employed supervised classifiers are based on Support Vector Machines (SVM), Random
Forests (RF), and Logistic Regression (LR), i.e., the machine learning techniques that have proven
to be more efective for binary credibility assessment in the literature [
Similar to the 2016, 2017, 2018, and 2020 pools, we created the pool using the rank-biased
precision (RBP)-based Method A (Summing contributions) [
The credibility, understandability, and topicality assessments were performed by 26 volunteers (19 women and 7 men) in May–June 2021. Of these assessors, 16 were from Australia, 4 from Italy, 3 from France, 2 from Ireland, and 1 from Finland. We recruited, trained, and supervised them by using bespoke written materials from April to June 2021. The recruitment took place via email and on social media, using both our existing contacts and snowballing.
We implemented these assessments online by expanding and customising the Relevation! tool for relevance assessments [33] to capture the three dimensions of document relevance, and their scale (see [17, Figures 4–6] for illustrations of the online assessment environment). 1https://github.com/mmihaltz/word2vec-GoogleNews-vectors 2https://github.com/ncbi-nlp/BioSentVec We associated every query with 250 documents to be assessed with respect to their credibility, understandability, and topicality. Initially, we allocated each assessor with 2 queries for their assessment and then revised these allocations based on their individual needs and availability. In the end, every assessor completed 1–4 queries.
Ethical approval (2021/013) was obtained for all aspects of this assessment study involving human participants as assessors from the Human Research Ethics Committee of the Australian National University (ANU). Each study participant provided informed consent.
We considered evaluation measures that allowed to evaluate both: 1. the efectiveness of the systems with respect to the ranking produced by taking into account the three criteria considered, and 2. the accuracy of the (binary) classification of documents with respect to credibility (Subtask 3).
Specifically, with regard to the first aspect, this included the following performance
evaluation measures: Mean Average Precision (MAP), preference-based BPref metric, normalized
Discounted Cumulative Gain (nDCG), understandability-based variant of Rank Biased Precision
(uRBP), and credibility-ranked Rank Biased Precision (cRBP) [
With respect to the second aspect, we referred to classical measures to assess the goodness of a classifier, such as Accuracy and the Area under the Receiver Operating Characteristic (ROC) Curve (AUC) [34]. In particular, since Subtask 3 is also devoted to assess credibility in relation to information needs (topics), also the average of a Topic-based Credibility Precision w.r.t. each topic, namely CP(), has been considered.
A brief explanation of the three measures that need more detail, namely uRBP, cRBP, and
CP() is provided below.
3.6.1. Understandability-Ranked Biased Precision
The uRBP measure evaluates IR systems by taking into account both topicality and
understandability dimensions of relevance. In particular, the function for calculating uRBP was [35]:
uRBP = (1 − ) ∑︁ − 1(@)· (@),
=1
(1)
where (@) is the relevance of the document at position , (@) is the understandability
value of the document at position , and the persistent parameter models the user desire to
examine every answer, which was set to 0.50, 0.80, and 0.95 to obtain three version of uRBP,
according to diferent user behaviors.
3.6.2. Credibility-Ranked Biased Precision
In CLEF eHealth 2020, we have adapted uRBP to credibility, obtaining the so-called
credibilityranked biased precision (cRBP) measure [
cRBP = (1 − ) ∑︁ − 1(@)· (@).
=1 (2) As in uRBP, the parameter was set to three values, from an impatient user (0.50) to more persistent users (0.80 and 0.95). 3.6.3. Topic-based Credibility Precision The precision in retrieving credible documents can be calculated over the top- documents in the ranking, for each topic (query) , as follows:
CP() = #____() #___() .
In 2021, 43 teams registered for the task on the web site and two teams submitted runs to the subtasks. We provided the registered participants with the crawler code and the domain list of the crawl for both the web documents and social media documents. They also had access to indexes built by organizers from the document collection on demand. Participants’ submissions were due by 8 May 2021.
Of the four run submissions, two were to Subtask 1 on Adhoc IR, one was to Subtask 2 on Weakly Supervised IR, and one was to Subtask 3 on Document Credibility Prediction. The teams were from two countries (i.e., China and Italy) in two continents (i.e., Asia and Europe). In Subtask 1, the submissions were by 1. a 4-member team from the School of Computer Science, Zhongyuan University of Technology (ZUT) in Zhengzhou, China [36] and 2. a 2-member team from the Information Management Systems (IMS) Research Group,
University of Padova (UniPd), Padova, Italy [37].
In Subtasks 2 and 3, the submissions were by the leader of this IMS UniPd team [37] — a regular participant in our previous CHS tasks.
The two teams submitted the following types of document ranking approaches as runs to the Adhoc IR subtask (Table 4): Team ZUT used a learning-to-rank approach in all its four submitted runs, but with four diferent machine learning algorithms to train their models [ 36]. In contrast, Team UniPd founded their four submissions on a renown Python framework for IR called PyTerrier as variants of Reciprocal Rank Fusion with the provided Terrier index [37].
The UniPd team submitted closely related approaches to the other two subtasks (Table 4). Namely, they also submitted their aforementioned four approaches to the Weekly Supervised IR subtask and two of them and another two of their variants to the Document Credibility Prediction subtask [37]. LM MR RFs RB BM25, QLM, & DFR
variants using RM3 pseudo RF
In 2021, the CLEF eHealth CHS task generated a new representative Web corpus including Web pages acquired from a 2021 CommonCrawl, and social media content from Twitter and Reddit. A new collection of 55 manually generated layperson medical queries was also created, along with their respective credibility, understandability, and topicality assessments for returned documents. In total, 15 runs were submitted to the three subtasks on adhoc IR, weakly supervised IR, and document credibility prediction, respectively.
A total of 12, 500 assessments were made on 11, 357 documents: 7, 400 Web documents, and 3, 957 social media documents. Figure 1 shows the number of social media and Web documents that were assessed for each query. The bottom part shows which queries were created from discussion with patients (Expert queries), and queries created from discussions on social media (Social media queries). We can see that Web documents took a bigger part of the pool of documents. Nevertheless, the proportion of social media documents was bigger for queries based on social media. A special case is presented in queries 22, 63, and 116, where the pool # of documents
Web
2, 540 3, 014 3, 123
Not 4, 259 1, 202 753 of documents was composed by Web documents only. This means that the submitted runs retrieved only Web documents.
While the distribution of the assessments on topicality and understandability dimensions were similar on social media and Web documents, the credibility assessments presented a big diference (Table 5.1), with only 12 documents (less than 1% of social media assessed documents) were assessed as highly credible in the social media set, in contrast to the 4, 552 (54% of Web assessed documents) highly credible Web documents. Finally, an important part of social media documents were assessed as not credible. Figure 2 shows the relationship between the three dimensions of relevance; its diagonal exemplifies the distribution of each dimension. Each plot exposes the number of documents evaluated as “highly”, “somewhat”, or “not”, with respect to two dimensions of relevance (one in each axis). By example, in social media assessments the Figure 2 shows for the not credible documents that a fraction is not topical relevant, a set of documents is somewhat topical relevant, and a small part of the not credible is anyways highly topically relevant.
In this section we present the results for the Adhoc IR subtask where the systems were evaluated in diferent dimensions of relevance. For topicality, we evaluated the systems using MAP, BPref, and NDCG@10 performances metrics. For readability, we made use of uRBP that considers topicality and understandability of the documents. Finally, in order to include credibility in our evaluation, we measured systems’ cRBP performance that considers topicality and credibility of the document.
Table 5.2 presents the ranking of participant systems and organizers baselines, with respect to three metrics of topical relevance: MAP, BPref, and NDCG@10. The team achieving the highest results was UniPd. Their top run, original_rm3_rrf used Reciprocal Rank Fusion with BM25, QLM, DFR approaches using pseudo relevance feedback with 10 documents and 10 terms (query weight 0.5). It achieved 0.43 MAP and 0.51 BPref. The best system of ZUT team was their run3, which used learning to rank techniques with a model trained using Random Forests. ZUT run3 was the best systems of the team on the three metrics with 0.4 MAP, 0.47 BPref, and 0.66 NDCG@10. For BPref and NDCG@10, the organizers baseline using TF× IDF with query expansion obtained higher results.
Table 5.2 compares the results of our baseline systems with respect to last year’s Adhoc IR task results. In the case of MAP and BPref metrics, all the systems had better performance on 2021 test collection; for NDCG@10, DirichletLM with and without relevance feedback had a better performance on 2020 test collection. Despite this clear improvement from 2020 to 2021, the ranking of systems has changed in all the metrics. Namely, in 2020, the best MAP and BPref performance was achieve by DirichletLM, and TF× IDF with respect to NDCG@10 metric. In contrast, in this year’s task, the best baseline MAP, Bpref, and NDCG@10 was achieved by TF× IDF with relevance feedback.
Rank
Team - run 1 2 3 4 5 6 7 8 9 10 11 12 13 14
UniPd original_rm3_rrf UniPd simplified_rm3_rrf ZUT run3_clef2021_task2 Baseline terrier_TF× IDF_qe Baseline terrier_BM25_qe ZUT run4_clef2021_task2 Baseline terrier_DirichletLM Baseline terrier_TF× IDF Baseline terrier_BM25 ZUT run2_clef2021_task2 ZUT run1_clef2021_task2 Baseline terrier_DirichletLM_qe UniPd original_rrf UniPd simplified_rrf 0.431 0.431
The results for all participants for readability and credibility evaluation is shown in Table 5.2. For readability evaluation the best run was the baseline system TF× IDF with relevance feedback. The best participant system was ZUT’s run3 that is the same system that presented the best performance in terms of topicality as well. For credibility evaluation, the best run was the baseline system DirichletLM while the best participant system was UniPd’s original_rm3_rrf — the best system of the team in topicality evaluation.
The purpose of the task was to evaluate systems based on machine learning, trained on the weakly supervised dataset. UniPD submitted their runs to Subtask 1 for this subtask, which are not trained on the weakly supervised dataset. Their submission provides a kind of baseline for a system that does not use any information. Since no other team submitted runs to this task, we cannot provide any evaluation.
The UniPd team, to assess document credibility, reused the runs computed in Subtask 1 and grouped them in order to produce a single score for each document. Their simple hypothesis was that documents that have a higher score across diferent search engines are also more credible. Their approach did not consider any additional information about the provenance of the document. 5.4.1. Credibility Assessment as a Binary Classification Problem The results illustrated in this section were obtained by performing a binary credibility assessment using the CLEF 2020 eHealth dataset for training the baselines and testing on a subset of the CLEF 2021 eHealth data, that is, those that were employed for both runs subtask1_ims_original and subtask1_ims_simplified submitted by the UniPd team. In this case, the topics against which the documents were retrieved are not taken into account. The results of classifying documents according to their credibility using both simple baselines (illustrated in Section 3.3) and considered runs are illustrated in Table 7.
The results obtained from both baselines and runs were evaluated by means of classical measures in the context of document classification, that is, the AUC and Accuracy (as discussed in Section 3.6).
As it can be observed from the table, having considered only linguistic features, the classification results obtained were not particularly significant to the considered problem, neither for the runs nor for the baselines considered. The idea discussed by the members of the UniPd group, that documents having a higher score across diferent search engines are also more credible, did not appear to be supported by these results, but nevertheless merited future investigation, in particular in relation to the results reported in the next section. Regarding the baselines, it was possible to observe that, when supervised classifiers employ the word embedding text representation obtained by Word2vec pre-trained on biomedical datasets, such as Pubmed and MIMIC-III, we obtained best results as compared to their counterparts employing text representation features based on TF× IDF and Word2vec pre-trained on the general-purpose Google News dataset. However, it must be considered that, in general, the Accuracy values in particular could be influenced by the fact that the dataset considered was strongly unbalanced, being made up of about 20% of documents labeled as credible and 80% labeled as non-credible. Probably, this should also raise the need to deepen an analysis about the way in which human assessors evaluate the documents proposed to them. 5.4.2. Topic-based Credibility Assessment Regarding the problem of assessing the credibility of the documents retrieved with respect to the topics considered, we provide below the results relative to the runs sent by UniPD, namely subtask2_ims_original and subtask2_ims_simplified. In this case, for each topic considered, the precision in retrieving credible documents with respect to the topic was evaluated based on the value of CP() calculated as shown in Section 3.6.3.
In Table 8, we report the average CP() values with respect to the set of topics considered. Specifically, the top-100 and top-200 documents retrieved with respect to each topic were taken into account in computing the CP() metric.
As it can be observed from the table, with respect to the evaluations carried out in the previous case related to binary credibility assessment (i.e., Section 5.4.1), it seems that the methods applied in the two submitted runs that consider credibility associated with documents retrieved with respect to specific topics, actually managed to find credible information in the first top- positions, in particular as decreases. In general, the most efective method seems to be the one implemented in the run subtask2_ims_original; however, from specific observations with respect to the results submitted by the participants (which are omitted in this paper), we can state that the second method, the one implemented in subtask2_ims_simplified, proved to be particularly eficient for some topics (even more than the method implemented in subtask2_ims_original) and much less so with respect to others, leading to a lower average CP() value. It is therefore worth investigating this aspect more in the future.
This paper has described methods, results and analysis of the Consumer Health Search (CHS) challenge at the CLEF eHealth 2021 Evaluation Lab. The task considered the problem of lay people searching for medical information related to their health condition on the web. In particular, across web pages and social media content. The task included three subtasks on ad hoc IR, weakly supervised IR, and document credibility prediction. 15 runs were submitted to these tasks.
The CLEF eHealth CHS challenge, first ran at the inaugural CLEF eHealth lab in 2013, and has ofered since then TREC-style IR challenges with medical datasets consisting of large medical web and lay person query collections. As a by-product of this evaluation exercise, the task contributes to the research community a collection with associated assessments and evaluation framework that can be used to evaluate the efectiveness of retrieval methods for health information seeking on the web. Queries, assessments, and participants’ runs for the 2021 CHS challenge are publicly available at https://github.com/CLEFeHealth/CHS-2021 and previous years’ CHS collections are available at https://github.com/CLEFeHealth/.
Thanks: The CHS task of the CLEF eHealth 2021 evaluation lab has been supported in part by the CLEF Initiative. It has also been supported in part by the Our Health in Our Hands (OHIOH) initiative of the Australian National University (ANU), as well as the ANU School of Computing, ANU Research School of Population Health, and Data61/Commonwealth Scientific and Industrial Research Organisation. OHIOH is a strategic initiative of the ANU which aims to transform health care by developing new personalised health technologies and solutions in collaboration with patients, clinicians, and health care providers. Moreover, the task has been supported in part by the bi-lateral Kodicare (Knowledge Delta based improvement and continuous evaluation of retrieval engines) project funded by the French ANR (ANR-19-CE230029) and Austrian FWF. Finally, the task has been supported in part by the EU Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie Grant Agreement No 860721 – DoSSIER: “Domain Specific Systems for Information Extraction and Retrieval”. We are also thankful to the people involved in the query creation and relevance assessment exercises. Last but not least, we gratefully acknowledge the participating teams’ hard work. We thank them for their submissions and interest in the task.
Author Contribution Statement: With equal contribution, Task 2 was led by LG, GP, and HS, and organized by EB, NB-S, GG-S, LK, PM, GP, SS, HS, RU, MV, and CX. [33] B. Koopman, G. Zuccon, Relevation!: an open source system for information retrieval relevance assessment, in: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, ACM, 2014, pp. 1243–1244. [34] H. Suominen, S. Pyysalo, M. Hiissa, F. Ginter, S. Liu, D. Marghescu, T. Pahikkala, B. Back, H. Karsten, T. Salakoski, Performance evaluation measures for text mining, in: M. Song, Y. Wu (Eds.), Handbook of Research on Text and Web Mining Technologies, IGI Global, Hershey, Pennsylvania, USA, 2008, pp. 724–747. [35] G. Zuccon, Understandability biased evaluation for information retrieval, in: Advances in
Information Retrieval, 2016, pp. 280–292. [36] H. Yang, X. Liu, B. Zheng, G. Yang, Learning to rank for Consumer Health Search, in: CLEF 2021 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS, September 2021. [37] G. Di Nunzio, F. Vezzani, IMS-UNIPD @ CLEF eHealth Task 2: Reciprocal Ranking Fusion in CHS, in: CLEF 2021 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS, September 2021.