In this paper, we provide an overview of the CLEF eHealth Task 2 on Information Retrieval (IR), organized as part of the eighth annual edition of the CLEF eHealth evaluation lab by the Conference and Labs of the Evaluation Forum. Its aim was to address laypeople's di culties in retrieving and digesting valid and relevant information, in their preferred language, to make health-centred decisions. The task was a novel extension of the most popular and established task in CLEF eHealth on Consumer Health Search (CHS), which makes responses to spoken ad-hoc queries. In total, ve submissions were made to its two subtasks; three addressed the ad-hoc IR task on text data and two considered the spoken queries. Herein, we describe the resources created for the task and evaluation methodology adopted. We also summarize lab submissions and results. As in previous years, organizers have made data, methods, and tools associated with the lab tasks available for future research and development.
1
In recent years, electronic health (eHealth) content has become available in a variety of forms ranging from patient records and medical dossiers, scienti c publications, and health-related websites to medical-related topics shared across social networks. Laypeople, clinicians, and policy-makers need to easily retrieve, and make sense of such medical content to support their decision making. The increasing di culties experienced by these stakeholders in retrieving and digesting valid and relevant information in their preferred language to make healthcentred decisions has motivated CLEF eHealth to organise yearly shared challenges since 2013.
More speci cally, CLEF eHealth7 was established as a lab workshop in 2012 as part of the Conference and Labs of the Evaluation Forum (CLEF). Since 2013, it has o ered evaluation labs in the elds of layperson and professional health information extraction, management, and retrieval with the aims of bringing together researchers working on related information access topics and providing them with datasets to work with and validate the outcomes. These labs and their subsequent workshops target: 1. developing processing methods and resources (e.g., dictionaries, abbreviation mappings, and data with model solutions for method development and evaluation) in a multilingual setting to enrich di cult-to-understand eHealth texts, support personalized reliable access to medical information, and provide valuable documentation; 2. developing an evaluation setting and releasing evaluation results for these methods and resources; 3. contributing to participants and organizers' professional networks and interaction with all interdisciplinary actors of the ecosystem for producing, processing, and consuming eHealth information.
The vision for the Lab is two-fold: 1. to develop tasks that potentially impact laypeople's understanding of medical information, and 2. to provide the community with an increasingly sophisticated dataset of clinical narrative, enriched with links to standard knowledge bases, evidencebased care guidelines, systematic reviews, and other further information, to advance the state-of-the-art in multilingual information extraction (IE) and information retrieval (IR) in healthcare. 7 https://clefehealth.imag.fr/
The eighth annual CLEF eHealth lab, CLEF eHealth 2020 [
The CHS task was a continuation of the previous CLEF eHealth IR tasks
that ran in 2013, 2014, 2015, 2016, 2017 and 2018 [
The multilingual IE task focused on Spanish. Further details on this challenge
are available in [
The remainder of this paper is structured as follows: First, in Section 2, we detail the task, evaluation, and datasets created. Second, in Section 3, we describe the task submissions and results. Finally, in Section 4, we discuss the study and provide conclusions. 2
In this section, we describe the materials and methods used in the two subtasks. After specifying our document collection, we address the spoken, transcribed, and speech recognized queries. Then, we describe our evaluation methods. Finally, we introduce our human relevance assessments for information topicality, understandability, and credibility. 2.1
The 2018 CLEF eHealth Consumer Health Search document collection was used
in this year's IR challenge. As detailed in [
An initial list of websites was identi ed for acquisition. The list was built by submitting queries on the 2018/2020 topics to the Microsoft Bing Application Programming Interfaces (APIs), through the Azure Cognitive Services, repeatedly over a period of a few weeks, and acquiring the uniform resource locators (URLs) of the retrieved results. The domains of the URLs were then included in the list, except some domains that were excluded for decency reasons.
The list was further augmented by including a number of known reliable health websites and other known unreliable health websites. This augmentation was based on lists previously compiled by health institutions and agencies. 2.2
Historically, the CLEF eHealth IR task has released text queries representative of layperson medical information needs in various scenarios. In recent years, query variations issued by multiple laypeople for the same information need have been o ered. In this year's task, we extended this to also include spoken queries.
These spoken queries were generated by six laypeople in English. All native
English speakers. E orts were made to include a diverse set of accents. Narratives
for query generation were those used in the 2018 challenge. These narratives
relate to real medical queries compiled by the Khresmoi project [
To create a spoken query the layperson listened to the narrative; and generated their spoken query associated with the narrative. The layperson then listened to their generated spoken query and created a textual transcript of the query. To ensure accuracy in transcription, they were required to repeat this process of listening to their narrative and textually transcribing it. This allowed us to generate accurate transcripts of the spoken queries. We did not preprocess the textual transcripts of queries; for example, any spelling mistakes that may be present were not removed.The nal generated query set consisted of 50 topics, with 6 query variations for each topic.
Ethical approval was obtained to generate the spoken queries, and informed consent obtained from study participants. Spoken queries were downloadable from a secured server for the purpose of participating in this year's CLEF eHealth IR challenge, on completion of a signed use agreement by the participating team.
We also provided participants with the textual transcripts of these spoken queries and automatic speech-to-text translations. This transcription of the audio les was generated using the End-to-End Speech Processing Toolkit (ESPNET), Librispeech, CommonVoice, and Google API (with three models). Speech recognition is assessed using Kaldi [40], an open-source speech recognition toolkit distributed under a free license. We use mel-frequency cepstral coe cient (MFCC) acoustic features (13 coe cients expanded with delta and double delta features and energy : 40 features) with various feature transformations including linear discriminant analysis (LDA), maximum likelihood linear transformation (MLLT), and feature space maximum likelihood linear regression (fMLLR) with speaker adaptive training (SAT).
The speech transcription process is carried out in two passes: an automatic transcript is generated with a GMM-HMM model of 12000 states and 200000 Gaussians. The second pass is performed using DNN (nnet3 recipe in kaldi toolkit) acoustic model trained on acoustic features normalized with the fMLLR matrix.
TEDLIUM dataset [42] was used for training acoustic models. It was developed for large vocabulary continuous speech recognition (LVCSR). The train part of the dataset is composed 118 hours of speech.
The English language model is trained with MIT language model toolkit
using following corpora : News commentary 2007-2012 [55], Gigaword version
5 [
Similar to the 2016, 2017, and 2018 pools, we created the pool using the
RBPbased Method A (Summing contributions) by Mo at et al. [
Since the 2018 topics were used in 2020,the pool used in the 2020 CHS task was an extension of 2018's pool. In other words, the merged 2018&2020 pool was used in 2020. For Subtasks 1 and 2, participants could submit up to 4 runs in the TREC format. Evaluation measures were NDCG@10, BPref, and RBP. Metrics such as uRBP were used to capture various relevance dimensions, as elaborated below. 2.4
Relevance assessments were conducted on three relevance dimensions: topicality, understandability, and credibility. Topicality referred to a classical relevance dimension ensuring that the document and the query are on the same topic and the document answers the query. Understandability was an estimation of whether the document is understandable by a patient. In assessment guidelines, assessors were required to estimate how readable they think the documents were to a layperson, that is, a person without any medical background. Topicality and understandability have been used as relevance dimensions in the CHS task of CLEF eHealth for several years.
This year, to take into consideration the phenomenon of the spread of
disinformation online (especially on health-related topics), we introduced a novel
dimension, that is, credibility. Over the years, the interest in studying the concept
of credibility has gradually moved from traditional communication environments,
characterized by interpersonal and persuasive communication, to mass
communication and interactive-mediated communication, with particular reference to
online communication [
In general, credibility is described in the literature as a perceived quality of
the information receiver [
For this reason, when evaluating information credibility in the health-related
context, assessors were asked in the CLEF eHealth 2020 Task 2 to evaluate the
aforementioned multiple aspects by considering, at the same time:
1. any information available about the trustworthiness of the source [
Assessors considered the three dimensions in assessments (i.e., topicality, understandability, and credibility) on a 3-levels scale: { not relevant/understandable/credible, { somewhat relevant/understandable/credible, and { highly relevant/understandable/credible.
In particular, we added a 4th option for credibility for assessors uncertainty: I am not able to judge. This was motivated by the fact that, as illustrated above, the documents to be assessed may actually lack (or it may not be entirely clear) the minimum information necessary to assess their level of credibility.
Assessments were implemented online by expanding and customising the
Relevation! tool for relevance assessments [
CLEF eHealth IR/CHS tasks o ered in 2013{2020 have brought together
researchers working on health information access topics. The tasks have provided
them with data and computational resources to work with and validate their
outcomes. These contributions have accelerated pathways from scienti c ideas
through in uencing research and development to societal impact. The task niche
has been addressing health information needs of laypeople (including, but not
limited to, patients, their families, clinical sta , health scientists, and healthcare
policy makers) | and not healthcare experts only | in a range of languages and
modalities | in retrieving and digesting valid and relevant eHealth information
to make health-centered decisions [
Next, we report on the 2020 participants, method submissions, and their
resulting evaluation outcomes. This expands the brief results section of our lab
overview [
In 2020, 24 teams registered to CLEF eHealth Task 2. Of these teams, 3 took part in the task. Registering for a CLEF task consisted of lling in a form on the CLEF conference website with contact information, and tick boxes corresponding to the labs of interest. This was done several months before run submission, which explains the drop in the numbers.
Also, the task was di cult and demanding which is another explanation for the drop.
Although the interest and participation numbers were considerably smaller than before [48{50], organizers were pleased with this newly introduced task, with its novel spoken queries element attracting interest and submissions (Table 1). In addition, every participating team took the o ered more-traditional ad-hoc task. 3.2
Among ve submissions to the 2020 CLEF eHealth Task 2, the ad-hoc IR subtask was the most popular with its three submissions; the subtasks that used transcriptions of the spoken queries and the original audio les received one submission each. Speci cally, the subtask that used transcriptions of the spoken queries had one submission and the subtask where the original audio les were processed had one submission. The submitting teams were from Australia, France, and Italy and had 4, 5, and 3 team members, respectively.8 Each team had members from a single university without other partner organizations.
The Italian submission to the ad-hoc search and spoken queries using
transcription subtasks was from the Information Management System (IMS) Group
of the University of Padua [
The French team, LIG-Health, was formed by Dr Philippe Mulhem, Aidan
Mannion, Gabriela Gonzalez Saez, Dr Didier Schwab, and Jibril Frej from the
Laboratoire d'Informatique de Grenoble of the Univ. Grenoble Alpes [
The Australian team { called SandiDoc { was from the Our Health In Our
Hands (OHIOH) Big Data program, Research School of Computer Science,
College of Engineering and Computer Science The Australian National
University [43]. Its members were Sandaru Seneviratne, Dr Eleni Daskalaki, Dr Artem
Lenskiy, and Dr Zakir Hossain. Di erently from the other two teams,
SandiDoc took part in the ad-hoc search task only. Its IR method had three steps
and was founded on TF IDF scoring: First, both the dataset and the queries
were pre-processed. Second, TF ID scores were computed for the queries and
used to retrieve the most similar documents for the queries. Third, the team
supplemented this method by working on the clefehealth2018 B dataset using
the medical skip-gram word embeddings provided. To represent the documents
and queries, the team used the average word vector representations as well as
the average of minimum and maximum vector representations of the document
or query. In documents, these representations were derived using the 100 most
frequent words in a document. For each representation, the team calculated the
similarity among documents and queries using the cosine measure to obtain the
nal task results. The team's aim was to experiment with di erent vector
representations for text.
In addition to these participants' methods, we as organizers developed baseline
methods that were based on the renowned OKapi BM25 but now with query
expansion optimized in a REINFORCE [58] fashion. In this section, we introduce
the two steps of query expansion and document retrieval. First, we pre-trained
our query expansion model on the generally available corpora. Similar to the
REINFORCE learning protocol as introduced in [58], this pre-training step was
done by iterations of exploration trials and optimization of the current model by
rewarding the explorations. Each time an input query was enriched by the query
expansion model from the last iteration, and from where several candidate new
queries were generated. The system was rewarded or penalized by matching the
retrieved documents from these candidate queries against the ground truth
document ranking. To expand the trial of generating new queries and thus provide
more training sources, the system used the context words found in these newly
retrieved documents to construct queries for the next iteration. For this baseline
model, the same datasets from [
One key challenge for information retrieval in the eHealth domain is that
layperson may lack the professional knowledge to precisely describe medical
topics or symptoms. A layperson's input query into the system can be lengthy
and inaccurate, while the documents to be matched for these queries are usually
composed by people of medical mind and background and thus rigorous in
wording. The query expansion phase was added to increase the chance of matching
more candidate documents by enriching the original query. With this intuition
in mind, we employed similar candidate query construction method and
optimization target as introduced in [
R) X where R and R are the reward from the new query and baseline reward, and t 2 T are words from the new query. With the actions and reward being properly de ned, the system can be optimized under the REINFORCE learning framework [58]. At inference stage, the system will greedily peek the optimal selection operations to generate a few candidate queries from the input query.
After enriching the input queries by the pre-trained query expansion model, the second step of this baseline model reused the commonly-used BM25 algorithm [41]. 3.4
shows the task metrics (MAP, BPREF, NDCG, uRBP and cRBP) for all the participants runs, and the organizers baselines. ad-hoc search task. Bold cells are the highest scores. Statistical signi cance tests were conducted but the highest participants runs were not statistically signi cantly higher than the best baselines.
Team Baseline
IMS
LIG SandiDoc the scores are not statistically signi cantly higher than the best baseline. Their top run, original rm3 rrf run uses Reciprocal Rank fusion with BM25, QLM, DFR approaches using pseudo relevance feedback with 10 documents and 10 terms (query weight 0.5) and achieves 0.28 MAP and 0.43 BPref. The second best run for MAP and BPref is also from IMS, using the same ranking system without PRF. For NDCG, the organizers baseline using ElasticSearch BM25 without query expansion obtains higher results. Interstingly, we observe in that can give very di erent results (MAP ranging from 0.11 to 0.26).
Since the ad-hoc task used the same topics and documents as 2018 but intended to extend 2018's pool, we compared teams results for all the ad-hoc task metrics in Figure 1. The
gure shows that the extension of the pool had a
relatively limited impact on the performances of each submitted systems, except
for Bpref measure that shows a consistent decrease. Bpref, correlated to the
average precision, is more robust to reduced pools [
Particularly, the function for calculating uRBP was: (1) where r(d@k) is the relevance of the document d at position k, u(d@k) is the understandability value of the document d at position k, and the persistent parameter models the user desire to examine every answer, which was set to 0.50, 0.80 and 0.95 to obtain three version of uRBP, according to di erent user behaviors.
The results for all the participants for understandability evaluation is shown in the second last column of Table 2. Table 5 shows the top ten runs submitted in the ad-hoc Task. The best run was obtained with Reciprocal rank fusion with BM25 run on manual variants of the query by team IMS. The BM25 baseline gives very close performances. The run ranking does not di er from the topicality evaluation. This could be due to the fact than none of the submitted runs included features speci cally designed to assess understandability of the results.
These results have been obtained with the binary relevance assessment, and the graded understandability assessment, and rbp eval 0.5 as distributed by RBP group. For further details, please refer to https://github.com/jsc/rbp eval. In this section, we report the results produced for the two subtasks: ad-hoc search and spoken queries retrieval. In particular, for the ad-hoc subtask, only the adhoc IR subtask is considered (no speech recognition). For each subtask, the results of both baseline methods and team submission methods in the context of credibility assessment related to IR are reported and commented. The measures employed to assess the credibility of the tasks considered are detailed below.
In the literature, accuracy is certainly the measure that has been used most frequently to evaluate a classi cation task, and, as such, it has been usually employed to evaluate the e ectiveness of credibility assessment. In fact, to date, the problem of assessing the credibility of information has mostly been approached as a binary classi cation problem (by identifying credible versus non-credible information) [56]. Some works have also proposed the computation of credibility values for each piece of information considered, by proposing users a credibilitybased ranking of the considered information items, or by leaving to users the choice of trusting the information items based on these values [36, 56, 60]. Obviously, once an adequate threshold has been chosen, it would be possible to transform these approaches into approaches that produce a binary classi cation.
However, our purpose in asking assessors to evaluate the documents from the point of view of their credibility is to be able to generate IR Systems that can retrieve credible information, besides understandable and topically relevant. For this reason, in CLEF eHealth 2020 we have adapted the understandabilityranked biased precision (uRBP) illustrated in [61] to credibility, by employing the so-called cRBP measure. In this case the function for calculating cRBP is the same used to calculate uRBP (see Equation 1 in Section 3.5 , replacing u(d@k) by the credibility value of the document d at position k, c(d@k) : (2) As in uRBP the parameter was set to three values, from impatient user (0.50) to more persistent users (0.80 and 0.95).
It is important to underline that, in de ning IR approaches implemented by both the organizers and the three groups that submitted runs, no explicit reference was made to solutions for assessing the credibility of documents. Therefore, any potential increase in the evaluation gures must be considered purely coincidental.
Evaluation with Accuracy. The results illustrated in this section were obtained with a binary credibility assessment for 2020 and trustworthiness for 2018 data. A document assessed with a credibility/trustworthiness value 50% was considered as credible. The accuracy of the credibility assessment was calculated over the top 100 documents retrieved for each query as follows: acc(q) = #credible retrieved docs top 100(q) #retrieved docs top 100(q) :
Table 6, referring to the ad-hoc Search subtask, shows that most of the approaches tested presented a good accuracy value when it comes to the credibility of the retrieved documents. However, SandiDoc's submission had, unfortunately, a very low accuracy value. With respect to the spoken queries IR subtask, the results were available only for IMS and LIG, as follows:
With respect to this second subtask, the accuracy values illustrated in Table 7 were lower than those referred to the previous task with respect to all the approaches tested by IMS and LIG. This is most likely due to errors from speech recognition multiplying in IR, similar to what we experienced in the CLEF eHealth 2015 and 2016 tasks on speech recognition and IE to support nursing shift-change handover communication [47, 54]. Evaluation with cRBP. These results have been obtained with the binary relevance assessment, and the graded credibility assessment, with the same program referred in section 3.5 (rbp eval 0.5). To obtain cRBP with di erent persistence values, rbp eval was ran as follows for credibility: rbp eval -q -H qrels.credibility.clef201820201.test.binary runName
As for the values obtained by using the cRBP measure, at 0.50, 0.80, and 0.95, it is possible to say that, regardless of the speci c approach used, and with respect to the ad-hoc search task, they range in the 0.15{0.57 interval for the baseline, with an average of 0.40; in the 0.41{0.57 interval for IMS, with an average of 0.50; in the 0.40 { 0.49 for LIG, with an average of 0.45; in the 0.16{0.23 interval for SandiDoc, with an average of 0.20. These results are pretty coherent with those obtained for accuracy. Considering the spoken queries retrieval subtask, also in this case the results are available only for IMS and LIG. In this case, for IMS they range in the 0.37{0.50 interval, with an average of 0.44, while for LIG they range in the 0.30{0.45 interval, with an average of 0.37. 4
This year's challenge o ered an ad-hoc search subtask and a spoken query retrieval subtask. In Section 3 we provided an analysis of the results obtained by the 3 teams who took part in these tasks. We showed the results achieved by several baselines provided by the task organizers. We also discussed and compared these results in Section 3. We consider three dimensions of relevance - topicality, understanability, and credibility.
The importance of considering several dimensions of relevance, beyond the traditional topicality measure is highlighted in the results obtained, where we nd that di erent retrieval techniques score higher under each of the relevance dimensions (topicality, understanability, and credibility).
As might be expected, retrieval performance is impacted when the queries are presented in spoken form. Speech-to-text conversion is required before the queries can be used in the developed retrieval approaches. The retrieval performance then is impacted by the quality of the speech-to-text conversion. Future studies will explore this phenomenon in greater detail.
We next look at the limitations of this year's challenge. We then re ect on
prior editions of the challenge and the challenges future, before concluding the
paper.
As previously illustrated, the concept of credibility has been studied in many
di erent research elds, including both psychology/sociology and computer
science. Introducing the concept of credibility in the context of IR is not an easy
task. On one hand, it represents a characteristic that can be only perceived
by human beings [
As illustrated in Section 2.4, measures that have been used so far to evaluate the e ectiveness of a systems in identifying credible or not credible information are the traditional ones that are used in machine learning for classi cation tasks, in particular accuracy. To be able to evaluate IR systems that are capable to retrieve credible information (as well as understandable and topically relevant) one could consider more IR-oriented metrics. In CLEF eHealth 2020, we have adapted uRBP to credibility, by employing the so-called cRBP measure (as illustrated in Section 3.6).
However, this measure considers the credibility related to an information item as subjective, while we believe that we should assess credibility in an objective way. This makes this measure only partially suitable to our purposes (as well as the accuracy was only partially suitable). In this scenario, it becomes essential to develop measures that go beyond taking information credibility into account as a a binary value, as is done in classi cation systems. This problem calls for advancing IR systems and developing related evaluation measures that factor in the joint goodness of the ranking produced by a search engine with respect to multiple dimensions of relevance. These include, but are not limited to, topicality, understandability, and credibility. Including credibility is critical in order to balance between the subjectivity (of assessors) and the objectivity of a fact reported in an information item. Consequently, its inclusion is certainly one of the most ambitious goals we set for our future work.
The inaugural CLEF eHealth CHS/IR task was organized in 2013 on the
foundation set by the 2012 CLEF eHealth workshop. The principal nding of this
workshop, set to prepare for future evaluation labs, was identifying laypeople's health
information needs and related patient-friendly health information access
methods as a theme of the community's research and development interest [46]. The
resulting CLEF eHealth tasks on CHS/IR, o ered yearly from 2013 to 2020 [
This achieved impact has led to CLEF eHealth establishing its presence and
becoming by 2020 one of the primary evaluation lab and workshop series for
all interdisciplinary actors of the ecosystem for producing, processing, and
consuming eHealth information [
The general purpose of the CLEF eHealth workshops and its preceding CHS/IR
tasks has been throughout the years from 2012 to 2020 to assist laypeople into
nding and understanding health information in order to make enlightened
decisions concerning their health and/or healthcare [
Improving multilingual and multi-modal methods is crucial to guarantee a better access to information, and to understand it. Breaking language and modality barriers has been a priority in CLEF eHealth over the years, and this will continue. Text has been the major media of interest, but as of 2020, also speech has been included as a major new way of people interacting with the systems.
Patient-centered IR/CHS task has been running since 2013 | yet, every
edition has allowed to identify unique di culties and challenges that have shaped
the task evolution [
In 2020 the CLEF eHealth lab o ered two shared task. One on multilingual IE and the other on consumer health search. These tasks built on the IE and IR tasks o ered by the CLEF eHealth lab series since its inception in 2013. Test collections generated by these shared tasks o ered a speci c task de nition, implemented in a dataset distributed together with an implementation of relevant evaluation metrics to allow for direct comparability of the results reported by systems evaluated on the collections.
These established CLEF IE and IR tasks used a traditional shared task model for evaluation in which a community-wide evaluation is executed in a controlled setting: independent training and test datasets are used and all participants gain access to the test data at the same time, following which no further updates to systems are allowed. Shortly after releasing the test data (without labels or other solutions), the participating teams submit their outputs from the frozen systems to the task organizers, who evaluate these results and report the resulting benchmarks to the community.
The annual CLEF eHealth workhops and evaluation labs have matured and established their presence in 2012{2020 in proposing novel tasks in IR/CHS. Given the signi cance of the tasks, all problem speci cations, test collections, and text analytics resources associated with the lab have been made available to the wider research community through our CLEF eHealth website9.
We gratefully acknowledge the contribution of the people and organizations involved in CLEF eHealth in 2012{2020 as participants or organizers. We thank the CLEF Initiative, Benjamin Lecouteux (Universite Grenoble Alpes), Jo~ao Palotti 9 https://clefehealth.imag.fr/ (Qatar Computing Research Institute), Harrisen Scells (University of Queensland), and Guido Zuccon (University of Queensland). We thank the individuals who generated spoken queries for the IR challenge. We also thank the individuals at University of Queensland who contributed to the IR query generation tool and process. We are very grateful to our assessors that helped despite the COVID-19 crisis: Paola Alberti, Vincent Arnone, Nathan Baran, Pierre Barbe, Francesco Bartoli, Nicola Brew-Sam, Angela Calabrese, Sabrina Caldwell, Daniele Cavalieri, Madhur Chhabra, Luca Cu aro, Yerbolat Dalabayev, Emine Darici, Marco Di Sarno, Mauro Guglielmo, Weiwei Hou, Yidong Huang, Zhengyang Liu, Federico Moretti, Marie Revet, Paritosh Sharma, Haozhan Sun, Christophe Zeinaty. The lab has been supported in part by The Australian National University (ANU), College of Engineering and Computer Science, Research School of Computer Science; the Our Health in Our Hands (OHIOH) initiative; and the CLEF Initiative. OHIOH is a strategic initiative of The ANU which aims to transform healthcare by developing new personalised health technologies and solutions in collaboration with patients, clinicians, and healthcare providers. We acknowledge the Encargo of Plan TL (SEAD) to CNIO and BSC for funding, and the scienti c committee for their valuable comments and guidance. 36. Pasi, G., De Grandis, M., Viviani, M.: Decision making over multiple criteria to assess news credibility in microblogging sites. In: IEEE World Congress on Computational Intelligence (WCCI) 2020, Proceedings. IEEE (2020) 37. Pasi, G., Viviani, M.: Information credibility in the social web: Contexts, approaches, and open issues. arXiv preprint arXiv:2001.09473 (2020) 38. Popat, K., Mukherjee, S., Strotgen, J., Weikum, G.: Credibility assessment of textual claims on the web. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. pp. 2173{2178 (2016) 39. Popat, K., Mukherjee, S., Strotgen, J., Weikum, G.: Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In: Proceedings of the 26th International Conference on World Wide Web Companion. pp. 1003{ 1012 (2017) 40. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P., et al.: The kaldi speech recognition toolkit. In: IEEE 2011 workshop on automatic speech recognition and understanding. No. CONF, IEEE Signal Processing Society (2011) 41. Robertson, S.: The probabilistic relevance framework: BM25 and beyond.
Foundations and Trends® in Information Retrieval 3(4), 333{389 (2010). https://doi.org/10.1561/1500000019 42. Rousseau, A., Deleglise, P., Esteve, Y.: Ted-lium: an automatic speech recognition dedicated corpus. In: LREC. pp. 125{129 (2012) 43. Sandaru Seneviratne, Eleni Daskalaki, A.L., Hossain, M.Z.: SandiDoc at CLEF 2020 - Consumer Health Search : AdHoc IR Task. In: Conference and Labs of the Evaluation (CLEF) Working Notes. CEUR Workshop Proceedings (CEURWS.org) (2020) 44. Sba , L., Rowley, J.: Trust and credibility in web-based health information: a review and agenda for future research. Journal of medical Internet research 19(6), e218 (2017) 45. Self, C.C.: Credibility. In: An integrated approach to communication theory and research, pp. 449{470. Routledge (2014) 46. Suominen, H.: CLEFeHealth2012 | The CLEF 2012 Workshop on Cross-Language Evaluation of Methods, Applications, and Resources for eHealth Document Analysis. In: Forner, P., Karlgren, J., Womser-Hacker, C., Ferro, N. (eds.) CLEF 2012 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1178/ (2012) 47. Suominen, H., Hanlen, L., Goeuriot, L., Kelly, L., Jones, G.: Task 1a of the CLEF eHealth evaluation lab 2015: Clinical speech recognition. In: Online Working Notes of CLEF. CLEF (2015) 48. Suominen, H., Kelly, L., Goeuriot, L.: Scholarly in uence of the Conference and Labs of the Evaluation Forum eHealth Initiative: Review and bibliometric study of the 2012 to 2017 outcomes. JMIR Research Protocols 7(7), e10961 (2018). https://doi.org/10.2196/10961 49. Suominen, H., Kelly, L., Goeuriot, L.: The scholarly impact and strategic intent of CLEF eHealth Labs from 2012 to 2017. In: Ferro, N., Peters, C. (eds.) Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF. pp. 333{363. Springer International Publishing, Cham (2019) 50. Suominen, H., Kelly, L., Goeuriot, L., Krallinger, M.: Clef ehealth evaluation lab 2020. In: Jose, J.M., Yilmaz, E., Magalh~aes, J., Castells, P., Ferro, N., Silva, M.J., Martins, F. (eds.) Advances in Information Retrieval. pp. 587{594. Springer International Publishing, Cham (2020) 51. Suominen, H., Kelly, L., Goeuriot, L., Neveol, A., Ramadier, L., Robert, A., Kanoulas, E., Spijker, R., Azzopardi, L., Li, D., Jimmy, Palotti, J., Zuccon, G.: Overview of the clef ehealth evaluation lab 2018. In: International Conference of the Cross-Language Evaluation Forum for European Languages, pp. 286{301. Springer Berlin Heidelberg (2018) 52. Suominen, H., Kelly, L., Goeuriot, L., Neveol, A., Ramadier, L., Robert, A., Kanoulas, E., Spijker, R., Azzopardi, L., Li, D., Jimmy, Palotti, J., Zuccon, G.: Overview of the CLEF eHealth evaluation lab 2018. In: Bellot, P., Trabelsi, C., Mothe, J., Murtagh, F., Nie, J.Y., Soulier, L., SanJuan, E., Cappellato, L., Ferro, N. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. pp. 286{301. Springer International Publishing, Cham, Switzerland (2018) 53. Suominen, H., Salantera, S., Velupillai, S., Chapman, W.W., Savova, G., Elhadad, N., Pradhan, S., South, B.R., Mowery, D.L., Jones, G.J., Leveling, J., Kelly, L., Goeuriot, L., Martinez, D., Zuccon, G.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Information Access Evaluation. Multilinguality, Multimodality, and Visualization, pp. 212{231. Springer Berlin Heidelberg (2013) 54. Suominen, H., Zhou, L., Goeuriot, L., Kelly, L.: Task 1 of the CLEF eHealth evaluation lab 2016: Handover information extraction. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS (2016) 55. Tiedemann, J.: Parallel data, tools and interfaces in opus. In: Lrec. vol. 2012, pp.
2214{2218 (2012) 56. Viviani, M., Pasi, G.: Credibility in social media: opinions, news, and health information|a survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 7(5), e1209 (2017) 57. Wayne, C., Doddington, G., et al.: Tdt2 multilanguage text version 4.0 ldc2001t57.
Philadelphia: Linguistic Data Consortium (LDC) (2001) 58. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. In: Reinforcement Learning, pp. 5{32. Springer US (1992) 59. Yamamoto, Y., Tanaka, K.: Enhancing credibility judgment of web search results.
In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1235{1244 (2011) 60. Yang, K.C., Varol, O., Davis, C.A., Ferrara, E., Flammini, A., Menczer, F.: Arming the public with arti cial intelligence to counter social bots. Human Behavior and Emerging Technologies 1(1), 48{61 (2019) 61. Zuccon, G.: Understandability biased evaluation for information retrieval. In: Advances in Information Retrieval. pp. 280{292 (2016) 62. Zuccon, G., Palotti, J., Goeuriot, L., Kelly, L., Lupu, M., Pecina, P., Mueller, H., Budaher, J., Deacon, A.: The IR Task at the CLEF eHealth Evaluation Lab 2016: User-centred Health Information Retrieval. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS (September 2016)