Judging the credibility of information is a subjective process and prone to biases. This issue can be especially concerning in health information seeking. Some eforts have been made to define robust credibility assessment guidelines that support the development of reliable test collections. This is of the utmost importance since the applicability of retrieval algorithms to real use case scenarios relies on the quality of the labelled data. Yet, the question persists as to whether the labels created by these guidelines can efectively serve as a surrogate for the genuine judgements of credibility as perceived by end-users. Motivated by this, we conducted a user study with 1,000 participants. We demonstrate that there is a correlation between participants' judgements and the reference values produced following existing guidelines. Further analyses of the data reveal worrying insights into people's ability to judge the credibility of online medical content, leading to potential personal harm.
CEUR ceur-ws.org
The Internet has become the dominant platform for accessing health information, ofering
convenient access to a wealth of medical knowledge [
Nevertheless, there is still a need for a rigorous examination of the relationship between this
type of guidelines and credibility as real end-users perceive it. While we know that judgements
vary across users, annotations need to be both consistent and reflective of average users’
perceptions. Previous research has studied the main elements influencing individual credibility
perceptions [
In this work, we perform a study to understand how end-users perceive the credibility of online health information1. The ultimate goal being to determine whether the labels created from guidelines can serve as surrogate of credibility perceived by end-users. We attempt to answer the following research questions: • RQ1. Can current credibility annotation guidelines act as a proxy of the real perception of credibility of end-users? • RQ2. To what extent users are able to recreate the judgements of experts? • RQ3. How do user variables, such as familiarity with the search topic, educational background, and other human factors, afect the user’s perception of credibility?
The credibility of online information and the spread of misinformation have been extensively
studied [
Easting et al. [21] demonstrated that both the source and the prior knowledge about the
content have influence on users’ perception of online health information. Other studies have
also demonstrated that, apart from the characteristics of the web elements, the receiver’s
characteristics also influence the perception of the information [ 22]. Other researchers analysed
in-depth the factors that influence end-users’ perceived credibility [
In this paper, we present a systematic user study that shows how well expert annotations reflect the subjective judgements of a broad population of users. We also evaluate a number of personal factors that may influence the credibility estimations. 1https://github.com/MarcosFP97/perceived-credibility-study
T10
Do antioxidants help female subfertility? Do sealants prevent dental decay in permanent teeth? Does melatonin help treat and prevent jetlag? Does traction help low back pain?
We hypothesise that reference values (created by expert annotators using formal guidelines) will correlate with users’ judgements. To test this, we conducted a crowd-sourced user study whereby participants provided credibility judgements for webpages.
We utilised a pre-existing dataset from the medical domain that had been originally compiled
by Pogacar et al. [
For these webpages, we also produced annotations generated by human assessors according
to the guidelines from [
• Reference values: variable indicating the credibility-level perceived by the human annotators according to the guidelines. There are three possible levels: 0 (non-credible), 1 (credible), and 2 (highly credible).
• User credibility score: the credibility score assigned by the crowdsourcers in a 7-point
Likert scale. 2We did not use the raw HTML, since we consider visual elements as key to the perception of credibility. 3Three PhD students with background in British Studies, Computer Science and Information Science, respectively, and a Master’s Degree student in Information Science
G1 G2 G3 G4 G5 G6
Label 2 1 1 0 0 0
Guideline Source is a scientific paper, or a Medical publisher or hospital/clinic or government website or university.
Document is citing the information they provide in their articles. They provide links or specific references to their sources. They cite sources with credibility 2 (i.e. medical publications and/or lab studies).
Document is written by an expert in the field/someone qualified to write this document (irrespective of publishing venue).
The document is actually for advertising or marketing purposes. If so, the website might be biased or a scam designed to trick people into fake treatments or into buying medical products that do not live up to their claim.
The information posted by a non-expert person providing a medical product review or providing medical advice without proper citations (links/list of references).
The website provides or states claims that go against well-known medical consensus (e.g. smoking cigarettes does not cause cancer).
NOTE: It is generally allowed to look up authors to check whether they have the required knowledge to be regarded as an expert and look up websites to find out if they are legitimate. • Time of completion (in minutes): variable representing the total amount of time it took for a crowdsourcer to complete the assessment (measured from the moment the screenshot of the website was shown until successful completion).
We also studied some variables that can influence or have some connection with the credibility scores gathered in the study (this relation is further explored in Section 4): • Topic familiarity: in the pre-task questionnaire, see Figure 1 (left side), participants were asked about their prior knowledge on the topic.
• Personal data: in a post-study questionnaire, see Figure 1 (right side), we gathered additional information about the participants in the study (educational background, gender, and age). • Justifications : we also provided the participants with the possibility of justifying their rating in their own words (free text field).
Once participants had been presented with the goals of the study, its methodology, and the implications of their involvement, they provided their permission by signing a consent form. Next, they proceeded with 3 steps to satisfactorily complete the study: 1. Each participant was randomly assigned one webpage from the collection. Before seeing the webpage’s screenshot, they needed to fulfil a pre-task questionnaire about their expertise on the topic of the website, see Figure 1 (left side). 2. Subjects were shown a screenshot of the entire webpage and they needed to assess its credibility in a 7-point Likert scale (from not credible at all to very credible), see Figure 2. They had no time limit to provide this estimate, and they could scroll through the entire screenshot and provide a free-text justification about their judgement (this step was not mandatory, however a high number of participants provided this feedback, see Section 4.6). 3. Before ending the study, participants were shown a post-study questionnaire, see Figure 1 (right side). Our main goal was to gather additional data, such as educational background, age, and gender.
We recruited a total of 1,000 users to guarantee at least 5 judgements per webpage. We used the Prolific 4 platform and each participant received £0.32 (equivalent to £9.60 per hour). Our participants were fluent English speakers, resident in the United States or the United Kingdom. The annotators belonged to an age range between 18 and 85 years old. 53% identified as female, 45% as male, and the remaining participants identified either as diverse or other. In terms of educational background, 40% had a bachelor’s degree, 37% completed secondary education, and only 2.6% of the participants reported a level of education below high school.
RQ1 seeks to determine whether the current annotation guidelines, whose goal is to produce
robust assessments of credibility for medical websites, serve as a proxy for the human’s perception
of credibility [
Despite this correlation, there are some signs of concern regarding RQ2. Webpages annotated as non-credible according to the guidelines were often perceived as reliable by the participants. This can be observed by the fact that non-credible documents have very high perception scores and their median score is 5. This confirms previous research findings that people tend to overestimate credibility and have problems identifying low-quality sites [27, 28]. In general, webpages labelled as credible or highly credible by the reference judgements were also considered as of high quality by the crowdworkers. However, people struggled to detect contents that are regarded as low quality by the reference annotations.
Summing up, regarding RQ1, we can conclude that the participants’ judgements and the reference values derived from the guidelines are correlated. As for RQ2, we found out that crowdworkers are less prone to errors when evaluating high quality pages, but they struggled for pages with lower levels of credibility.
Prior to completing the study and to partially answer RQ3, we asked participants about their level of knowledge or familiarity with their assigned topic in a 5-point Likert scale (ranging from Not at all familiar to Extremely familiar, see Figure 1 (left side)).
Figure 4 (left side) shows the relation between levels of familiarity and the participants’ judgements. It seems that the higher the familiarity, the higher the credibility judgements provided by participants. Spearman’s correlation yielded a = 0.10 and − < 0.01 , demonstrating that there is a very weak correlation between the two variables [26].
To further explore the user study data, we also computed the deviation per level of familiarity between the reference values and the user study’s judgements. First, we applied a Min-Max normalisation to both sets of scores. Then, the diference between the crowdsourcing judgements and the reference values was computed –0 represents a perfect match, while a positive (negative) value means that people overestimated (underestimated) credibility– see Figure 4 (right side). From the figure, we might conclude that there is not a strong relation between familiarity and how efective are users at rating webpages. However, Spearman’s correlation yielded a = 0.13 and − < 0.01 .
As a complementary analysis, we also computed the mean familiarity per topic (and its standard deviation): T1 has a mean familiarity of 1.55 (0.89), T5 of 1.79 (1.07), T8 of 2.44 (1.25), and T10 of 2.14 (1.15).
Figure 5 (left side) reports a topic-level analysis. As can be expected, there are individual diferences among the topics. Spearman’s test also revealed statistically significant correlations between reference credibility scores and crowdsourced credibility scores for all topics. However, the correlation for T1 was lower. These results fit with the familiarity scores described above, where T1 was shown to be the topic that users had less knowledge about.
Again, we also computed the deviation per topic between the reference values and the user study’s judgements, see Figure 5 (right side). Spearman’s test revealed an statistically significant correlation between both variables ( = 0.20 and − < 0.01 ). An interesting finding is that for the topics users are more familiar with, T8 and T10, they tend to overestimate their perceived credibility.
To fully answer RQ3, the relation between additional user variables (gender, age, and educational background) and their perception of credibility was explored. For the first two variables, no significant correlations or revealing trends were found. However, for the educational background (Figure 6), we found an interesting conclusion: all groups were equally good at estimating credibility, except for the less educated (less than high school group) and the most educated (doctorate group). The Spearman’s test reported a = 0.05 and − = 0.12 rejecting the hypothesis that there is a correlation between the educational level and the quality of the assessments, measured by the deviation between the crowdsourced and reference values.
We also analysed the time (in minutes) users needed to complete the assessment. Figure 7 shows that users who spent less time analysing the web (between 0-6 minutes) tended to deviate less from the reference values (deviation close to 0). We speculate that “overthinking” might be counterproductive for this task. Alternatively, the lower quality of the estimates at the right end of the graph could be due to other factors such as distractions. Related to this, previous studies showed that people who take more time on this type of tasks tend to be more influenced by visual elements and their prior knowledge [29]. In any case, the Spearman’s correlation test revealed no statistical significance (with a = 0.009 and a − = 0.78 ) between completion time and deviation between crowdsourced and reference judgements.
We ofered users the possibility of justifying their judgements. This was actively used by participants, with 93% providing a textual explanation. This gave us valuable evidence to analyse the reasons behind credibility judgements. Yet, manual inspection was infeasible because we had thousands of datapoints. We therefore opted for exploiting the summarisation capabilities of current Large Language Models (LLMs). To that end, the justifications were grouped by the diferent levels of perceived credibility and GPT-4 was provided with these textual extracts and asked to generate a summary for each level. The template used for prompting the LLM was as follows: “We are a group of scientists that have conducted an online survey on webpage credibility. For each webpage, we asked a human assessor to provide a score credibility from 1 to 7 (very low credible to very credible). Assessors could also provide a justification on why they assigned a given credibility score. Given a series of justifications between <>, I want you to generate an understandable summary. <justifications>. The summary is: ”. The resulting summaries are presented in Table 3.
Some interesting patterns were observed in the provided explanations. Low credibility was
usually associated with a poor visual appearance of the webpage. Moreover, several users
mentioned the lack of reputable references as a critical aspect to mistrust information. This
suggests the existence of a direct relationship between the real perception of end-users and
the existing guidelines [
lack of reliable sources as reasons for low credibility scores. Additionally, some expressed distrust towards Wikipedia and belief that the information could be biased or incorrect.
The justifications mainly indicate that the webpages appear amateur, unprofessional, and lacking credible sources or evidence. Additionally, many assessors express skepticism due to outdated information, poor design, and potential bias from advertisements or afiliate links.
The webpages were considered less credible due to factors such as unprofessional design, outdated information, informal language, lack of evidence or citations, and clickbait-like content. The credibility of some pages was questioned due to unfamiliar brands, personal opinions, and the absence of proper qualifications or accreditations.
Assessors found some webpages to be credible due to professional appearance, links to studies or association with reputable organizations, while others were seen as less credible due to informal language, lack of citations or references, and potential for errors. The credibility of some pages was dificult to judge without further investigation or knowledge of the subject matter.
The justifications highlight the presence of credible sources, professional appearance, and author qualifications as positive factors for credibility. However, some concerns are raised due to missing citations, outdated information, and potential biases. Survey participants found the webpages credible due to their professional appearance, use of medical facts and references, reputable sources, well-structured content, and qualified authors. The credibility was also often influenced by personal experiences or previous knowledge about the subjects discussed.
The majority of the justifications indicate that the webpages are credible due to their professional appearance, reputable sources, and being associated with trusted organizations such as the NHS, CDC, and various academic journals. Additionally, assessors mentioned the presence of scientific research, citations, author credentials, and detailed information as contributors to the credibility of the webpages.
1 (low credibility)
In this study, we showed that there is correlation between the annotations produced from
existing credibility guidelines and the end-user’s perception of credibility. This highlights the
value of existing guidelines [
We also demonstrated that this relation is topic-dependent, as users tend to deviate more from the reference ground truth when they are less familiar with the topic. Results also confirmed previous research that suggests that people tend to overestimate credibility and, often, struggle to identify sites labelled as low-quality. We also studied the influence of other variables, such as the time of completion or the educational background, and some interesting conclusions arose. Regardless of the educational background, people have dificulties judging the credibility. This even happens with individuals that have a strong educational background (for example, graduated students who have often been trained in skills such as critical thinking). It also appears that “overthinking” and spending too much time to emit a judgement does not lead to better estimates of credibility. Text-free justifications were also inspected, confirming a direct relationship between certain elements from the credibility guidelines and user’s perceptions.
In this paper, we have conducted a user study on people’s perception to the credibility of online health information. First, we used a previous study in the field to produce reference values based on a series of guidelines. We found out a correlation between these values and the judgements collected in the study. However, some worrying facts were also found: people tend to overestimate the credibility of the sites (this can be specially damaging when health information seeking) and it seems that the educational background has not a direct efect in their perceptions. As future work, we want to diferentiate between closely related concepts such as credibility (more subjective) or correctness (more factual) and study how they afect users judgements.
The authors thank: i) the financial support supplied by the Xunta de Galicia - Consellería de Cultura, Educación, Formación Profesional e Universidades (Centro de investigación de Galicia accreditation 2019-2022 ED431G-2019/04 and Reference Competitive Group accreditation 20212024, ED431C 2022/19) and the European Union (European Regional Development Fund - ERDF) and ii) the financial support supplied by project PID2022-137061OB-C22 (Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; supported by the European Regional Development Fund).
The third author thanks the financial support obtained from project SUBV23/00002 (Ministerio de Consumo, Subdirección General de Regulación del Juego).
The authors also thank the funding of project PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU). and mitigating online misinformation, IEEE Transactions on Computational Social Systems (2023). [17] A. L. Ginsca, A. Popescu, M. Lupu, et al., Credibility in information retrieval, Foundations and Trends in Information Retrieval 9 (2015) 355–475. [18] D. H. McKnight, C. J. Kacmar, Factors and efects of information credibility, in: Proceedings of the ninth international conference on Electronic commerce, 2007, pp. 423–432. [19] M. Viviani, G. Pasi, Credibility in social media: opinions, news, and health information—a survey, Wiley interdisciplinary reviews: Data mining and knowledge discovery 7 (2017) e1209. [20] B. J. Fogg, C. Soohoo, D. R. Danielson, L. Marable, J. Stanford, E. R. Tauber, How do users evaluate the credibility of web sites? a study with over 2,500 participants, in: Proceedings of the 2003 conference on Designing for user experiences, 2003, pp. 1–15. [21] M. S. Eastin, Credibility assessments of online health information: The efects of source expertise and knowledge of content, Journal of Computer-Mediated Communication 6 (2001) JCMC643. [22] C. N. Wathen, J. Burkell, Believe it or not: Factors influencing credibility on the web,
Journal of the American society for information science and technology 53 (2002) 134–144. [23] S. Sikdar, B. Kang, J. ODonovan, T. Höllerer, S. Adah, Understanding information credibility on twitter, in: 2013 International Conference on Social Computing, IEEE, 2013, pp. 19–24. [24] S. K. Sikdar, B. Kang, J. O’Donovan, T. Hollerer, S. Adal, Cutting through the noise:
Defining ground truth in information credibility on twitter, Human 2 (2013) 151–167. [25] S. Zimmerman, A. Thorpe, C. Fox, U. Kruschwitz, Privacy nudging in search: Investigating potential impacts, in: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, 2019, pp. 283–287. [26] N. S. Chok, Pearson’s versus Spearman’s and Kendall’s correlation coeficients for continuous data, Ph.D. thesis, University of Pittsburgh, 2010. [27] J. Schwarz, M. Morris, Augmenting web pages and search results to support credibility assessment, in: Proceedings of the SIGCHI conference on human factors in computing systems, 2011, pp. 1245–1254. [28] E. R. Carlson, Evaluating the credibility of sources: A missing link in the teaching of critical thinking, Teaching of Psychology 22 (1995) 39–41. [29] M. Kattenbeck, D. Elsweiler, Understanding credibility judgements for web search snippets, Aslib Journal of Information Management (2019).