In this paper we propose a method that integrates the notion of understandability, as a factor of document relevance, into the evaluation of information retrieval systems for consumer health search. We consider the gain-discount evaluation framework (RBP, nDCG, ERR) and propose two understandability-based variants (uRBP) of rank biased precision, characterised by an estimation of understandability based on document readability and by different models of how readability influences user understanding of document content. The proposed uRBP measures are empirically contrasted to RBP by comparing system rankings obtained with each measure. The findings suggest that considering understandability along with topicality in the evaluation of information retrieval systems lead to different claims about systems effectiveness than considering topicality alone.
Searching for health advice on the Web is an increasingly
common practice. A recent research has found that in 2012
about 58% of US adults (72% of all US Internet users – 66%
in 2011) have consulted the Internet for health advice [
Previous studies within health informatics have
investigated online consumer health information beyond
topicality to specific health topics; in particular, with respect to
the understandability and reliability of such information.
For example, Wiener and Wiener-Pla [
Research on the user perception of document relevance
has shown that users’ relevance assessments are affected by
a number of factors beyond topicality, although topicality
has been found to be the essential relevance criteria. For
example, Xu and Chen proposed and validated a five-factor
model of relevance which consists of novelty, reliability,
understandability, scope, along with topicality [
The integration of understandability within the
evaluation methodology is achieved by extending the general
gaindiscount framework synthesised by Carterette [
The proposed evaluation measure is further instantiated
by considering specific estimations of understandability based
on readability measures computed for each retrieved
document. While understandability encompasses other aspects
in addition to text readability (e.g., prior knowledge), the
use of readability measures is a good first approximation for
understandability. This choice is also supported by prior
work in health informatics regarding understandability of
consumer health information (e.g., see [
The impact of the proposed framework and the specific
resultant measures on the evaluation of IR systems is
investigated in the context of the consumer health search task of
CLEF eHealth 2013 [
We tackle the problem of jointly evaluating topicality and
understandability for measuring IR system effectiveness within
the gain-discount framework synthesised by Carterette [
N k=1
where g(k) and d(k) are respectively the gain and discount
function computed for the document at rank k,2 K is the
depth of assessment at which the measure is evaluated, and
1/N is a (optional) normalisation factor, which serves to
bound the value of the sum into the range [
Different measures developed within the gain-discount
framework are characterised by different instantiations of its
components. For example, the discount function in RBP is
modelled by d(k) = βk−1, where β ∈ [
Without loss of generality, we can express the gain
provided by a document at rank k as a function of its probability
of relevance; for simplicity we shall write g(k) = f (P (R|k)),
where P (R|k) is the probability of relevance given the
document at rank k. Note that a similar form has been used for
the definition of the gain function for time-biased evaluation
measures [
Traditional TREC-style relevance assessors are instructed to consider topicality as the only (explicit) factor influencing 2For simplicity of notation, in the following we override k to represent the rank position k, or the document at rank k: the context of use will determine the meaning of k. relevance, thus P (R|k) = P (T |k), i.e., the probability that the document at k is topically relevant (to a query). 2.2
As discussed by previous work, e.g. [
To compute the joint probability we assume that topicality and understandability are compositional events and their probabilities independent, i.e., P (T , U |k) = P (T |k)P (U |k). This is a strong assumption and its limitations are briefly discussed in Section 4. Following this assumption, the gain function in the gain-discount framework is expressed as: g(k) = f (P (R|k)) = f P (T |k)P (U |k) (2)
Different evaluation measures that may be developed within this framework would instantiate f P (T |k)P (U |k) in different ways. In the following we will propose two RBP-based instantiations; other instantiations are left for future work. 2.3
In the traditional TREC settings, assessments about the topicality of a document to a query are collected through manual annotation of query-document pairs from assessors (i.e., binary or graded relevance assessments3); these are then turned into estimations of P (T |k). This process may be mimicked to collect understandability assessments; in this paper however we do not explore this possibility. Instead, we explore the possibility of computing understandability as a property of a document and integrate this in the evaluation process, along with standard relevance assessments. To this aim, readability is used as a proxy for understandability. (The limitations of this choice are briefly noted in Section 4.) Its use is however justifiable because readability is one of the aspects that influence the understanding of text.
To estimate readability (and thus understandability), we
employ established general readability measures as those
used in [
The use of such general readability measures to assess the
readability of documents concerning health information has
been questioned [
cno 0.15 lite
frrcyscoeoo
0.10 iilts
ysedbaoned
fr
0.05 it
ity measures in this paper is a first step towards
demonstrating the use of the proposed understandability biased
measures and analyse how system rankings would change
accordingly. In addition, their usage is partially supported
by previous work within health informatics on assessing the
readability of online health advice [
Given the readability score for a document at rank k, P (U |k) needs to be estimated; this is achieved by considering user models that encode different ways in which a user is affected by document readability.
We first consider a user model P1(U |k) where a user is
characterised by a readability threshold th and every
document that has a readability score below th is considered
certainly understandable, i.e., P1(U |k) = 1; while documents
with readability above th are considered not understandable,
i.e. P1(U |k) = 0. This is a (Heaviside) step function centred
in th; this function is depicted in Figure 1 (P1(U |k)) with
th = 20, along with the FOG readability score distribution
for documents from CLEF e-Health 2013 [
To understand how accounting for understandability
influences the evaluation of IR systems tailored to searching
health advice on the Web, we consider the runs submitted to
the CLEF eHealth 2013 [
K sures (RBP) and understandability-based measures (uRBP1
uRBP1 = (1 − β) βk−1r(k)u1(k) (4) and uRBP2). System rankings are compared using Kendall
k=1 rank correlation (τ) and AP correlation [
A second user model (P2(U |k)) is proposed, where the been retired due to removal of duplicates and copyrighted probability estimation is similar to a step function, but smoothed documents; we thus use the CLEF eHealth 2014 collection in the surroundings of the threshed value; this provides a (which is a subset of the CLEF eHealth 2013 collection) to more realistic transition between readable and not-readable allow reproducibility of the reported results and the 2013 content: qrels for relevance assessment. For each document in the 1 arctan F OG(k) − th collection, the FOG readability scores (Equation 3) were P2(U|k) ∝ 2 − π (5) cCoLmEpFuteeHde a–tlthhe2s0c1o3reqrdeilsstrisibsuhtoiwonn ifnorFaiglludreoc1u.mTehnrtese itnhrtehseholds on the FOG readability values were explored for the computation of the two alternative formulations of uRBP: th = 10, 15, 20; documents with a FOG score below 10 should be near-universally understandable, while documents with FOG scores above 15 and 20 increasingly restrict the audience able to understand the text.
where arctan is the arctangent trigonometric function and F OG(k) is the FOG readability score of document at rank k; other readability scores could be used instead of FOG. The distribution of P2(U |k) values is shown in Figure 1. Equation 5 is not a proper probability distribution, but this can be obtained by normalising Equation 5 by its integral between [min F OG(k) , max F OG(k) ]; however Equation 5 is rank equivalent to such distribution, not changing the effect on the uRBP variant. These settings lead to the formulation of a second understandability-based RBP, uRBP2, based on the second user model, by simply substituting u2(k) = P2(U |k) to u1(k) in Equation 4.
Note that in both understandability-based measures (as well as in the original RBP) the contribution of an irrelevant document is zero, irrespective of its P (U |k). The contribution (to the gain) of a relevant document with readability score above th is 1 for RBP, 0 for uRBP1 and less than 0.5 for uRBP2 (for uRBP2 the score will quickly tend to 0 the more the readability score is above the threshold value).
Finally, note that it is possible to design other user models representing how readability inuflences document understandability; the challenge is to determine which model better represents the relationship between readability and document understanding.
Figure 2 reports RBP vs. uRBP of IR systems participating to CLEF eHealth 2013 for the two user models proposed in Section 2.4 and for the three readability thresholds considered in the experiments. Similarly, Table 1 reports the values of Kendall rank correlation (τ) and AP correlation (τAP ) between system rankings obtained with RBP and the two versions of uRBP.
Higher correlation between systems rankings obtained with RBP and uRBP is observed for higher values of th, irrespectively of uRBP version (see Table 1). This is expected as the higher the threshold, the more documents will be characterised by a P (U |k) = 1 (or ≈ 1 for uRBP2), thus reducing uRBP to RBP. The fact that in general uRBP2 is correlated with RBP more than uRBP1 is to RBP highlights the effect of smoothing obtained by the arctan function; specifically, the increase of readability scores for which P (U |k) is not zero
RBP RBP Figure 2: RBP vs. uRBP of CLEF eHealth 2013 systems (left: uRBP1; right: uRBP2) for varying values of threshold on the readability scores (th = 10,15,20).
RBP vs. uRBP1 RBP vs. uRBP2 th = 10 τ = .1277 τAP = −.0255
τ = .5887 τAP = .2877 beyond th narrows the scope for ranking differences between systems effectiveness. These observations are confirmed in Figure 2, where only few changes in the rank of systems are shown for th = 20 (× in Figure 2), while more changes are found for th = 10 (◦) and th = 15 (+). Note that the small differences in the absolute values of effectiveness recorded by uRBP with th = 10 should not be interpreted as a lack of discriminative power. When th = 10 only 1.4% of the documents in the CLEF eHealth 2013 qrels are relevant and readable, thus contributing to uRBP.
Figure 2 demonstrates the importance of considering understandability along with topicality in the evaluation of systems for the considered task. The system ranked highest according to RBP (MEDINFO.1.3.noadd) is second to a number of systems according to uRBP if user understandability of up to FOG level 15 is wanted. Specifically, the highest uRBP1 for th = 10 is achieved by UTHealth_CCB.1.3. noadd, which is ranked 28th according to RBP, and for th = 15 by teamAEHRC.6.3, which is ranked 19th according to RBP and achieves the highest uRBP2 for th = 10, 15.
In this paper, we have investigated how understandability can be integrated in the gain-discount framework for evaluating IR systems. The approach studied here is general and can be adopted to other factors of relevance, such as reliability. Information reliability plays an important role in consumer health advice search; its integration will be studied in future work.
In the proposed approach, the relevance (P(R|k)) was
modelled as the joint probability P (T , U |k). This joint
probability was assumed to be independent and the two events
to be compositional, thus allowing to derive P (T , U |k) =
P (T |k)P (U |k) and to treat topicality and understandability
separately. This is a strong assumption and it is not
necessarily true; alternatives are under investigation, e.g. [
The approach was demonstrated by deriving understandability-based variants of RBP; other measures can also be extended, e.g., nDCG and ERR. Note, however, that nDCGstyle versions would require normalising the gain function by the ideal gain, which in turns requires finding the optimal ranking based on two criteria, relevance score and understandability, instead of one as in the standard nDCG.
Xu and Chen [
In this paper, we have used readability as a proxy for
understandability, but this is only one aspect that influences
understandability [