How do we calculate how many relevant documents are in a collection? In this abstract, we discuss our line of research about total recall systems such as interactive system for systematic reviews based on an active learning framework [4-6]. In particular, we will present 1) the problem in mathematical terms, and 2) the experiments of an interactive system that continuously monitors the costs of reviewing additional documents and suggests the user whether to continue or not in the search based on the available remaining resources. We will discuss the results of this system on the ongoing CLEF 2019 eHealth task.
• Information systems → Clustering and classification ; Probabilistic retrieval models; • Applied computing → Health care information systems; Health informatics.
Given a collection of documents and an information need, can we estimate the number of relevant documents in the collection for that information need? This question seems trivial but, for some tasks, a suficiently accurate answer may help the user to save a lot of resources in terms of time and money. In fact, if we knew this number, we could stop the search process as soon as the last relevant document is found or we may decide to stop earlier if it is no longer convenient to continue the search.
The type of retrieval tasks that we refer to are, for example,
eDiscovery [
In this abstract, we want to discuss our line of research that
follows from the studies in interactive system for systematic reviews
based on an active learning framework [
In order to avoid confusion with similar topics in the IR research
ifeld, we want to stress the fact that we are not studying whether
the subset of relevant documents judged is suficient to compare the
accuracy of IR systems (like in TREC or CLEF) [
The paper is organized as follows: in Section 2, we present the problem in mathematical terms by means of the hypergeometric distribution to model the sampling of documents without replacement. In Section 3, we present a brief summary of the application of this approach to the ongoing CLEF 2019 eHealth task on Technology Assisted Medical Reviews. 2
We assume to have a collection of N objects which can be classified as either relevant or non-relevant. The number of relevant objects is K ; hence, the number of non-relevant documents is N − K . We draw n samples from the collection of objects without replacement and we want to compute the probability of observing k relevant objects. This probability function is represented by the hypergeometric distribution:
P (X = k; N , K, n) = (1) K N −K k n−k
N n where X is the random variable with a hypergeometric distribution with parameters N , K , and n; ba represents the binomial coeficient.
For example, if we have a collection of N = 100 objects with K = 20 relevant objects, the probability of observing k = 10 relevant objects by drawing n = 30 objects from the collection is P (X = 10; N = 100, K = 20, n = 30) = = 0.17
(2) If we could draw t times n samples from a hypergeometric distribution with parameters K and N , we would obtain a sampling distribution with mean y and standard error SE
1 Õt ki y = (3)
t i=1 ni
SE[y] = r sn2 1 − Nn (4) where s2 is the sample variance and 1 − Nn the finite population correction factor ([10, Chapter 2]).
We can use the sampling distribution to compute how accurate our estimates of the mean are by means of confidence intervals which define the lower and upper bound of our estimate: [y − zα /2SE[y], y + zα /2SE[y]] (5) where zα /2 is the (1 - α /2)th percentile of the standard normal distribution. For example, if we drew t = 10000 times n = 30 objects from the hypergeometric distribution with K = 20 and N = 100, we would obtain a sampling distribution with mean y ≃ 0.1994 and SE[y] ≃ 0.0094. If we wanted a 95% confidence interval ( α = 0.05), the range of the number of relevant objects would be between 18 and 22.
Our use case is building a system for systematic medical reviews which are a method to collect the findings from multiple medical studies in a reliable way. Given budget and time constraints, we need to provide the physician with a suficient amount of (possibly all) relevant medical studies.
In such real life cases, we do not have a perfect knowledge about the collection of documents at our disposal: we may or may not know the exact number of documents N in the collection, or the exact number of relevant documents K , or both. In our experiments, we assume the following: 1) we know N , and 2) we know (for example an “oracle” tells us) that there are “at least” K − relevant documents; we use a “minus” at superscript to indicate that this number is a lower bound for K . In other words, we know the total number of documents in the collection, but we have just a partial knowledge (a lower bound) on the number of relevant documents. Moreover, by “relevant” object we mean that some user has judged the document. Initially, we do not know whether K − is close or not to the “true” value K ; consequently, we want to estimate how costly it is to build a confidence interval for K − accurate enough to tell whether to stop the search of additional relevant documents. In our previous example, suppose that the oracle says that there are at least K − = 20 relevant documents in a collection of N = 100 objects. How many documents n do we need to draw (or read) to get a desired confidence interval?
The systems we propose uses a mixed approach to 1) find the
relevant documents in a collections of medical documents given a
query of a physician, and 2) compute the confidence interval of the
estimate of the number of relevant documents left in the collection.
On the one hand, we apply a Continuous Active Learning (CAL)
approach using the BM25 ranking model [
The system allows to adjust the proportion of documents that are sampled against those that are ranked in order to balance the costs of estimating the confidence intervals accurately versus finding the most relevant information as quick as possible. In particular: • we set a number of documents d that the user is willing to read and a number m that tells the algorithm when to randomly sample a document from the collection; • the first half of documents d/2 are used to continuously update the relevance weights of the terms according to the explicit relevance feedback given by the user; • for the second half documents we use a Naïve Bayes classifier to select the subset of documents to read.
We will discuss the latest version of this system that participated
in the last three editions of the eHealth CLEF 2019 lab and we
will give some suggestions about future work that include query
aspects [