In this paper, we present the initial ndings about a possible geometric interpretation of the BM25 model and a comparison of the BM25 with the Binary Independence Model (BIM) on a two-dimensional space. A Web application was developed in R to show an example of this geometric view on a standard TREC collection. The application is accessible at the following link: http://gmdn.shinyapps.io/shinyRF04 wi0 =
The Binary Independence Model (BIM) [
ri + R + + ; qi =
N
qi)
pi)
ni
;
ri +
R +
+
where pi (or qi) is the probability that a relevant (or non-relevant) document
contains the term ti. The estimates of these probabilities are:
where ri is the number of relevant documents that contain ti, ni the number of
documents that contain term ti, R and N the number of relevant documents and
the total number of documents, respectively. The parameters and are used to
smooth pi and qi in order to avoid arithmetical anomalies (in [
The BM25 model goes one step further by introducing the frequency of the
term and the length of the document in the weight of the term ti [
tfi
tfi + K
where tfi the frequency of the term ti in the document, and K is a function of
some parameters about the global statistics of the collection of documents:
K = k1 ((1
b) + b dl= )
? This work is an extended abstract of [
In this paper, we want to study the problem of the optimisation of the
parameters of two models, the BIM and the BM25, and to show a direct comparison
of the two models by means of a visual interpretation of probabilities based on
the idea of Likelihood Spaces [
The BIM ranks documents according to the probability of relevance (R = 1)
given a document d and a query q, P (R = 1jd; q). This probability can be
approximated by the sum of the weights wi de ned in Eq. 1 (see [
1 {z x pi pi
X log ti } |
qi 1 qi {z y } > 0 (5) (6) (7) which can be interpreted as a ranking line (or a decision line) y < M x + Q. This formulation allows us to study the problem on a two-dimensional space where documents are represented by two coordinates shown and ranking can be optimised according to the parameters of the decision line. 1 http://gmdn.shinyapps.io/shinyRF04 2 http://trec.nist.gov/data/t13_robust.html
P (R = 1jd; q)
X ti2d\q wi The BM25 ranking formula can be expressed with the same sum over the terms that appear in the document and the query, replacing wi with wi0 of Eq. 3.
In the two-dimensional representation of probabilities, we keep P (R = 1jd; q)
distinct from the probability of a document being not relevant P (R = 0jd; q).
With some algebraic manipulation (see [
M X ti |
pi 1 {z x pi } +Q
X ti |
qi 1 qi {z y } > 0
Description of the Interface The Web application that we developed takes into account many important parameters which characterise the two retrieval models: { we can change the smoothing parameters and and see how the probabilities pi and qi change (for both BIM and BM25); { we can decide whether the ranking is computed over the terms of the document or the terms of the query, and study the di erences when pseudorelevance feedback is available (both for BIM and BM25). { we can change the BM25 parameters k1 and b; { we can change the proportion of training documents to estimate pi and qi and change the number of terms of the vocabulary (both BIM and BM25); { we can adjust the decision line by changing the angular coe cient M (and the intercept Q which does not a ect ranking).
The main window is split into two parts: the sidebar with the interaction widgets on the left and the main panel with the output on the right (in Fig. 1 we show only half of the interface for space limits). 3 Interaction The user can interact with the following widgets 1. Select the topic of interest from the drop-down menu. 2. Select the retrieval model (if BM25 is not selected, the BIM is on). 3. Change the value of the parameters and . 4. Choose the number of folds that we need to compute the probabilities of the terms of relevant and non relevant documents. 5. Change the parameters M and Q of the ranking line.
Visualization The main panel is divided into two columns: the rst column shows the results on the validation set, the second column (not shown in the gure) the results on the test set. Both columns contain the following pieces of information: the text box shows the total number of objects used for validation and the number of positive examples (red points, the pseudo-relevant documents of the chosen topic). The table shows performance measures in terms of precisionat-j (j = 5, 10, 20, 100, 500, 1000). The two-dimensional plot shows in red the relevant documents of the chosen topic (pseudo-relevant for validation, true relevant for test) and in black all the other documents of the collection. 4
Description of the Interface In this paper, we have presented a Web application developed in R which allows users to interact with two retrieval models, the BIM and the BM25 models, on a standard TREC collection. The two models are projected on a two-dimensional space based on the idea of Likelihood spaces. The interactive application shows, in a real machine learning setting, how the human pattern recognition capabilities can immediately detect whether the model is close to the optimal solution or not. We believe that this interactive approach may be a crucial step in setting the initial parameters of an automatic procedure that optimises these parameters. 3 http://gmdn.shinyapps.io/shinyRF04