Gianni Claudio and Giovanni Amati1, Carpineto1, Romano1
2.3 Deviation from randomness (DFR) t t2q^d 2.1 Okapi 2.2 Statistical language modeling (SLM) (1 b) + b t
5 Results = 1, = 0.5 of the three weighting modesl, one without and one with retrieval feedback (RF). measured using average precision (AV-PREC), precision at 5 retrieved docugreater than the early precisions; this is due to the fact that for these collections not always achieve the best results (i.e., for PREC-AT-5 and PREC-AT-10 on For each collection we show in bold the best result without retrieval feedback Table 1, Table 2, and Table 3 show the retrieval performance of each method the mean number of relevant documents per query is, on average, small, and that there are many queries with very few relevant documents. Note that for the French and Italian collections the average precision was and the best result with retrieval feedback. excellent performance of the DFR model is conrmed also when comparing the weighting models without query expansion, although in the latter case DFR did ments (PREC-AT-5), and precision at 10 retrieved documents (PREC-AT-10). with retrieval feedback, with notable improvements on several data points. The Italian, and for PREC-AT-5 on Spanish). For each collection and for each query, we computed six runs: two runs for each on the French, Italian, and Spanish collection, respectively. Performance was The rst main nding of our experimen ts is that the best absolute result for each collection and for each evaluation measure was always obtained by DFR
Okapi + RF SLM + RF SLM DFR DFR + RF attainable by the three methods (with retrieval feedback) for each query. The Thus, the length of each bar depicts the range of performance variations Fig. 1. Performance variation on individual queries for French Fig. 3. Performance variation on individual queries for Spanish Fig. 2. Performance variation on individual queries for Italian These results seem to suggest that the retrieval performance of a weighting across dieren t experimental situations, in that we chose simple and untuned investigation. The next step of this research is to experiment with a wider range fective than both Okapi and SLM, which achieved comparable retrieval permodel is only moderately aected b y the choice of the language, but this hyspecic experimen tal conditions. the length of the queries used in the experiments. parameter values and made typical indexing assumptions, the issue needs more pothesis should be taken with caution, because our results were obtained under esis that the retrieval performance of a weighting model is independent of the absolute results for any evaluation measure and across all test collections. of factors, such as the length of queries, the values of each weighting model’s would also be useful to experiment with other languages, to see if the hypothparameters, and the combination of parameter values for retrieval feedback. It formance. In particular, DFR with query expansion obtained the best average The main conclusion of our experiments is that the DFR model was more efformance of Okapi and DFR, whereas it was often detrimental to the retrieval Although there are reasons to believe that similar results might hold also language receives further support. The second conclusion is that retrieval feedback always improved the pereectiv eness of SLM, although the latter nding ma y have been inuenced b y obtained by DFR and Okapi was similar, but, on the whole, DFR was ranked over the other two models. The superiority of DFR over Okapi and SLM was To get a more complete picture, we counted, for each collection, the number of The results, shown in Table 4, conrm the better retriev al eectiv eness of DFR ahead of Okapi for a much larger number of queries. the other two test collections. For French and Italian, the number of best results does not tell us which method performed best. queries for which each method achieved the best, median, or worst performance. clear for Spanish, while DFR and Okapi obtained more comparable results on results show that the intermethod variations on sigle queries was ample, but