Modern column-store databases are perfectly suited for carrying out IR experiments, but they are not widely used for IR research. A plausible explanation would be that setting up a database system and populating it with the documents to be ranked provides enough of a hurdle to never get started on this route. We took up the OSIRRC challenge to produce an easily replicable experimental setup for running IR experiments on modern database architecture. OldDog, named after a short paper on SIGIR proposing the use of column-stores for IR experiments, implements standard IR ranking using BM25 as SQL queries issued to MonetDB SQL. This provides a baseline system on par with custom IR implementations and a perfect starting point for the exploration of more advanced integrations of IR and databases. Reflecting on our experience in OSIRRC 2019, we found a much larger efectiveness penalty than anticipated in the prior work for using the conjunctive variant of BM25 (requiring all query terms to occur). Simplifying the SQL query to rank the documents using the disjunctive variant (the normal IR ranking approach) results in longer runtimes but higher efectiveness. The interaction between query optimizations for eficiency and the resulting diferences in efectiveness remains a research topic with many open questions.
• Information systems → Search engine indexing; Evaluation of retrieval results. information retrieval, replicability, column store
OldDog is a software project to replicate and extend the
database approach to information retrieval presented in
Mühleisen et al. [
OldDog represents the data using the schema proposed by
Mühleisen et al. [
Like [
After initialisation, document ranking is performed by issuing SQL queries that specify the retrieval model. Interactive querying can support the researcher with additional examples. 2
robust4, core18
init, index, search, interact
The OldDog docker image itself consists of bash/python ‘hooks’ that wrap the underlying OldDog, Anserini and MonetDB commands.
Anserini builds a Lucene index, where we are happy endusers of the utilities provided to index common test collections. The code has been tested for Robust04 and Core18 in our ifrst release; extending to other collections readily supported by Anserini should be trivial.
OldDog provides the Java code to convert the Lucene index created by Anserini into CSV files that are subsequently loaded into the MonetDB database. OldDog further contains the Python code necessary to call Java code to pre-process the topics (calling the corresponding Anserini code, to guarantee that topics are processed exactly the same way as the documents) and issue SQL queries to the MonetDB database to rank the collection. 1https://lucene.apache.org, last accessed June 26th, 2019. 2The intermediate step exporting and importing CSV files is not strictly necessary, but simplifies the pipeline and is robust to failure.
Apart from the required init, index and search hooks, OldDog supports the interact hook to spawn an3 SQL shell that allows the user to query the database interactively. 2.1
Consider an example document doc1 with contents I put on my shoes after I put on my socks. to illustrate the database schema. Indexing the document results in tables 1, 2 and 3:
OldDog implements the BM25 [
OldDog implements disjunctive query processing as well, included after noticing a surprisingly large diference in efectiveness when compared to other systems applied to Robust04. In the disjunctive variant, documents are considered when they contain at least one of the query terms. As expected, runtimes for an evaluation increase when using this strategy.
Table 4 summarises efectiveness scores for both methods on two test collections, Robust04 and Core18.
Listing 1 shows the conjunctive BM25 SQL query for robust topic 301: International Organized Crime. The disjunctive variant simply omits the having clause. 3Or, ‘a SQL shell’, pronouncing SQL as sequel like database folk do.
Kamphuis and de Vries Overall, we look back at an excellent learning experience taking part in the OSIRRC challenge. The setup using Docker containers worked out very well during coding, by multiple people on diferent machines. Automated builds on Docker Hub and the integration with Zenodo complete a fully replicable experimental setup.
The standardised API for running evaluations provided by the ‘jig’ made it easy to learn from other groups; and mixing version management using git (multiple branches) with building (locally) Docker containers with diferent tags allowed progress in parallel when implementing diferent extensions of the initial code (in our case, including disjunctive query processing and adding Core18).
Recording the evaluation outcomes at release time let us catch a bug that would have been easily overlooked without such a setup - after including Core18, a minor bug introduced in the code to parse topic files lead to slightly diferent scores on Robust04, that we could easily detect and fix (one topic was missing) thanks to the structured approach of recording progress.4 Let us conclude the paper by discussing a few advantages of database-backed IR experiments. Using the interact hook, it is possible to issue SQL queries directly to the database. This is useful if one wants to try diferent kinds of ranking functions, or just to investigate the content of the database. We show some examples of queries on the Robust04 test collection.
The three most occurring terms are easily extracted from the dict table:
SELECT ∗ FROM dict ORDER BY df DESC LIMIT 3; +--------+-------+--------+ | termid | term | df | +========+=======+========+ | 541834 | from | 355901 | | 563475 | ha | 320097 | | 894136 | which | 302365 | +--------+-------+--------+
As expected, the term distribution is skewed with a very long tail; consider for example the number of distinct terms that occur only once:
SELECT COUNT(∗) AS terms FROM dict WHERE df = 1; +--------+ | terms | +========+ | 516956 | +--------+
Apart from applying a brief static stopword list to all preprocessing (defined in StandardAnalyzer.STOP_WORDS_SET), Anserini ‘stops’ the query expansions in its RM3 module by ifltering on document frequency, thresholded at 10% of the number of documents in the collection.
Having such a collection-dependent stoplist would be an interesting option in the initial run as well, so let us use the interactive mode to investigate the efect on query efectiveness of applying this df filter to the initial run.
We can easily evaluate the efect of removing the terms with high document frequency, e.g. by modifying the dictionary table as follows:
ALTER TABLE dict RENAME TO odict; CREATE table dict AS SELECT ∗ FROM odict WHERE df <=
(SELECT 0.1 ∗ COUNT(∗) FROM docs); 4We may also conclude that a unified topic format for all TREC collections would be a useful improvement to avoid errors in experiments carried out on these test collections.
We find the efectiveness scores shown in table 5 for disjunctive BM25. Performance drops for both MAP and early precision, suggesting that filtering query term presence based on document count is not a good idea, and should be limited to pseudo relevance feedback (not yet implemented in OldDog).
Core18 MAP P@30 0.1907 0.2693
A natural next step is to include the ‘qrel’ files in the
database, to explore more easily the relevant documents that
are (not) retrieved by specific test queries.
We conclude that we could successfully apply the methods
from [
This work is part of the research program Commit2Data with project number 628.011.001 (SQIREL-GRAPHS), which is (partly) financed by the Netherlands Organisation for Scientific Research (NWO).
We also want to thank Ryan Clancy and Jimmy Lin for the excellent support with the ‘jig’ framework.