Environmental Monitoring System of Natural Water
still waiting to be allocated or there is a problem with Elasticsearch). Conditions
This was a major pain point during development of our system and 947C0156B Kiev GIDROBIOLOGICHESKIY ZHURNAL in Russian
Vol 29 No 3, May-Jun 93 pp 88-95
lead to many unexpected issues.
At this stage, Elasticsearch is available for indexing documents. 947C0156B Kiev GIDROBIOLOGICHESKIY ZHURNAL
Documents in the collection path (provided by the jig) are trans-
formed by our TREC collection parser 4 and indexed in bulk. When
indexing different collections, we do not attempt any document
Language: Russian
cleaning or pre-processing (e.g., stemming). We generally index Article Type:CSO
titles into one field, then the content of the document from all other
fields (e.g., paragraphs, sub-headings) under the content part of [Article by S.V. Kreneva, Azov Scientific
Research Institute
the document (e.g., TEXT/body/contents, etc.) into a single text of Fishery Industry, Rostov na Donu; UDC [593.17:574.63](08)(28)]
field. Examples of how collection documents are transformed into a [Abstract] The widening gap between the rates at which new
contaminants appear the maximum...
representation suitable to be indexed in Elasticsearch are described
in detail in the following sections.
2.2.1 robust04. Documents in this collection are in a XML-like
format (in fact we parse them using a generic XML parser). One
aspect of this file format that caused the XML parser to fail are the (a) Contents of robust04 document FBIS3-61525.
unquoted attributes on elements (e.g., , the 100 should
be wrapped in quotation marks to be valid XML). To overcome this, {
"DocNo": "FBIS3-61525",
we simply removed all attributes from every element. When trans- "Text": "Language: Russian Article Type:CSO [Article
forming the files in the robust04 collection, we index everything in by S.V. Kreneva, Azov Scientific Research
the tag as a single string, ignoring structure. An example Institute of Fishery Industry, Rostov na Donu;
UDC [593.17:574.63](08)(28)] [Abstract] The
of this transformation is presented in Figure 1. widening gap between the rates at which new
contaminants appear the maximum..."
2.2.2 core17. The documents for core17 are in a standardised XML }
format. We take a similar approach to indexing these documents to
the robust04 documents. Unlike the robust04 documents, there is a (b) Elasticsearch representation of robust04 document FBIS3-61525.
clear title in the header of the core17 documents which we extract.
We then proceed to index everything between the tags as Figure 1: Example transformation of a document in robust04
a single string, ignoring structure. An example core17 document into a representation suitable to be indexed in Elasticsearch.
and what we transform it into as a JSON representation suitable
for indexing in Elasticsearch is presented in Figure 2. this issue, we simply cast all values to a string for this field before
indexing.
2.2.3 core18. The documents in the core18 collection are already
in a format suitable for Elasticsearch to index (i.e., they come as a
2.3 search
JSON object). One challenge we faced here, however is that the main
content of these documents is stored as an array of individual JSON The search hook is used to perform ad-hoc searches on a previously
objects, all with the same field name (content), but with different indexed collection. This hook uses TREC topics as queries, and as
data-types for the values. This issue is illustrated in Figure 3. This is such, these topic files for a collection must be parsed. Our topic
a problem because Elasticsearch infers the types of values to create parsing and searching tool 5 reads a topic file, issues the title of
a mapping based on the first document indexed. When Elasticsearch the topic as an Elasticsearch “query string” query, and parses the
encounters a value that does not conform to the data-type specified Elasticsearch API response into a TREC run file to be evaluated
by the mapping, it will fail to index the document. To overcome automatically by the jig.
4 https://github.com/osirrc/ielab-docker/tree/master/cparser 5 https://github.com/osirrc/ielab-docker/tree/master/tsearcher
58
ielab at the Open-Source IR Replicability Challenge 2019 OSIRRC 2019, July 25, 2019, Paris, France
{
"content": "Homicides remain steady in the Washington region",
"mime": "text/plain",
"type": "title"
ART: DAVID SALLE'S WORKS SHOWN AT THE WHITNEY }
(a) Data-type of content is a string.
{
"content": 1483230909000,
"mime": "text/plain",
"type": "date"
}
(b) Data-type of content is an integer.
ARTREVIEWS Figure 3: Example of JSON objects in the content section in a
ART SHOWS core18 document which contain different data-types for the
WHITNEY MUSEUM OF AMERICAN...SMITH, ROBERTA
same field (content).
SALLE, DAVIDReview python3 run.py prepare \
Top/Features/Arts --repo osirrc2019/ielab \
--collections robust04=/path/to/disk45=trectext
(a) robust04
python3 run.py prepare \
ART: DAVID SALLE'S WORKS SHOWN AT THE WHITNEY --repo osirrc2019/ielab \
--collections core17=/path/to/NYTcorpus=nyt
By ROBERTA SMITHSMITH, ROBERTA (b) core17
python3 run.py prepare \
--repo osirrc2019/ielab \
LEAD: THE exhibition of David Salle's work that has just ...
For American art of the 1980's, Salle's painting stands...
Figure 4: Prepare commands issued to the jig used to index
each collection in our Elasticsearch Docker image.
(a) Contents of core17 document 6135.
{
"DocNo": "6135", We are also aware of optional parameters in the jig that can be
"Title": "ART: DAVID SALLE'S WORKS SHOWN AT THE WHITNEY", used to pass configuration items into the Docker image. If we were
"Text": "ART: DAVID SALLE'S WORKS SHOWN AT THE WHITNEY By
ROBERTA SMITH SMITH, ROBERTA LEAD: THE exhibition
to exploit these optional parameters in future work, we would use
of David Salle's work that has just...For American them to specify aspects of Elasticsearch such as sharding settings
art of the 1980's, Salle's painting stands..." and document pre-processing using the index hook, or the fields
}
to consider when scoring documents or the retrieval model using
(b) Elasticsearch representation of core17 document 6135.
the search hook.
Figure 2: Example transformation of a document in core17 3 JIG ARGUMENTS
into a representation suitable to be indexed in Elasticsearch. Here we provide the arguments passed to the jig in order to index
and search the three collections compatible with our Docker image.
2.4 Unimplemented Hooks The following two sections describe the two jig commands and the
arguments we pass to them to obtain our baseline results.
The two unimplemented hooks are train and interact. As future
work, if the train hook were to be implemented, we would use it
to optimise retrieval function parameters (e.g., the K1 and B param- 3.1 prepare
eters of BM25, similar to work by Jimmy et al. [4]). Furthermore, The prepare jig command prepares a Docker image for performing
if the interact hook were to be implemented, we would use it experiments. This command calls the init and index hooks of the
to expose the REST API of Elasticsearch. This would allow one to Docker image. Figure 4 presents the arguments passed to the jig in
connect other tools such as Kibana, and explore the data further. order to prepare our Elasticsearch Docker image to run experiments.
59
OSIRRC 2019, July 25, 2019, Paris, France Harrisen Scells and Guido Zuccon
python run.py search \ The jig of the OSIRRC 2019 introduced a standardised framework
--repo osirrc2019/ielab \
--output out/ielab \
for anyone to implement hooks to a search system, to allow for
--qrels qrels/qrels.robust04.txt \ replicability. However, due to the lack of standardisation of the
--topic topics/topics.robust04.txt \ fine-grain parametrisation of the systems, it becomes difficult to
--collection robust04
control systems comparisons. What the jig introduced is a way to
(a) robust04
perform experiments, however, it lacked a standard way to config-
ure experiments. Many participants have exploited the optional
python run.py search \
--repo osirrc2019/ielab \ arguments for indexing and searching, however the problem now is
--output out/ielab \ that each participant uses different syntax and naming conventions
--qrels qrels/qrels.core17.txt \ for standard parameters (e.g., which retrieval model to use, whether
--topic topics/topics.core17.txt \
--collection core17 to perform stemming or not, etc.). We suggest that if this challenge
were to be run again, another hook be added which defines the
(b) core17 capabilities of a Docker image, which can include official, standard
python run.py search \ capabilities, such as the name and parameters of the retrieval model,
--repo osirrc2019/ielab \ what document pre-processing steps to take, or what fields to index
--output out/ielab \
--qrels qrels/qrels.core18.txt \
(which generally apply to all search systems), as well as self-defined
--topic topics/topics.core18.txt \ capabilities such as the compression algorithm to use (which may
--collection core18 only apply to a subset of systems). Currently, while it is limiting to
directly compare the evaluation results of our image to the images
(c) core18 of other participants due to the reasons listed above, it becomes
possible for others working with Elasticsearch to easily compare
Figure 5: Search commands issued to the jig used to search their system to this baseline. The current Docker images developed
each collection in our Elasticsearch Docker image. as part of this initiative only allow runs to be replicable and not
comparable. By listing the capabilities of systems, it would allow
Run MAP P@30 nDCG@20 one to fairly compare between two or more systems that share
capabilities. Although system comparison was not the goal of this
robust04 (BM25, k=1000) 0.1826 0.3236 0.3477
challenge, it could easily be facilitated by adopting solutions similar
core17 (BM25, k=1000) 0.0831 0.2333 0.1779
to that described above.
core18 (BM25, k=1000) 0.1899 0.2760 0.3081
The OSIRRC 2019 initiative was valuable from both the per-
Table 1: Results of runs for the robust04, core17, and core18 spective of implementing the jig hooks, and from comparing how
collections using our Elasticsearch Docker image. Each run others chose to implement them in their hooks. Despite the fact
uses the default Elasticsearch BM25 scorer for ranking, and that Elasticsearch is primarily a commercial/production product,
each run retrieves a maximum of 1,000 documents (k). and not a research oriented tool, there are several published aca-
3.2 search demic research papers that use it, e.g., those from our ielab research
team6 [4–6], and other groups [1, 3, 7]. This may be in part due
The search command issues queries from TREC topic files and to the ease with which one can index and search documents in
outputs a TREC run file which is evaluated by the jig. This command Elasticsearch (very little to no configuration is necessary, provided
calls the search hook of the Docker image. Figure 5 presents the an adequate parser is available). We hope that our contribution
arguments passed to the jig in order to run the search experiments to and participation in this challenge leads to further replicabil-
on the collections compatible with our Elasticsearch Docker image. ity and reproducibility of Elasticsearch and other search systems,
and that others can more easily compare their own Elasticsearch
4 RESULTS experiments to this common baseline.
Table 1 presents the evaluation results for each collection indexed
using our Elasticsearch Docker image. The evaluation measures REFERENCES
reported in this table are chosen based on the measures reported [1] Giuseppe Amato, Paolo Bolettieri, Fabio Carrara, Fabrizio Falchi, and Claudio
by the jig. Although all participants report the same evaluation Gennaro. 2018. Large-Scale Image Retrieval with Elasticsearch. In The 41st In-
measures it is important to note that one cannot fairly compare ternational ACM SIGIR Conference on Research & Development in Information
Retrieval (SIGIR ’18). ACM, New York, NY, USA, 925–928.
between systems and these results can only be used to demonstrate [2] Leif Azzopardi, Matt Crane, Hui Fang, Grant Ingersoll, Jimmy Lin, Yashar Mosh-
the replicability of our system. We elaborate on this detail in the feghi, Harrisen Scells, Peilin Yang, and Guido Zuccon. 2017. The Lucene for
information access and retrieval research (LIARR) workshop at SIGIR 2017. (2017).
following section, and present a possible solution. [3] Rafael Glater, Rodrygo L.T. Santos, and Nivio Ziviani. 2017. Intent-Aware Semantic
Query Annotation. In Proceedings of the 40th International ACM SIGIR Conference
5 DISCUSSION & CONCLUSIONS on Research and Development in Information Retrieval (SIGIR ’17). ACM, New York,
NY, USA, 485–494.
Our contribution at the OSIRRC 2019 is an Elasticsearch Docker [4] Jimmy, Guido Zuccon, and Bevan Koopman. 2016. Boosting titles does not gener-
image capable of indexing and searching the robust04, core17, and ally improve retrieval effectiveness. In Proceedings of the 21st Australasian document
computing symposium. 25–32.
core18 collections. While we had many difficulties getting Elastic-
search to work in this setup, and correctly parsing and indexing
documents, we were able to implement the main hooks for the jig. 6 http://ielab.io/
60
ielab at the Open-Source IR Replicability Challenge 2019 OSIRRC 2019, July 25, 2019, Paris, France
[5] Daniel Locke, Guido Zuccon, and Harrisen Scells. 2017. Automatic Query Gen- 26th ACM International Conference on Information and Knowledge Management.
eration from Legal Texts for Case Law Retrieval. In Asia Information Retrieval 2291–2294.
Symposium. Springer, 181–193. [7] Luca Soldaini, Arman Cohan, Andrew Yates, Nazli Goharian, and Ophir Frieder.
[6] Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, 2015. Retrieving medical literature for clinical decision support. In European
and Shlomo Geva. 2017. Integrating the framing of clinical questions via PICO| conference on information retrieval. Springer, 538–549.
into the retrieval of medical literature for systematic reviews. In Proceedings of the
61