With emerging of structured data, retrieving entities instead of documents becomes more prevalent in order to satisfy the information need related to a query. Therefore, several high-performance entity retrieval methods have been introduced to the Information Retrieval (IR) community in recent years. Replicating and reproducing the standard entity retrieval methods are considered as challenging tasks in the IR community. Open-Source IR Replicability Challenge (OSIRRC) has addressed this problem by introducing a unified framework for dockerizing a variety of retrieval tasks. In this paper, a Docker image is built for six diferent entity retrieval models including, LM, MLM-tc, MLM-all, PRMS, SDM, FSDM. Also, Entity Linking incorporated Retrieval(ELR) extension, has been implemented that can be applied on top of all the mentioned models. The entity retrieval docker can retrieve relevant entities for any given topic. Image Source: https://github.com/osirrc/entityretrieval-docker Docker Hub: https://hub.docker.com/r/osirrc2019/entityretrieval
In the past two decades, search engines have been dealing with
unorganized and unclassified data i.e., unstructured data until the
emergence of semantic search. In order to satisfy the information
need behind a query using structured data, retrieving
machinerecognizable "entities" has proven to be a suitable complementary
for document retrieval for multiple reasons. For instance, Returning
a document in response to a query that is looking for an entity might
not be the best option because users have to look into the document
to find their desired information need. That could be one of the
main reason why retrieval operations are getting more and more
entity-centric, particularly on the web. Document retrieval difers
from entity retrieval in a couple of senses. In document retrieval,
entities are usually used for query expansion or retrieval features in
order to improve learning-to-rank frameworks and consecutively
to enhance document retrieval performance. On the other hand,
entity retrieval is defined as searching for an entity in a knowledge
base where entities are first class citizens[
In this docker image, the following standard entity retrieval models have been implemented:
• LM [
Furthermore, an extension of the Markov Random Field (MRF)
model framework for incorporating entity annotations into the
retrieval model, which is called Entity linking incorporated
Retrieval(ELR) has been leveraged on top of mentioned retrieval model.
Applying ELR on the state-of-the-art entity retrieval models results
in having the following ELR-integrated entity retrieval models [
DBpedia version 3.9 has been used as the knowledge base in the entity retrieval tasks in this Docker image. A term-based index and an entity-based index had created from the knowledge base by utilizing Lucene. Within the former index, entities URI objects are resolved to terms and the default Lucene stop words have been removed from them. Meanwhile, within the latter, URI objects are preserved and literal objects are ignored. In total, entity-based index contains 3,984,580 entities. While the term-based index is used in the standard entity retrieval model, both term based and entity based indices are used in ELR entity retrieval models.
One of the critical components of the ELR approaches is entity
annotations of the queries. TAGME, which is an open source entity
linker, has been adopted to annotate entities in queries with the
default threshold 0.1. Hasibi et al.[
In summary, given a query Q and a retrieval model M, the entity retrieval operates according to Figure 1, where D is the representation of entities. In section 2, there will be a more in-depth elaboration on standard entity retrieval methods in addition to the combination of them with ELR extension. Section 3 provides more details on the technical design aspect of the docker image. Section 4 describes our motivation and experience in participating in the OSIRRC 2019 challenge. The last section, i.e., section 5, gives an insight on further work that has to be carried out in this area and conclude the paper. 2
As was mentioned in the Overview section, several standard
retrieval models including LM, MLM-tc, MLM-all, PRMS, SDM and
FSDM have been implemented in this docker. In addition, ELR
extension can be applied on top of them, which results in having
twelve diferent retrieval models collectively. There will be two
different representation of entities; the term-based representation and
the entity-based representation. For the standard retrieval models,
term-based representation of DBpedia collection (using term-based
index) is used and when it comes to ELR extension. On the other
hand, both term-based and entity-based representation of entities
in DBpedia are used (term-based and URI-based indices). The
termbased and entity-based representations are compared in Figure 2
[
Retrieval models can be categorized into Language modelingbased models and Sequential Dependence models and ELR models. 2.1
Language modeling-based models consider dependencies among
query terms. LM [
Sequential Dependence Models (SDM) are popular Markov Random Field based retrieval models. Given a document D and a query Q, the conditional probability of P(D|Q) is estimated based on Markov Random Field as in equation (1) where D is term-based representation of entities.
rank P (D |Q) =
In equation (1), C(G) is set of cliques in graph G . The nodes of the
graph G consist of query terms and documents and the edges among
nodes illustrates the dependencies among nodes. λc is weight of
the feature function f(c) . More details can be found in the original
paper [
Considering dependencies among query terms results in having equation (2) based on Markov Random Field (equation (1)) as SDM ranking function with respect to λT + λO +λU = 1 : (1) (2)
P (D |Q) rank = λT Õ 2.2.1 Fielded Sequential Dependence Models (FSDM) .
Fielded Sequential Dependence Models (FSDM) considers
document structure by computing linear interpolation of probability of
each documents’ fields.Thus, feature functions are also calculated
based on field representation of documents. In other words, in FSDM
model [
Incorporating Entity Linking into entity Retrieval leads to improve
entity retrieval performance [
The entity retrieval Docker image is consisted of init, index
and search hooks invokes by Python3 as its interpreter. The jig
triggers the init hook first and then index and search respectively
in the Docker image. Finally , if there golden standard for the topics
are available, it will evaluate the results. Since we can get data into
and out of the container built from the Docker image [
This section describes diferent components of the docker image and supported hooks and extra options which can be passed to the jig for the entity retrieval Docker. 3.1
The latest oficial version of Ubuntu 2 is installed in the Docker with all the required commands. In addition, compatible versions of other requirements such as java8, Apache Ant, Apache Ivy, g+, and so on are installed on the Docker. Making all the components compatible with each other was a quite challenging issue. Since installing all the requirements is a time-consuming step, a docker image is prepared with all the basic requirements and pushed it to Docker Hub3. Hence, this prepared image is used as our Dockerfile base image in order to decrease the building time of the Docker. This sets the stage for COPYing the init,index and search hooks which should be executable files. 3.2
DBpedia version 3.9 4 has been used as the corpus of entity retrieval Docker image. In order to reduce the run time cost of the docker, the original index is used. Both term-based indexed and URI-based indexed of the collection will be downloaded once in the preparation step of the jig. However, to make the docker run using "the jig", a dummy collection is needed to pass. 3.3
This section elaborates on the role of each hook separately. init and index hooks are triggered in the jig preparation step and search hook script will run in the jig search step.
3.3.1 init.
The actual implementation of the retrieval models is cloned in this hook from the GitHub repository5. The required compatible packages are installed. Running this hook may take a while because of downloading,building and installing PyLucene which is time-consuming. Once the installation are done, two indexed collection which are DBpedia term-based index and URI-based index are downloaded (∼ 18GB) and extracted.
3.3.2 index.
The indexed DBpedia collection is already downloaded in the init hook. Hence, nothing is happening in this hook in this docker.
3.3.3 search.
When the image is prepared, retrieval models can run with respect
to their relevant customized parameters in the search hook. The
search hook runs the main implementation of models6 which were
provided by Hasibi et al.[
All the queries in the jig e.g, topics of Robust04, ClueWeb09, ClueWeb12, Gov2, Core17, Core18 and etc. are supported in this Docker. Any other queries is acceptable in this docker as long as each query is represented in the following format "query number or name:query terms" in each line. An instance of a topic file would be like: wt09-1:obama family tree wt09-2:french lick resort and casino wt09-3:getting organized ...
If relevant entities (qrel) are available for the associated topics, the jig will utilize Trec Eval 7 to evaluate the retrieval performance by diferent metrics such as MAP. If there is no ground truth ranked list of entities for the topic, a dummy qrel file is needed to pass to the jig in order to make the Docker run by the jig. 4
The crucial role of repeatability, replicability, and reproducibility
cannot be neglected in any research domain; especially when it
comes to practical experiments. Deciding to participate in this
challenge was easy because either it is repeating your computation,
replicating another researchers’ experiments or reproducing other
team’s research with a totally diferent setup, there will be an
endeavored struggle. This docker tackles all these 3 challenges for
entity retrieval. Not only the docker is built by a non-author of the
main paper (replicability) [
Dockerizing the entity retrieval models was a challenging task. furthermore, standardizing the Docker with the jig increased its complexity. One of the main issues regarding this Docker was the compatibility of diferent components e.g, Python, PyLucene, Java, Apache Ant, etc. It was a time-consuming task to find all the compatible version of all those components and this is one of the critical benefits of this task. Utilizing the entity Docker, Researchers do not have to spend lots of time on combining and connecting diferent packages, libraries and components anymore to run the mentioned entity retrieval models.
Another issue is that topics appear in diferent formats. we must be able to work with every topic format available in the jig. Therefore a standard topic format is defined in section 3.3.3 so that any topic can be used in this Docker.
For the methods with ELR extension, the annotation step has
to be added to the code that was implemented by Hasibi et al. [
The more reproducible and replicable research papers are, the more baselines will be accessible for researches to compare their results with. This means, by increasing repeatability, reproducibility,and replicability researchers can can spend less time on implementing other researchers work. Therefore, they will have more time spending on their own research and not on implementing the baselines. Consecutively, studies would make progress faster. Hence, more works needed to be done in this specific area.
According to entity retrieval docker image, this work can be
extended by supporting more collection such as DBpedia-entity v2
[
Furthermore, entity retrieval models can be added to the Docker. In terms of entity retrieval applications, it can be used to expand queries in order to improve document retrieval performance.
To sum up our work, Docker image is wrapped around the jig introduced for Open-Source IR Replicability Challenge 2019. This platform provides a unified framework for diferent retrieval task. Entity retrieval Docker image contains implementation of six diferent entity retrieval model. ELR extension also can be applied on any of the models. All models can be customized with desired parameters and there is no limit in the supported topics. This docker image is implemented based on very lightweight Linux-centric design to tackle repeatability, reproducibility and replicability problem in the IR domain.
Thanks to Faegheh Hasibi for her valuable suggestions during the implementation of the Docker image and preparation of this paper.