Reproducibility is a central aspect of ofline as well as online evaluations, to validate the results of diferent teams and in diferent experimental setups. However, often it is dificult or not even possible to reproduce an online evaluation, as solely a few data providers give access to their system, and if they do, it is limited in time and typically only during an oficial challenge. To alleviate the situation, we propose STELLA: a living lab infrastructure with consistent access to a data provider's system, which can be used to train and evaluate search- and recommender algorithms. In this position paper, we align STELLA's architecture to the PRIMAD model and its six diferent components specifying reproducibility in online evaluations and illustrate two use cases with two academic search systems.
Reproducibility1 is still an open issue in TREC-style IR ofline
evaluations. Hanbury et al. [
In theory EaaS might enhance reproducibility by keeping the data, algorithms, and results in a central infrastructure that is accessible through a standard API and allows for sharing open-source software components. However, the live environment for evaluating experimental systems typically has the consequence that the results are not reproducible since the users’ subjective impression of relevance is very inconstant. This makes reproducibility of online experiments more complicated than their ofline counterpart. To what extent online experiments in living labs can be made reproducible remains a central question. Although the user interactions Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). OSIRRC 2019 co-located with SIGIR 2019, 25 July 2019, Paris, France. 1One can diferentiate between repeatability (same team, same experimental setup), replicability (diferent team, same setup), and reproducibility (diferent team, diferent setup). For the sake of simplicity, we use the term reproducibility to refer to all of these three types. and all rankings generated by the systems can be stored and used for subsequent calculations, there are no clear guidelines as to how the logged interaction data can contribute to a valid and reproducible evaluation result. The major problem is that the recorded interactions are authentic only for the particular situation in which they were recorded.
Conceptionally, Ferro et al. [
To overcome these issues, we introduce STELLA (InfraSTructurEs for Living LAbs) a living lab infrastructure. STELLA allows capturing document data, algorithms, and user interactions in an online evaluation setup that is based on Docker containers. We align the diferent components of the infrastructure to the PRIMAD model and discuss their match to the model. We see a particular need to pay attention to the actor and data components. These components represent the human-factors like users’ interactions that afect the outcomes of online experiments and not only the experimenters’ perspective on the experiment.
In the following paper, we present the design of the STELLA living lab Docker infrastructure (cf. Section 2). We align STELLA’s components to the dimensions of the PRIMAD model and illustrate a use case with two academic search systems (cf. Section 3). Finally, we discuss the benefits and limitations of our work and conclude the paper (cf. Section 4). 2
The STELLA infrastructure allows researchers to evaluate search
and recommender algorithms in an online environment, i.e., within
a real-world system with real users. When using STELLA, the
researchers’ primary goal is to introduce ranking models for search
results and recommendations that outperform the existing
baseline. In the following, we describe STELLA’s workflow as well as
its technical infrastructure. We adhere to the wording of TREC
OpenSearch [
STELLA’s main component is the central living lab API. It connects data and content providers (sites) with researchers in the ifelds of information retrieval and recommender systems (participants). When linked to the API, the sites provide data that can be used by participants for implementing search and recommendation algorithms, e.g., metadata about items, users, session logs, and click paths. Participants use the API in order to obtain this data from the sites and enhance their experimental system, e.g., by delivering personalized search results. Subsequently, this experimental system is integrated into the living lab infrastructure and made accessible to site users. Within the site, the submitted system is implemented automatically. The system is used within an A/B-testing or interleaving scenario, such that the system’s results are presented to the real users of the site. The users’ actions, e.g., clicking or selecting a ranked item which determines the click-through rate, is recorded and send back to the central living labs API. There, this data is aggregated over time in order to produce reliable results to determine the system’s usefulness for that specific site.
To support reproducibility, we employ the Docker technology in order to keep as many components as possible the way they were in the first experiment. This includes reusing the utilized software, tools being used to develop the method, and in a user-oriented study, usage data generated by the users. Within the framework, experimental retrieval and recommender systems are distributed with the help of Docker images. Sites deploy local multi-container applications running these images. Participants contribute their experimental systems by extending prepared Dockerfiles and code templates. The underlying infrastructure will assure the synchronization and comparability of experimental retrieval and recommender systems across diferent sites. See figure 1 for a schematic visualization of the framework.
Sites deploy a multi-container environment that contains the experimental systems. Queries by site users are forwarded to these systems. A scheduling mechanism assures even distribution of queries among the participants’ systems. The multi-container environment logs user feedback and forwards this usage data to the experimental systems. Likewise logged user feedback is sent to the central STELLA server where it is filed, and overall statistics can be calculated.
The main functionalities of the STELLA server are the administration and synchronization of infrastructural components. After the submission of extended template files, the STELLA server will initiate the build process of Docker images. The build process is triggered by new contributions or changes within the experimental systems. Participants and sites will be able to self-administrate configurations and have insights into the evaluation outcomes by visiting a dashboard service.
In previous living lab campaigns [
Participants develop their systems in local environments and submit their systems upon completion. They contribute their systems by extending Dockerfile templates and providing the necessary source code. The submission of systems can be realized with already existing infrastructures like online version control services in combination with the Docker Hub. 3
The PRIMAD model ofers orientation to what extent reproducibility in IR experiments can be achieved. Table 1 provides an overview of the PRIMAD model and corresponding components in the STELLA infrastructure. The platform is provided by our framework, which mainly relies on Docker and its containerization technology. Increasing retrieval efectiveness is the primary research goal. Sites, e.g., a digital library, and participants, i.e., the external researchers, benefit from the cooperation with each other using STELLA. Sites, for instance, may be interested in finding adequate retrieval and recommender algorithms, whereas participants get access to data from real-world user interactions. Both the implementation and the method is chosen by the participants. In ad-hoc retrieval experiments, solely the researcher would be considered an actor. In a living lab scenario, this group is extended by site users who afect the outcome of experiments. Finally, data will consist of logged user interaction as well as domain-specific text collections.
In the following, we present use cases of how the STELLA
infrastructure is concretely aligned to the PRIMAD components. Two
early adopters from the domain of academic search implement
our framework such that they are part of the STELLA
infrastructure, LIVIVO2 [
Platform: In STELLA the platform is implemented via a Dockerbased framework as shown in Figure 1. It connects the sites with the participants and assures (i) the sites’ flow of data for computing IR-models, (ii) deploying the participants’ IRmodels on the sites, and (iii) obtaining the usefulness of the 2https://www.livivo.de, Accessed June 2019 3https://www.gesis.org/en/home/, Accessed June 2019 deployed IR-model, e.g., via accumulated click-through rates. Both sites are configured in a way that they can interact with the central STELLA server, i.e., provide data on the sites’ content and users as well as their queries and interactions. Research Goal: The research goal is the retrieval of scientific datasets and literature which satisfy a user’s information need. This includes retrieval using string-based queries as well as recommending further information using item-based queries. In LIVIVO, the research goal is finding appropriate domain-specific scientific literature in medicine, health, nutrition, and environmental and agricultural sciences. Besides scientific publications in the social sciences, the GESIS-wide Search ofers to search for research data, scales, and other information. The research goal also includes finding appropriate cross-item recommendations as recommending research data based on a currently viewed publication.
Method and Implementation: The participant chooses both the method and implementation. With the help of the Dockerbased framework, participants are free to choose which methods for retrieval to use as well as the methods’ implementation as long as the interface guidelines between the sites and the Docker images are respected.
Actor: The actors in online evaluations are (i) the potential site users and (ii) the participants that develop an experimental system to be evaluated on a site. In both LIVIVO and GESIS-wide Search, the site users range from students to scientists as well as librarians. Specifically, the users of both sites difer by their domain of interest (medicine and health vs. social sciences), their experience, and the granularity of their information need, which can be trivial, but also very complex. In any case, the site users’ interactions are captured by STELLA, which allows observing diferences in their behavior. Participants submit their experimental systems using the Docker-based framework and receive the evaluation results from STELLA after some evaluation period. This way, they can adapt their systems and re-submit them whenever possible.
Data: The data comprises all information the sites are willing to provide to participants for developing their systems. It usually includes some structured data, such as database records containing metadata on research data and publications, as well as unstructured data like full texts or abstracts. LIVIVO uses the ZB MED Knowledge Environment (ZB MED KE) as a data layer that semantically enriches (by annotating the metadata with concepts from life sciences ontologies) the textual content of metadata from about 50 diferent literature resources with a total of about 55 Million citations. Additionally, LIVIVO allows users to register and maintain a personalized profile, including a watch list. The GESIS-wide Search integrates metadata from diferent portals into a central search index that uses a specific metadata schema based on Dublin Core. Additionally, each data record is enriched with explicit links between diferent information items like links between a publication and a dataset. This link specifies that the publication uses that particular dataset. As there is no option for users to register, GESIS-wide Search provides users’ session logs and click paths but no user profiles. The entire document corpora of LIVIVO and GESIS-wide Search is made available for participants.
In this living lab scenario P, R, I, and M would be conserved within the Docker container and the STELLA infrastructure. A → A′ and D → D ′ would remain as components that change in another system and another point in time. By recording the interaction and click data, we can try to preserve parts of A and D.
Compared to previous living lab campaigns, the Docker-based infrastructure results in several advances towards enhancing reproducibility in online evaluations. The most important advances are: Transparency: Specifying retrieval and recommender systems, as well as their specific requirements in a standardized way, may contribute to the transparency of these systems.
limited to the top-k queries. Even though this is an elegant solution, it may influence evaluation outcomes. Using locally deployed Docker applications, there is no need for this restriction anymore. Rankings and recommendations can be determined based on the complete corpus.
retrieval of rankings or recommendations might afect user behavior. Also, implementing workarounds like timeouts resulted in additional implementation efort for sites. By deploying local Docker images, these latencies are eliminated. Lower entrance barrier for participation: Participants can contribute already existing systems to the STELLA infrastructure by simply dockerizing them. By specifying the required components and parameters, the deployment procedure is less error-prone. Researchers can use software and programming languages of their choice. Sites solely need to implement the REST-API and set up the local instance of the Docker application. Letting Docker deploy the application, human errors are avoided, and eforts are reduced.
The benefits mentioned above come at a cost. Especially the following limitations have to be considered:
Central server: The proposed infrastructure relies on a central server. This vulnerability might be a target for malicious intents and is generally a single point of failure.
Hardware limitations: The experimental systems will be deployed at sites. Available hardware capacities may vary across diferent sites. Furthermore, the hardware requirements of experimental systems should be following the available resources of sites. For instance, participants contributing machine/deep learning systems should not outsource training routines to external servers. In the first place, we will focus on lightweight experiments in order to keep the entrance barrier for participation low.
User interaction data: While all interaction data within the STELLA infrastructure can be logged, a reuse-setup is not outlined yet and must remain as future work.
Recording usage and interaction data for later reuse are not
novel. An example of interaction data recorded to allow later
verification and simulation was the NewsREEL lab at CLEF 2017 [
NewsREEL Replay participants had access to a dataset comprising a collection of log messages analogous to NewsREEL Live. Analogous to the online evaluation, participants had to find the configuration with the highest click-trough-rate. By using the recorded data, participants experimented on a reasonable trade-of amid prediction accuracy and response time.
We present a living lab platform for the evaluation of online experiments and a Docker-based infrastructure which bridges the gap between experimental systems and real user interactions. Concerning the PRIMAD model, it is possible to assess reproducibility and corresponding components in our infrastructure proposal. Participants are free to choose which method and implementation to use and can rely on adequate deployment and environments. User interaction data will be logged and is accessible for optimizing systems and future applications.