In this extended overview paper we discuss the first Living Labs for Information Retrieval Evaluation (LL4IR) lab which was held at CLEF 2015. The idea with living labs is to provide a benchmarking platform for researchers to evaluate their ranking systems in a live setting with real users in their natural task environments. LL4IR represents the first attempt to offer such experimental platform to the IR research community in the form of a community challenge. For this first edition of the challenge we focused on two specific use-cases: product search and web search. Ranking systems submitted by participants were experimentally compared using interleaved comparisons to the production system from the corresponding use-case. In this paper we describe how these experiments were performed, what the resulting outcomes are, and provide a detailed analysis of the use-cases and a discussion of ideas and opportunities for future development.
Evaluation is a central aspect of information retrieval (IR) research. In the past few
years, a new evaluation paradigm known as living labs has been proposed, where the
idea is to perform experiments in situ, with real users doing real tasks using real-world
applications [
The first edition of the lab focuses on two use-cases and one specific notion of what
a living lab is (with a view to expanding to other use-cases and other interpretations of
living labs in subsequent years). Use-cases for the first lab are: product search (on an
e-commerce site) and web search (through a large commercial web search engine). For
each of the two use-cases, challenge participants receive a set of frequent queries along
with candidate documents for each of these queries. For the product search use-case,
each document represents a product with attributes. This use-case also has actual query
strings and historical click log data. The other use-case, web-search, only provides
feature vectors. There is no content for documents nor query strings. The web-search
usecase represents a typical learning to rank setting [
Given the queries and documents, participants produce rankings for each query and
upload these rankings through an API. These rankings are evaluated online, on real
users, using an interleaved comparison method [
The official CLEF evaluation round took place between May 1 and May 15, 2015. The positive feedback and growing interest from participants motivated us to organize a subsequent second evaluation round, from Jun 1 to Jun 15, 2015. We will refer to these as Round #1 and Round #2 of the LL4IR Challenge, respectively. It is planned that further challenge rounds will be organized continuously (2 weeks every month) so that participants do not have to wait for an annual event to test their developments.4
In the next section we provide a brief background on online evaluation. In Section 3 we describe the LL4IR Challenge operation and give extensive details on our architecture. We then describe each of the two use-cases of the first edition of the lab in turn in Sections 4 and 5, and provide details and analysis of the submissions received. Finally, in Section 6, we conclude the paper. 2
In this section we firstly introduce related ideas regarding living labs in general. We then provide background on evaluation, and in particular on online evaluation. 2.1
Ideas for living labs for IR instances have been proposed over the last few years [
The need for more realistic evaluation, involving real users, was reiterated at several
recent IR workshops [
Our initiative is a first of its kind for IR. CLEF Newsreel [
Evaluation has always played an important role in IR research. Traditionally,
evaluation was performed offline following the Cranfield paradigm, using static test
collections [
Interleaved comparisons Many interleaving approaches have been proposed over the
past few years (for instance [
Our livings labs evaluation methodology heavily depends on a novel architecture which we describe in this section. We start with an overview, we then introduce the organization of our challenge, and we finish with implementation details. 3.1
For each of the use-cases, described in Sections 4 and 5, challenge participants take part in a live evaluation process. For this they use a set of frequent queries as training queries and a separate set of frequent queries as test queries. Candidate documents are provided for each query along with historical information associated with the queries. When participants produce their rankings for each query, they upload these through the provided LL4IR API. The commercial provider then interleaves a given participant’s ranked list with their own ranking, and presents the user with the interleaved result list. Participants take turns in having their ranked list interleaved with the commercial provider’s ranked list. This process of interleaving a single experimental system with the production system at a time is orchestrated by the LL4IR API, such that each participant gets about the same number of impressions. The actions performed by the users of the commercial provider’s system are then made available to the challenge participant (whose ranking was shown) through the API; i.e., the interleaved ranking, resulting clicks, and (aggregated) interleaving outcomes.
Figure 1 shows the Living Labs architecture and how the participant interacts with the use-cases through the LL4IR provided API. As can be seen, frequent queries (Q) with candidate documents for each query (DjQ) are sent from a site through the API to the experimental systems of participants. These systems upload their rankings (r0) for each query to the API. When a user of the site issues one of these frequent queries (q), then the site requests a ranking (r0) from the API and presents it interleaved with r to the users. Any interactions (c) of the user with this ranking are sent back to the API. Experimental systems can then obtain these interactions (c) from the API and D D|Q Q c r’ Experimental
PSPayarstrittciecipmipaannt t
API
D D|Q Q q c r’ q
c
SSiStiteiete
r
update their ranking (r0) if they wish. We provided participants with example code and
guidelines to ease the adaptation to our setup.8 Our evaluation methodology is described
in more detail in [
Head Queries The distribution of search queries typically follows a power law [
Much like any other (information retrieval or machine learning) evaluation campaign, we split our the data into training and testing parts. Given that we perform online evaluation, it is slightly more involved than simply providing disjoint datasets. We describe our setup here.
Training Phase During the training phase, participants are free to update their rankings using feedback information. This feedback information is made available to them as soon as it arrives at the API. Their rankings can be updated at any time and as often as desired. Both click feedback and aggregated outcomes are made available directly and are updated constantly.
Test Phase In the test phase, challenge participants receive another set of frequent queries as test queries. Again, the associated historical click information as well as candidate results for these queries are made available. After downloading the test queries, participants could only upload their rankings until the test phase started or only once after it started. These rankings are then treated in the same way as training queries. That
is, they are interleaved with the commercial providers’ rankings for several weeks. As for the training phase, in the test phase each challenge participant is given an approximately equal number of impressions. A major difference is that for the test queries, the click feedback is not made available. Aggregated outcomes are provided only after the test phase ends.
As detailed before, the official CLEF evaluation phase took place between May 1 and May 15, 2015. The positive feedback and growing interest from participants motivated us to organize a subsequent second evaluation round, from Jun 1 to Jun 15, 2015. We will refer to these as Round #1 and Round #2 of the LL4IR Challenge, respectively. It is planned that further challenge rounds will be organized continuously (2 weeks every month) so that participants do not have to wait for an annual event to test their developments. 3.3
The overall evaluation of challenge participants is based on the final system performance, and additionally on how the systems performed at each query issue. The primary metric used is aggregated interleaving outcomes, and in particular we are interested in the fractions of winning system comparisons.
Interleaved comparisons In Section 2.2 we provided details on how interleaving,
and in particular Team Draft Interleaving (TDI) [
Aggregated Outcomes We report the following aggregated interleaving metrics, where Outcome serves as the primary metric for comparing participants rankings. These aggregations are constantly updated for training queries. For the test phase they are only computed after the phase is over. #Wins is defined as the number of wins of the participant against the production system, where a win is defined as the experimental system having more clicks on results assigned to it by TDI than clicks on results assigned to the production system; #Losses is defined as the number of losses against the production system; #Ties is defined as the number of ties with the production system; #Impressions is the total number of times when rankings (for any of the test queries) from the participant have been displayed to users of the production system; and Outcome is defined as the fraction of wins, so #Wins/(#Wins+#Losses). An Outcome value below the expected outcome (typically 0.5) means that the participant system performed worse than the production system (i.e., overall it has more losses than wins). Significance of outcomes is tested using a two-sided binomial test which uses the expected outcome; p-values are reported.
Note that using these metrics, we are in theory only able to say something about
the relationship between the participant’s system and the production system. However,
Radlinski et al. [
Our living labs API, as shown in Figure 1 and described in Section 3.1, is a middleman for queries, rankings and clicks. The API talks to both sites (the use-cases) and participants (the researchers). In this paper, we only describe how it interacts with participants. The interaction with sites is described in detail in the documentation.9
Our API is implemented as a REST-full service, meaning that communication uses the HTTP protocol and that URLs describe resources. All communication is encoded in JSON. Table 1 lists all the endpoints participants can communicate with. Below we describe them in some more detail.
Queries Participants would typically start by downloading the head queries for each use-case. They can do so using this endpoint: GET /api/participant/query/(key)
f The only parameter the participant needs to provide is the key which is their API key. Note that participants will only receive queries for the use-case they signed up for.
Each query is marked with its type. A query can be a “train” or “test” query indicating the phase they belong to, see Section 3.2. The default query type is “train”. Queries have an identifier (qid) and, depending on the use-case, an actual query string (qstr). Participants should regularly update their query set. An example output would look like this: "queries": [ f "creation_time": "Mon, 10 Nov 2014 17:42:24 -0000", "qid": "S-q1", "qstr": "jaguar", "type": "train" g, ...] Candidate documents Secondly, for each query, a set of candidate documents is provided. In order to obtain these, this endpoint is used:
The set of documents is provided as a list of document identifiers. This list can change over time as new documents arrive or old documents go stale. See the example output: "qid": "S-q22", "doclist": [ f"docid": "S-d3" g, f"docid": "S-d5"g, f"docid": "S-d10"g, ...] "content": f"description": "Lorem ipsum dolor",
"short_description" : "Lorem", ...g "creation_time": "Sun, 27 Apr 2014 23:40:29 -0000", "docid": "S-d1", Note that depending on the use-case, relevance signals may also be provided for each document. See Section 5 for details.
Documents details Lastly, participants will need the actual content of documents. The endpoint we provide for this purpose is:
An example document would look like this: The content of the document can be a simple string, or potentially it can have structure. In which case it would be a JSON object. Note that depending on the use-case, these documents may be empty, again see Section 5 for details.
Run After the participant obtains queries, candidate document sets and the content of documents, runs (rankings) can be prepared. These runs can then be uploaded by the participant to the API using this endpoint:
An example run would look like this: "qid": "S-q22", "runid": "82" "creation_time": "Wed, 04 Jun 2014 15:03:56 -0000", "doclist": [ f"docid": "U-d4"g, f"docid": "U-d2"g, ... ] This run provides the query for which it is a ranking (qid) and it provides a runid. Note that the runid is only for the participants own bookkeeping. It could be any string, for instance a timestamp or the version of the ranker.
Once uploaded, the run is displayed to users (interleaved with the commercial provider’s results) when the given query is issued. If other participants also uploaded a ranking for this query, the run to show to the given user is sampled uniformly from the available runs.
Note that for test queries (cf., Section 3.2) the runs can only be submitted before a test phase starts. Our API will reject runs for test queries during the testing phase. Feedback Once runs are submitted and exposed to users, their interactions are logged. The interactions with their own runs can be retrieved by participants using the feedback endpoint:
An example of feedback returned by our API would look like this: f "feedback":
[ f "qid": "S-q1", "runid": "baseline", "type": "tdi", "doclist": [ f g, f "docid": "S-d1", "clicked": false, "team": "site", "docid": "S-d4", "clicked": true, "team": "participant", g g, ] g, ...] In this feedback, the ranking that was shown to the user is provided for train queries (cf., Section 3.2). The runid is simply the run identifier provided by the participant. The ranking in the above example was a TDI ranking (cf., Section 2.2), an interleaving of the participant’s run with the production ranking from the site. The team field shows to which team each document was assigned. The clicked field simply indicates whether the document was clicked. This feedback is per impression, so not aggregated. Note that there can be multiple feedbacks for the same runid. Participants can use the feedback as they see fit, but typically one would compute whether the site or the participant won the interleaved comparisons, see Section 2.2 for details on how to do this. Outcome Finally, all interactions of users with the runs from participants can be aggregated into outcomes. These aggregated results are provided as an API endpoint at:
An example outcome would look like this: f "type": "test", "test_period": f "start": "Fri, 01 May 2015 00:00:00 -0000" "end": "Sat, 16 May 2015 00:00:00 -0000", "name": "CLEF LL4IR Round #1", g, "impressions": 10, "losses": 3, "ties": 5, "wins": 2 "outcome": "0.4", g, ...] In this example, we see the aggregation of outcomes over 10 impressions of the runs from a single participant for a single query; the participant lost more impressions than it won. The query identifier qid can be omitted in which case the outcome would be aggregated over all queries. Outcomes are split out per train and test phase (cf., Section 3.2). In case the participant participated in multiple test phases with the same query, multiple aggregations are shown. 4
The product search use-case is provided by REGIO Ja´te´k (REGIO Toy in English), the largest (offline) toy retailer in Hungary with currently over 30 stores. Their webshop10 is among the top 5 in Hungary. The company is working on strengthening their online presence; improving the quality of product search in their online store is directed towards this larger goal. An excerpt from the search result page is shown in Figure 2.
As described in Section 3, we distinguish between training and test phases. Queries are sampled from the set of frequent queries; these queries are very short (1.18 terms on average) and have a stable search volume. For each query, a set of candidate products (approximately 50 products per query) and historical click information (click through rate) is made available. For each product a structured representation is supplied (see below). The task then is to rank the provided candidate set.
Product Descriptions For each product a fielded document representation is provided, containing the attributes shown in Table 2. The amount of text available for individual products is limited (and is in Hungarian), but there are structural and semantic annotations, including: – Organization of products into a two-level deep topical categorization system; – Toy characters associated with the product (Barbie, Spiderman, Hello Kitty, etc.); – Brand (Beados, LEGO, Simba, etc.); – Gender and age recommendations (for many products); – Queries (and their distribution) that led to the given product.
Candidate Products The candidate set, to be ranked, contains all products that were available in the (recent) past. This comprises all products that are considered by the site’s production search engine (in practice: all products that contain any of the query terms in any of their textual fields). One particular challenge for this use-case is that the inventory (as well as the prices) are constantly changing; however, for challenge participants, a single ranking will be used throughout the entire test period of the challenge, without the possibility of updating it. The candidate set therefore also includes products that may not be available at the moment (but might become available again in the future). Participating systems were strongly encouraged to consider all products from the 10 http://www.regiojatek.hu/
Fig. 2. Screenshot of REGIO, our product search use-case. provided candidate set. Those that were unavailable at a given point in time were not displayed to users of the REGIO online store. Further, it may happen (and as we show in Section 4.3 it indeed does happen) during the test period that new products arrive; experimental systems are not able to include these in their ranking (this is the same for all participants), while the production system might return them. This can potentially affect the number of wins against the production system (to the advantage of the production system), but it will not affect the comparison across experimental systems. While we do not want to get into technical details here, there are three specific issues worth mentioning regarding the interleaving implementation at the site’s end. First, search typically involves a pipeline of components: query analysis (including, e.g., spell correction), ranking, filtering (e.g., considering facets), and finally presenting the SERP (with pagination, etc.). It is important that interleaving takes place in the ranking component, before any filtering is applied. Second, the rankings submitted by participants may include products that are currently not available. These need to be removed before the interleaving is performed. Finally, the same user should be presented with the same (interleaved) ranking for a given query throughout her session (in case the query is issued multiple times); this caching mechanism needs to be implemented by the site. 4.3
One peculiarity of the product search use-case is that the inventory is subject to frequent changes. Not all products that are part of the candidate set are available at all s t rcodu 20 P # New arrival Became available Became unavailable 05−01 05−03 05−05 05−07 05−09 05−11 05−13
05−15
Days times. Also, new products may arrive over time that are not available to experimental systems but do get returned by the production system (and hence get clicked by users). We further note that new arrivals are displayed distinctively in the webshop, which may also introduce a bias. Figure 3 displays the number of new arrivals (in green), and the products that became available (blue) or unavailable (red) from the day before. Only products that are candidates for any of the queries (either training or test) are considered. This figure shows absolute numbers. It is apparent that changes do occur, and in particular a great number of new products arrive. (This is actually the least desired type of change, as there is no easy way of dealing with it in our current platform.) An even more revealing statistic would be to measure the ratio of products that were unavailable at a given day, compared to all candidate products that were ever available during the test phase. This is shown in Figure 4. Note that unavailability ratio is specific to a given ranking; the reported numbers are computed for the organizers’ baseline. To keep things simple, we use a single value, the average unavailability ratio of all submitted rankings, which is 44%.
If all products were available, the expected probability of winning an interleaved comparison (assuming a randomly clicking user) would be 0.5. However, on average, 44% of the products were actually unavailable. During Round #1, these products were only ever present in the participants ranking (the site’s ranking never considered them). And, only after interleaving were these products removed from the resulting interleaved list. We note that this is undesired behavior, as they should have been filtered out before interleaving. The necessary adjustments were made to the implementation for Round #2 of the challenge. As for interpreting the Round #1 results, this means that the chances for products from the participants ranking to be clicked were reduced. We believe that o it a r y ilit b a li a v a n U this in turn reduces the expected probability to win to:
P (participant > site) = (1 0:44) 0:5 = 0:28: Consequently, if a participant’s system wins more than in 28% of the impressions, then this is more than expected. And thus the participant’s system can be said to be better than the site’s system if the outcome is (significantly) more than 28%. Importantly, this applies only to Round #1. For Round #2, the probability of winning an interleaving comparison is 50%, therefore an outcome above 0.5 is needed to beat the production system. 4.4
Two organizations submitted a total of four runs for Round #1. For Round #2, three organizations submitted five runs in total. In addition, a simple baseline provided by the challenge organizers is also included for reference. Table 3 presents an overview of participating teams.
Team GESIS [
Team IRIT [
Team UiS [
The organizers’ baseline (BASELINE in Tables 4 and 5) ranks products based on historical click-through rate. Only products that were clicked for the given query are returned; their attributes are not considered. In case historical clicks are unavailable (this happened for a single query R-q97), (all) candidate products are returned in an arbitrary order (in practice, in the same order as they were received from the API via the doclist request, see Section 3.4). For Round #1, we find that at least three submissions are likely to have improved upon the production system’s ranking. That is, three of the four submissions performed higher than the expected outcome of 0.28, see Section 4.3. Somewhat surprisingly, the simple baseline performed by far the best, with an outcome of 0.4691. This was also the only system that significantly outperformed the production system. The best performing participant run is UIS-MIRA, with an outcome of 0.3413.
For Round #2, the issue with the interleaved comparisons was corrected, see
Section 4.3. The expected probability of winning an interleaved comparison (assuming a
BASELINE 0.4691
UIS-MIRA [
IRIT [
Apart from the fact that none of the submission systems outperformed the production system in this round, their relative performance also changed. Where UIS-JERN came second in Round #1, it won in Round #2, which is more in line with the expectation of the team that submitted this system and the one it overtook. Even more strikingly is the large jump in relative system performance for GESIS. This can most likely be explained by numerous fixes to problems encountered in Round #1. Other than this it is hard to draw any conclusions from these differences as almost all of them have extremely high p-values. We would require more data in order to obtain more stable results before we can draw any conclusions. 5
The web search use-case is provided by Seznam,11 a very large web search engine in the Czech Republic. See Figure 5 for a screenshot of the user interface.
Seznam serves almost half the country’s search traffic and as such has very high
site traffic. Queries are the typical web search queries, and thus are a mixed bag of
transactional, navigational and transactional [
As described in Section 3, the web search use-case also consists of a training and test phase. For the test phase, there were 97 queries. For the training phase 100 queries were provided. On average, for each query there were about 179 candidate documents. In total, there were 35,322 documents. 11 http://search.seznam.cz
Every time one of the selected queries is issued by a user for the first time, Seznam
makes a call to our API while simultaneously calling the production rankers. When our
API responds too slow (> 100ms) it is ignored completely and the production ranking
is shown. When our API does respond in time, the participants’ ranking is interleaved
with the production ranking. Seznam implemented Team Draft Interleaving (TDI) [
Over the past months, there have been over 480K impressions on Seznam through our Living Labs API. On average this amounts to 2,428 impressions for each query. Approximately 6% of all impressions were used for the testing period. As can be seen in Table 6, the EXPLOITATIVE BASELINE outperformed the production system. An outcome (outcome measure described in Section 3) of 0.5527 was achieved, with 3,030 wins and 2,452 losses against the production system, and 19,055 ties with it. As expected, the UNIFORM BASELINE lost many more comparisons than it won. Both outcomes were statistically significant according to a binomial test.
Results for Round #2 are qualitatively the same as for Round #1. There were approximately the same number of impressions during this period and again one baseline is better and one worse than the expected outcome. Both significantly so. Both baselines did improve upon themselves though. This is a rather surprising result as the baselines did not update their rankings after Round #1 and therefore it is likely that some documents whet stale while other new documents were not included at all. This should have resulted in reduced performance. We do not yet have a good explanation for why this may be the case. 6
The living labs methodology offers great potential to evaluate information retrieval systems in live settings with real users. The Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab represents the first attempt at a shared community benchmarking platform in this space. The first edition of LL4IR focused on two use-cases, product 12 If use-cases uploaded their candidate documents in the order that represented their own ranking, then this was available to participants. We plan to change this in the future. search and web search, using a commercial e-commerce website, REGIO, and a commercial web search engine, Seznam. A major contribution of the lab is the development of the necessary API infrastructure, which is made publicly available.
The LL4IR CLEF Lab attracted interest from dozens of teams. There were 12 active participants, but only 3 teams ended up submitting results for the official evaluation (excluding the baseline systems, provided by the organizers). We found that, while many researchers expressed and showed their interest in the lab, our setup with an API, instead of a static test collection, was a hurdle for many. We plan to ease this process of adapting to this new evaluation paradigm by providing even more examples and by organizing tutorials where we demonstrate working with our API.
Overall, we regard our effort successful in showing the feasibility and potential of this form of evaluation. For both use-cases, there was an experimental system that outperformed the corresponding production system significantly. It is somewhat unfortunate that in both cases that experimental system was a baseline approach provided by the challenge organizers, nevertheless, it demonstrates the potential benefits to use-case owners as well. One particular issue that surfaced and needs addressing for the product search use-case is the frequent changes in inventory. This appears to be more severe than we first anticipated and represents some challenges, both technical and methodological.
The API infrastructure developed for the LL4IR CLEF Lab offers the potential to host ongoing IR evaluations in a live setting. As such, it is planned that these “challenges” will continue on an ongoing basis post-CLEF, with an expanding number of use-cases as well as refinements to the existing use-cases.13 In fact, a third round of our evaluation challenge is already underway at the time of writing, with some modifications to the initial setup. Moreover, this third round has already attracted more participants, in particular for the web search use case. 13 See http://living-labs.net/challenge/ for details.
We would like to acknowledge the support of (in alphabetical order): the CLEF Initiative; the Dutch national program COMMIT; the ESF ELIAS network program; the REGIO Ja´te´k online toy store; and Seznam commercial search engine. We would also like to thank the participants for their submissions and interest in the lab.