In this work we describe a system to evaluate multiple pointof-interest recommendation systems. In this system each recommendation service will be exposed online and crowdsourced assessors will interact with merged results from multiple services, which are responding to suggestion requests live, in order to determine which system performs best. This work builds upon work done previously as part of the TREC Contextual Suggestion Track and describes plans for how the track will be run in 2015.
Many point-of-interest recommendation systems have been developed, each using di erent techniques for making recommendations. Often, when these systems are evaluated, they are operating on di erent datasets or di erent factors are compared. Having a framework to t such systems into and be able to compare them using fair, standardized techniques will help us determine which systems performed the best.
The TREC Contextual Suggestion track [
One disadvantage of this setup is that, for this track, topics are actually personal preferences provided by crowdsourced assessors who, after providing their preferences, have Copyright c 2015 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
ECIR Supporting Complex Search Task Workshop ’15 Vienna, Austria Published on CEUR-WS: http://ceur-ws.org/Vol-1338/. to wait weeks for attraction suggestions. This wait makes the task of assessing more di cult as judgement is broken over long period and assessors have to remember previous interactions with the system. Also, the longer the wait, the more di cult it is to get crowdsourced assessors to return to the task.
In this article we will describe a setup that allows for attraction recommendation services to be compared with users issuing requests where suggestions are made live. Here participating recommendation services will have an online system which is able to respond to a suggestion request immediately. When a user (or an assessor) is ready for suggestions they make a request. The system will then send that user's pro le and the name of the city to services. Suggestions will be made by multiple recommendation services and the returned results will be merged and presented to the user. The user will then interact with the results which provides feedback on how good each service's suggestions are. A score for each service is continuously updated until the experiment ends.
We will describe the interface recommendation services need to implement in order to participate, how they will be compared, and some challenges that moving from a batchstyle to live evaluation setup introduces. 2.
Point-of-interest recommendation is an area several
researchers are perusing. Braunhofer et al. [
In addition to this framework being built upon the
Contextual Suggestion track our work is also inspired by work
done during the plista dataset challenge [
In order to develop a framework for testing point-of-interest recommendation services we need to determine an interface that users will use to communicate with them. As parameters for each recommendation request the systems take in the user's pro le (see Section 4 for a description of the pro le) and, as context, the city which the user wants recommendations for. We could also gather more contextual information about our users during each request, for example, a more precise location, who the user is travelling with (family, friends, alone), etc. However, currently, we only use the city and pro le as input. The result returned to users will consist of an ordered list of attractions that the service thinks the user will like.
This is similar to the information participants in previous iterations of the track had available to them but instead of getting a batch of pro les and cities and returning a batch of results, services will recieve a single pro le and city for each request. Additionally, instead of having a xed pro le for each user, on each request the pro le may be updated with liked suggestions from previous requests. 4.
This leads us to the question of what a user's pro le consists of. Initially, for a new user, no information is in their pro le. As the user asks for suggestions and interacts with them their pro le will expand. For each attraction that has been recommended to the user the data about their interaction is added to their pro le. Examples of interaction include the user viewing the attraction's website, the user \starring" the attraction, and the user rating the attraction. Currently, these three pieces of information are recorded for each attraction the user has interacted with, however other interactions, e.g., reviews written, could also be included in the pro le. As the user interacts with the system their pro le will expand giving recommendation services a better opportunity to make more personalized suggestions.
Once interaction data from an attraction has been added to the user's pro le it is essentially available publicly to all services. One issue with this setup is that certain requests will have small pro les and certain requests will have larger pro les. Limiting the size of the pro le will allow us not to worry as much about how the size of the pro le a ects service performance. One option to resolve this is, instead of adding every piece of interaction to a user's pro le, only add certain attractions. One simple method of doing this is to only add attraction interaction data for a subset of cities. This will also allow us to ask for suggestions for the same city multiple times (if that city is not part of the pro le) and necessarily not have to return to users for result interactions.
Another potential feature is to push any updates to the pro les to services. This will allow them to get feedback into how well they are performing and whether the suggestions they are making are actually liked by users without having to pool pro les or wait for another request from the same 1http://www.kaggle.com/ user. Services can then update their strategies and attempt to improve suggestion results for future requests. 5.
In previous iterations of this track services were allowed to recommend any attraction they found on the open web. For simplicity, the points-of-interest that services are allowed to recommend in this experiment come from a xed collection of attractions. Services will simply return a list of attraction IDs. When they are displayed to users each attraction will consist of a title, short description, and website URL with more information about the attraction. Users will use this information to make a decision about whether they like a particular attraction. Again, here we are presenting this basic information about each attraction but additional information, such as the attraction's category or reviews about the attraction, could also be presented to users.
This pool of attractions is collected as part of an ongoing e ort by several research groups who have expertise in dealing with gathering this sort of information due to participation in previous Contextual Suggestion TREC tracks. Having a xed data collection will allow us to limit which attractions are suggested and will allow for greater reusability of the judgements provided by users.
So, in order to develop a recommendation service that ts into this framework services must set up a server that responds to suggestions requests with a list of attraction IDs. The goal of forcing services into this framework is so that we can compare the performance between multiple services. Instead of having users communicate directly with a recommendation service they will communicate with an intermediary system. Services will be required to register themselves with this system and the system will then pass recommendation requests to each service, logging the services' responses.
Each service will be given an opportunity to make suggestions. One option to do this is, for each request, select one of the services at random and present the results from 0.2 0.0
A
B
C Services
D
E the service to the user. The user will then interact with the system and based on this interaction we can determine how good the suggestions were. As more suggestion requests are made each service will be given multiple chances to make recommendations and services can be compared.
We take a slightly di erent approach where, for each suggestion request a subset of the services are queried and the results from all these services are merged into a nal list of suggestions which is presented to users. If the user interacts with a suggestion the score of the service that made that suggestion will be a ected. It is possible for multiple services to make the same suggestion, in this case if the user interacts with the suggestion all the services that made it will have their score a ected.
This setup will give services more opportunities to make
suggestions (during each request rather than only some),
however we will need a method of merging requests from
multiple services. Multiple result interleaving approaches
have been discussed by Radlinski and Craswell which would
be appropriate for our purposes [
The reason a subset of the services rather than all services are selected is that, realistically, most users will only view the attractions near the top of the list. If we try to compare too many services the user may not see any suggestions from some of the services (or only see few suggestions from each service). In order to prevent this situation ve services are chosen for each suggestion request (this number is chosen somewhat arbitrarily). We also limit the number of suggestions each service can make per request to 50.
Again, we have our suggestion services which produce ranked lists of suggestions. Each suggestion request will be sent out to multiple services and user's will interact with a list of merged responses. The user's interaction with the suggestions will allow a score to be calculated for each service. For each point-of-interest interacted with we can calculate a usefulness score based on how highly the user rated it and whether the user visited the website or \starred" it. We also collect timing data for each session so we can incorporate how long users spent on each attraction into our scoring metric. Precision at rank k, mean reciprocal rank, and a modi ed version of time-biased gain have been used in previous iterations of this track and can be used here as well.
Since we are not involving each service in each suggestion request we need to choose which services to involve. The simplest way to choose is to pick services randomly, however we should keep our end goal in mind here. Our goal is to nd the correct ordering of services in terms of performance. So, for example, after a certain amount of requests have been made, if a particular service performs much more poorly than any other service we are not likely to learn more information about the correct ordering of services if we pick it as often as other services. On the other hand, if two services have very similar performance it may be worthwhile to pick them more often in order to determine which of the two services perform better. In Figure 2 we already know that service A performs poorly and there is probably more to gain by comparing services B and C.
We should also note that we are only interested in telling the di erence between two systems if they have enough of a di erence between them. If one service performs better than another but an end user would not realistically be able to tell the di erence between them then it is not worthwhile spending a bunch of resources determining their correct ordering. In Figure 2 services D and E perform so similarly that it is probably not worth comparing them.
We leave this issue of selecting services based on their current ranking for future work and for now simply select systems for each request randomly. It is worth nothing that we only expect a handful of services to register for this system initially and they can all probably be involved in every or most suggestion requests.
In previous iterations of this track services waited until the experiments were done to recieve feedback on how well they performed. An option being explored for this experiment is to provide scores or a leaderboard for services every so often so that services can see how well they are performing and use that feedback to improve their service throughout the experiment. 8.
So far we have been discussing a system which allows users
to interact with di erent recommendation services. Because
we don't have an existing userbase to run these experiments
on we will use paid assessors to interact with the system in a
similar way to real users. In past iterations of this track we
have found crowdsourced workers to be useful in these sorts
of tasks. Additionally Ageev et al. were successful in
simulating search interaction data with crowdsourced workers
[
Additionally, once the system has been set up and services are registered and running, the setup can provide value to real users who can continue to use the system outside of the track experiments. This will allow services to continue to receive feedback on their performance even outside of TREC.
When a request is made the user is expecting a response within a short amount of time. Services will have to be always available and be able to respond quickly. Once requests have been sent out to services, if a response takes too long to be returned that service will not be given an opportunity to contribute to the nal list of suggestions presented to the user. In order to help services maintain responsiveness they will be allowed to register multiple servers. For each request one of the service's servers will be chosen to respond to the request. This will provide some robustness to the system should a particular server become unavailable. Additionally we can optionally incorporate each service's respond time into their score and have e ciency in uence the ordering of services.
As an additional fallback mechanism, in case no service responds to a particular request, a baseline service will be developed that is always available and responds to every request. If no service responds the results from the baseline service will be presented to users. This ensures that users always receive some response. This fallback mechanism will gather its results from a commercial web service. 10.
The goal of each service is to select points-of-interest that the service predicts the user will like. These suggestions' titles, descriptions, and URLs are displayed to the user. The descriptions about each attraction shown to users are generic descriptions for that attraction. Services may want to modify the descriptions slightly in order to include, for example, why this particular user may nd the attraction interesting. Services will be given an opportunity to provide personalized descriptions for each attraction in order to include this kind of information. Evaluation for these descriptions will be done separately from the main evaluation into service performance.
In previous iterations of this track every service had to provide descriptions for all suggestions. The decision to make this an optional task was based on feedback that most services were simply providing generic descriptions, which in this experiment we are providing instead. Generating the generic descriptions ourselves will provide us with another point of standardization between services to allow for more fair comparisons. 11.
Currently the interface that users use to make suggestions requests and interact with service results is a web based interface. Users will select a city from a list and then be presented with merged results from multiple systems. This allows for crowdsourced assessors to easily provide system feedback. However, the system is designed so that other methods of presenting results to users could easily be used. In particular the API allows any developer to build a mobile application which enables users to interact with the system. Point-of-interest recommendation lends itself to mobile users and having multiple vectors for users to interact with the system is one of the future goals for this project.
The source for this project is currently available online: https://github.com/akdh/entertain-me.
We have brie y given an overview of a system that is used to evaluate multiple point-of-interest recommendation services live using crowdsourced workers. This system will be used to run the TREC 2015 Contextual Suggestion Track. This experiment di ers from previous years because services will have to available online during the experiment, suggestions will have to be delivered live, and assessment and evaluation can be a lot more uid. If you are interested in registering your service for this experiment you can nd out more about participating on the TREC website2 and the Contextual Suggestion Track website3. 13.