We here focus on Points of Interest (POIs) Recommender Systems (RSs), aimed at helping users visiting a city to discover new and relevant POIs. RSs are often assessed in ofline settings, hence, measuring the system's precision in predicting previously observed user behaviour. However, when deployed, the system produced recommendations are often of limited use, because they lack novelty. We conjecture that this phenomenon is primarily due to the limited capability of RSs in extracting from the observed behaviour general characteristics of POIs that are relevant for diferent classes of users (tourist types). We compare an Inverse Reinforcement Learning (IRL) based RS algorithm with more traditional Nearest Neighbour and Popularity-based ones. Through an ofline evaluation, we show that the nearest neighbour and popularity-based RSs excel in precision (ofline) and are perceived as not novel by users of a live-user study. On the contrary, despite a lower ofline precision, the IRL-based RS, which learns the preferences of tourists for POIs characteristics, can give a better support to a tourist.
This paper focuses on Recommender Systems (RSs) in the tourism domain and summarises the
results of Massimo and Ricci on Points of Interest (POIs) recommendation [
In [
In order to investigate these questions we also proposed a novel and adaptable RS, called Q - P O P
P U S H , derived from Q - B A S E and that generates recommendations that simultaneously optimise
two criteria: the reward of the recommendations (as for Q - B A S E ) but also their popularity. Q
P O P P U S H computes the harmonic means of the scores given by Q - B A S E to a POI visit and the
popularity of the POI (visit occurrences in the observed data). To weigh the relative importance
of Q - B A S E original score and the popularity-based score, Q - P O P P U S H uses a parameter . With
> 0.5 the popularity of the POI-visit has a higher importance, whereas with < 0.5 the Q - B A S E
component is weighted more. Equal importance is given with = 0.5 . We have tested Q-POP
PUSH in an ofline experiment by comparing its performance with Q - B A S E , a popularity baseline
and two Nearest Neighbour next-item RSs: S K N N and s - S K N N [
The rest of this paper summarises the ofline experiment, the user study and concludes with a discussion.
The ofline experiment was conducted along the classical Machine Learning evaluation approach,
with train and test sets [
In Table 1 we report the metric values for Top-5 recommendations, obtained in the ofline evaluation. S K N N and s - S K N N perform essentially the same on the reward, precision and popularity * There is a significant diference ( < 0.05 ) between the best performing model and SKNN (two-tailed paired t-test). The test is run for the metrics Rew, Prec and Pop for 5 repeated train-test split. metrics. Not surprisingly, s - S K N N produces, on average, recommendations lists that overlap up to 53% to the list of SKNN (SimKNN ). When comparing SKNN to POP, we note that the overlap of the recommendations is quite low, but still, POP has a reasonably good precision, considering the simplicity of the approach. Q - B A S E suggests much less popular items than SKNN and also with higher reward.
The analysis of Q - P O P P U S H performance allows to address the research question RQ1. By introducing a popularity bias to Q - B A S E , as it is done in Q - P O P P U S H , the generated recommendations become more similar to S K N N . In fact, when is increased, the popularity of the recommendations produced by Q - P O P P U S H rises, it reaches and passes that of S K N N . However, it is interesting to note that with a rather small value of = 0.009 Q - P O P P U S H obtains a precision of 0.062, which is very close to the precision of S K N N (0.068) and s - S K N N (0.063). With this setting Q - P O P P U S H has still a positive reward (0.02), while S K N N and s - S K N N have negative rewards. This means that a small popularity bias in Q - B A S E could be beneficial to improve the system’s precision. We must also note that with larger values of the reward of Q - P O P P U S H is becoming smaller and smaller and approaching that of the two nearest neighbour methods. Clearly, Q - P O P P U S H can be tuned to balance two objectives: precision and reward.
To answer the second research question RQ2 we have designed a live user study aimed at measuring the user’s perceived novelty and appreciation of the recommendations generated by Q - B A S E , Q - P O P P U S H , with parameter = 0.5 (to give equal importance to POIs’ popularity and reward) and the S K N N baseline. We have developed an online system to assess the quality of next-POI recommendations ofered to users who have already visited some POIs. The system initially asks to declare visited POIs, which are used to create a hypothetical itinerary the user is supposed to have already visited. Then, the system generates next-POI recommendations with the three tested RS algorithms, combines them in a unique list, and asks the user to evaluate them based on a description of the recommended items. The user does not know which algorithm recommends the displayed recommendations. The training data of the online system is the same we have used in the ofline study. The user evaluates each POI by marking it with one or more of the following labels: “I already visited it”, “I like it” for a next visit and “I didn’t know it”. The online user study participants were recruited via social media and mailing lists. Out of 202 users that accessed the application, we identified 158 reliable recommendation sessions. Table 2 shows the estimated probability that a user marks as “visited”, “novel”, “liked” or both “liked” and “novel” a POI recommended by a specific RS. Q - B A S E recommends POIs that are less likely to have been already visited by the user and more likely to be novel than those suggested by Q - P O P P U S H and S K N N . As in the ofline experiment, Q - P O P P U S H and S K N N performs similarly. Interestingly, Q - B A S E suggests fewer POIs that are liked when compared to the other two strategies. Hence, apparently a more precise RS, based on an ofline test, also recommends online items that the user will like more. Besides, the obtained results falsify our hypothesis that optimising the reward of a recommendation, as Q - B A S E do, will produce recommendations that the user will like more. However, Q - B A S E suggests more novel POIs and, interestingly, more recommendations that are both liked and novel (last column in Table 2).
In a successive analysis we computed the probability that a user will like a recommendation given the following three conditions: she knows the POI but has not yet visited it; she has already seen it; the item is novel (unknown). We have derived the following conclusions. The users liked more the novel POI-visit suggestions generated by S K N N and Q - P O P P U S H than those produced by Q - B A S E . This is due to the tendency of Q - B A S E to suggest POI visits that even if they have the properties typically liked by the user, e.g., they are of the same type of the POIs liked by the user, they are also not popular POIs. Hence, these suggested POIs are hard to be appreciated. The conclusion we derived from the user study is that users tend to like more the items they are familiar with, e.g., previously visited items or items that are not novel.
The results of our experiments seem to confirm that RSs that precisely predict the user choices (ofline) are also liked most by real users. In our case, this means that Q - P O P P U S H and S K N N are better RSs than Q - B A S E . Our explanation of this result is that both high ofline precision and large probability of liked recommendations (online) are influenced by the popularity of the recommended items. In fact, these popular items are often in the users’ test sets, and users are likely to be familiar with them.
Despite its lower ofline precision performance and a lower extent of liked recommendations in the user study, Q - B A S E is the RS that may better accomplish the true goal of a tourism RS: it suggests more next-POIs that are both liked and novel. So, by optimising the reward Q - B A S E is capable of discovering novel items that are also appreciated (when users are able to assess them).