Recommender systems are widely used tools in a large number of online applications due to their ability to learn the tastes and needs of the users. Venue recommendation approaches have recently become particularly useful, and even though these techniques have certain characteristics that difer from traditional recommendation, they deserve special attention from the research community due to the increase on the number of applications using tourism information to perform venue suggestions. In particular, how to properly evaluate (in an ofline setting) this type of recommenders needs to be better analyzed, as they are normally evaluated using standard evaluation methodologies, neglecting their unique features. In this paper, we discuss and propose some solutions to two specific aspects around this problem: how to deal with already interacted venues in the test set and how to incorporate the sequence of visited venues by the user when measuring the performance of an algorithm (i.e., in an evaluation metric).
The large development of location-based social networks (LBSNs)
in recent years has encouraged research on the problem of
Point-ofInterest (POI) or venue recommendation, i.e., suggesting new places
for users to visit by analysing diferent contexts such as interaction
patterns, friendship relationships, or geographical influence [
With this idea in mind, in this work we analyze some
important aspects we have detected related to how POI recommendation
tends to be evaluated in ofline settings. Our driving hypothesis is
that a recommender system should be evaluated in a situation as
close as that where it would be used. Because of this, we consider
that ofline evaluation should be performed by running a
temporal split, where the recommender should predict the present (or
future) user interactions based on her past interactions [
In classical recommender systems, no repetitions are typically
allowed or considered in the datasets, probably inherited by the
domains of the first available datasets (movies) [
Some papers explicitly state that they separate venues, instead
of check-ins, hence, in those cases it is clear that there are no
known or visited items in the test set by that user (see [
We argue that, by evaluating with items already interacted by
the user we are aiming at a diferent kind of algorithm than when
those items are removed. In other terms, a recommender system
that performs very well in the first scenario (with known items) is
expected to distinguish well which of the previously visited venues
the user will visit next. In this context, its final goal is to generate
recommendations already known by the user, probably the opposite
of a recommender evaluated with only new items in the test set, thus
aiming at recommending new, novel venues for each particular user
– in fact, some authors define explicitly such a task as recommending
new places [
Our assumption – that we would like to study in the future, since it is out of the scope of this position paper – is that those recommenders that better predict in the case of known venues, would probably generate less novel recommendations in general. We believe it would be interesting to understand this efect, in part, to improve current recommendation algorithms that perform well in either of these tasks by creating a hybrid algorithm useful in a real use-case scenario, in such a way that it would detect if new or already visited recommendations should be returned to a user, based on her previous interactions. 3
Another specific feature of venue recommendation that difers from
the more traditional recommendation problem is that the order in
which users visit the venues provides a lot of information. Recently,
some methods have been proposed that provide recommendations
based on temporal or sequential aspects, such as [
Furthermore, and related to the discussion presented in the previous section, classical ranking metrics fit the scenario with no repeated items, however, they cannot be adapted to the case where repetitions exist (at least, not to the case where there are repetitions in the test set). Because of this, we believe sequences should be formally integrated and considered when evaluating recommender systems in the venue recommendation context.
With this idea in mind, herein we propose an evaluation metric
based on the Longest Common Subsequence (LCS) algorithm, a
technique used to find a subsequence of elements (no necessary
consecutive) whose length is the maximum possible between two
sequences [
Finally, since the LCS between two sequences is not bounded, we need to normalize this value (lcs). We propose the following three variations when measuring rankings at cutof N of both recommended and test sequences: LCSP (Ru , Tu ) = lcs(Ru , Tu )/N (based on precision), LCSR(Ru , Tu ) = lcs(Ru , Tu )/|Tu | (based on recall), and LCS(Ru , Tu ) = lcs(Ru , Tu )2/(N · |Ru |).
The experiments have been performed using the global-scale
checkin dataset of Foursquare1 made public by the authors of [
We report results obtained by the following recommenders:
• Random (Rnd): random recommender.
• Popularity (Pop): recommender that suggests the most
popular items, i.e., items with more check-ins.
• AvgDis: baseline that recommends the closest POIs to the
user’s average location. The average is computed by
calculating the midpoint of the coordinates of the POIs visited by
the user.
• PGN: a hybrid approach similar to the USG model proposed
in [
Based on the temporal split presented in Table 1, we decided to focus on the 2 largest cities in terms of number of check-ins (Jakarta and Istanbul) and create 2 independent training-test datasets. Furthermore, in order to make a fair comparison among all the evaluated baselines, we removed repetitions in a user basis for the classical collaborative filtering algorithms; we kept two versions of the training set (with and without check-in frequencies) so that some POI recommendation algorithms, in our case AvgDis and IRenMF, could exploit the frequency of users when visiting a specific venue (denoted as AvgDisFreq and IRenMFFreq). Additionally, to test the experimental conditions discussed in Section 2, we created two test sets: one where those venues the user already interacted in the past (training set) are removed (with new venues) and another where they are kept (with known venues).
To evaluate the recommenders under the with known venues strategy we selected as candidates for each user all the venues that appear in the complete training set of each target city, while when working with new venues we remove the ones already rated by that user. 1https://sites.google.com/site/yangdingqi/home/foursquare-dataset
We use diferent ranking metrics to measure accuracy of the
recommenders: precision (P), recall (R), mean average precision
(MAP), and normalized discounted cumulative gain (NDCG) [
Table 2 shows the results for the cities mentioned before evaluated under the two methodologies presented in Section 2: where only new items for a user appear in her test set (with new venues) and where venues already interacted by the user are allowed in the test set (with known venues). Nevertheless, for the test set, we always removed the duplicated check-ins (i.e., the users only made one check-in in a POI). As a simple baseline, we have included a method that returns the venues observed in training for each user (Training), ordered by their score and popularity. This baseline, as expected, does not obtain any relevant result in the first scenario, however, when known items are allowed, it is a strong baseline to beat, and some of the more complex algorithms such as IRenMF tend to obtain performance values very close to the ones from this method.
We also notice that in the with new venues scenario, the well performing methods tend to be very close to each other (see PGN, UB, IRenMF, and IRenMFFreq in Jakarta), however, in the other scenario the diferences increase and some methods take more advantage than others of the diferent experimental condition.
Another interesting observation is that, as already happens in
classical recommendation [
To test the evaluation metric proposed in Section 3 based on the LCS algorithm, in Table 2 we have included two methods as skylines (named like this as opposed to the baselines, since their performance is almost impossible to achieve because they look into the test set): TestOrder, that returns the test set in the (ideal) observed order visited by the user (from lowest to highest timestamp), and TestInvOrder, that also returns the test but in the reverse order (from highest to lowest timestamp). The use of these recommenders will serve to justify the LCS-based metric, since besides taking into account the relevance, it also considers the order of visits (note that none of the other recommenders explicitly generates sequences of items, we aim to address this issue in the future). We observe that the LCS-based metrics (LCS, LCSP, LCSR) produce lower values for TestInvOrder than for TestOrder, as TestInvOrder only finds one item in the correct sequence when using these metrics; however, since TestInvOrder obtains much better results than traditional recommenders, we conclude that, for many users, the other algorithms are not able to obtain a single relevant item. At the same time, the skylines obtain the same values by any of the other ranking-based metrics (P, R, MAP, NDCG) since they do not consider the visiting order of the recommended list.
Based on these results, we can provide additional insights about how the diferent recommendation algorithms behave. For instance, in the with known venues scenario, the Training baseline seems to provide recommendations more often in the same order as the one observed in the test set, since the values for the LCS metric is always higher than for any of the sequence-agnostic evaluation metrics. In the other scenario, on the other hand, we do not observe too many variations on how the recommenders are being ranked by each evaluation metric, hence, further analysis and experiments should be performed to better understand this efect. One possible reason for this lack of variability in the performance could be related to the very small number of relevant items returned by the algorithms, in the future we would like to study this problem in more detail. 5
In this work, we discuss two aspects regarding how the
community should address the evaluation of venue recommendation
approaches. First, we analyze whether repeated interactions should
be included in the test splits, observing how state-of-the-art
recommendation algorithms change under these diferent experimental
conditions. Considering this type of behavior is common in the
tourism domain – and inherent to some type of tourists – the
presented observations could open up for discussion about how
this issue should be addressed in the community, especially, which
scenario is more interesting from an ofline point-of-view of the
evaluation process, without forgetting that some recommendations
might be obvious (hence, less useful) for the users [
The second aspect we have presented here is related to the use
of sequences in the evaluation of POI recommendation approaches.
We have defined an evaluation metric based on the Longest
Common Subsequence that takes into account how similar the
recommended list is with respect to the order the user checked in the
venues. In the future, we would like to explore how this metric
behaves on diferent tasks related to tourism recommendation, such
as next-POI recommendation and tour recommendation, where
the recommendation order plays an important role. Furthermore,
we aim to incorporate in our analysis algorithms that explicitly
recommend sequences of items [
This work was funded by the project TIN2016-80630-P (MINECO). 0.000 0.026 0.000 0.001 0.001 0.027 0.032 0.017 0.032 0.030 †0.033 (a) Istanbul (b) Jakarta 0.000 0.071 0.000 0.000 0.073 0.000 0.002 0.002 0.075 0.081 0.043 0.080 0.078 †0.088 0.000 0.038 0.078 0.002 0.003 0.047 0.065 †0.088 0.063 0.062 0.084