Our research is focused on interpreting user preference from his/her implicit behavior. There are many types of relevant behavior e.g. time on page, scrolling, clickstream etc. which we will further denote as Relevant Behavior Types (RBT). RBT s varies both in quality and incidence and thus we might need different approaches to process them. In this early work we focus on how to derive user preference from each RBT separately. We selected number of common indicators, design two novel e-commerce specific RBT interpreting methods and conducted series of offline experiments. After the off-line evaluation an A/B test on the real-world users of a travel agency was conducted comparing best off-line method with simple binary feedback. The experiments, although preliminary, showed importance of considering multiple RBTs together.
Recommender Systems have been widely studied in the last two
decades. They successfully complement search engines or on-site
catalogues on video streaming services, book databases1,
ecommerce2 etc. Although the recommender systems are relatively
widespread nowadays, we focus on yet neglected domain:
recommending on small e-commerce websites without dominant
position on the market. Among the most sewer challenges of this
domain is users’ disloyalty and high ratio between number of
objects and users. This disqualifies otherwise successful
Collaborative Filtering (CF) methods as they stuck in persistent cold-start
problem [
combine RBTs e.g. [
Coverage 99% 44% 49% 69% 0.5%
The key research question of this poster is to learn dependence
between values of each RBTs and user preference . It is possible
to use simple binary model like “all visited objects are equally
preferred” or simple numeric model with linear dependence
between the value of RBT and user preference [
for all visited objects.
tion of each indicator into the [
The hypothesis behind purchase-based approaches is that purchase is the only RBT with “guaranteed” effect on user preference, so if users evaluate other objects similarly, then although they did not purchase them, they still probably like them. Another reason for this approach is that although we can expect that higher value of each RBT implicates higher preference, the exact dependence is unknown. Purchasee-based approaches allow us to derive non-linear parametric dependence between the value of RBT and expected user preference . On the other hand in this approach we neglect different behavior patterns for different users as well as various cognitive demands to evaluate different objects. We would like to perform user clustering of more loyal users with enough feedback in the future work.
In the first phase of evaluation we compared various RBT interpretation methods on a travel agency dataset. As we did not consider any specific method for aggregating RBTs, we opted for pairwise comparison of purchased and non-purchased objects for each user. For each (strictly rated) pair and each indicator preference we state that this pair is correctly ordered, if the indicator preference of purchased object is greater than preference of non-purchased object . Incorrect and equal are defined likewise. Let Corr/Inc/Eq are sums of all correctly/incorrectly/equally ordered pairs. We can now define paired error metric as follows: Note that we consider Eq as an error too, however its significance is lower than Inc. We use α=0.5 in the evaluation.
The evaluation dataset contains 9 months usage data from a travel agency. For the purpose of the experiment, the dataset was restricted to only users with at least one purchase and at least two visited objects (some outliers were also removed) leaving over 8400 pairs of objects from 380 users with 450 purchases. for various values of ε.
Direct Dist, 0.2 Dist, 0.9 KNN, 0.01 KNN, 0.7 RBT
The KNN has peak performance around ε=0.7 for all RBTs except scrolling.
After the off-line evaluation we selected 3 methods for on-line testing: Binary user preference as baseline, average of direct normalization of all RBTs and average of best resulting methods for each RBT according to Table 2. The recommendations were computed via VSM algorithm and we opted for the number of click throughs (CT) as target metric. The evaluation was carried out in June 2015 with in total over 2900 users randomly assigned to one of the preference learning methods.
Binary Direct norm. Best according to Table 2 CT 208 232 213 Users 971 979 976 The on-line experiments are not conclusive yet, but it seems that direct normalization outperforms other methods. We will further experiment with other settings and definitions of purchase-based approaches in the future work. Our current working hypothesis is to use exact values instead of ε expressions.
In this poster, our aim was to design novel methods to infer user preference from relevant behavior types and to determine optimal approaches to handle different RBTs. The purchase-based methods succeeded in off-line experiments, however further tuning and enhanced on-line evaluation is necessary. Also incorporation of various aggregation methods is in our future work.
Acknowledgements: The work on this paper was supported by the grant SVV-2015-260222, GAUK-126313 and P46. 0.695 0.582 0.632 0.825 0.822 0.573 0.649 According to the off-line evaluation, each RBT needs to be treated differently. As for the Time, the best method was direct normalization for Mouse it was KNN with larger ε, for Pageview was optimal Distance with large ε and Scrolling on the other hand requires either KNN with small ε. For Distance method, we can clearly see grading improvements with increasing ε for all RBTs,