Due to the rise of the web, today's huge open source community and the numerous publications of industry as well as academia in the eld of computer science, nowadays even small and mid-sized companies can access state-of-the-art machine learning technologies, that can be leveraged for their businesses. In this paper, we present Carl, a hybrid recommender system utilizing content-based ltering combined with a context-aware sales model trained via XGBoost to recommend sports awards to customers. e computed recommendations are sent via e-mail to regular customers, who have already bought sports awards before. Hence, this systems aims to increase customer satisfaction by simplifying the decision which sports awards to buy every season. In oine experiments we observe, that XGBoost compared to other state-of-the-art approaches as Factorization Machines and Neural Networks provides the best recommendation performance. However, more importantly, in the complementary online evaluation, we monitor that the interaction- and conversion rates of the e-mails sent via Carl are a magnitude higher compared to our corporate newsleer, relying on a non-personalized most popular approach.
INTRODUCTION
e Pichl Medaillen GmbH is a family business founded in 1846
and specialized in producing custom medals and mints. Since the
1980s, Pichl also sells sports awards. Today, this segment is
responsible for about 20% of the whole annual turnover. Due to the
highly standardized products in this segment and the fact that
customers demand those products regularly to have dierent awards
for their events, Pichl decided to implement the company’s rst
recommender system in this segment as part of the digital
innovation agenda. e system Carl, named aer Karl Pichl introducing
the industrial manufacturing in the company, aims at helping the
company to (i) design the workow of selling sports awards more
eciently and (ii) increase customer satisfaction by suggesting
products, as these suggestions ease the choice overow by
decreasing the search time for sports awards. Because customers demand
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
eCom’18, Ann Arbor, Michigan, USA
© 2018 Copyright held by the owner/author(s). 123-4567-24-567/08/06. . . $15.00
DOI: 10.475/123 4
new products for each season, this segment is moreover
characterized by a regularly changing product assortment. Particularly,
every year, about one-third of the complete assortment is replaced
by new products. is is the reason why collaborative-ltering
approaches or model-based approaches leveraging a user-item matrix
as SVD [
e developed approach is a hybrid recommender system facilitating both, content-based ltering and predictive modeling. e rst component based on content-based ltering covers the personalization aspect, whereas the second component is a global (not personalized) classication model that is capable to predict whether a certain product (with certain features) is likely to be sold in a certain period (i.e., winter or summer seasons) or not. We refer to this as saleability.
Using an oine evaluation, we show that eXtreme Gradient
Boosting [
e remainder of this paper is structured as follows: We introduce the used machine learning methodologies in Section 3 and give the reader an overview about the whole recommender system in Section 4, where we also present the underlying recommendation model in more detail. Next, we introduce the dataset and the conducted oine evaluations in Section 5. We present the real-life application in Section 6 and nally conclude this work Section 7. 2
Since the 2000s, recommender system research focused on
collaborative ltering approaches and in particular, on matrix-factorization
techniques such as singular value decomposition (SVD) as these
approaches have been shown to achieve the best recommendation
accuracies [
In the late 2000s, research shied towards hybrid approaches
additionally integrating contextual information on top of the
presented content- and collaborative ltering-based approaches:
Several extensions of matrix factorization techniques have been
introduced, e.g., time-aware SVD++ [
To select the best methodology for classifying whether a product
will be successfully sold or not, we evaluate three state-of-the-art
classication approaches in an oine evaluation. We evaluate
eXtreme Gradient Boosting (XGBoost) [
As already outlined in the introduction, our proposed recommender system is based on two major components (C): C1 is responsible for nding the top-n new and similar products to the products that are found in the customers’ purchase history. C2 is responsible for
select purchase histories of relevant customers compute top-n nearest new products select current turnover data compute model for new product success prediction
compute top-1 recommendation send e-mail to customers predicting whether a certain product is likely to be sold. Hence, it additionally ranks the top-n new products by the probability whether these will be sold. Finally, the top-1 recommendation is sent by a personalized e-mail to the customer periodically. In particular, we send the e-mails two weeks prior to a potential customer order. A potential customer order is computed using last order plus one period. For example, if a customer buys once a year sports awards, he or she will get the e-mail 351 days aer their last order. An overview of the system is given in Figure 1. We describe both components along with their interaction next.
C1 nds the top-n nearest new products based on the products
contained in a user’s purchase history. For the computation of
product similarity, we utilize a generalization of the Gower
coecient [
Gi; j = gi; j;k = 1 Ík wi; j;k gi; j;k Ík wi; j;k jxi;k
xj ; k j rk gi; j;k = ( 1 i f xi;k = xj;k 0 i f xi;k , xj;k (1) (2) (3)
For the nal similarity computation of products that is used in the real-life system, we set all weights wi; j;k = 1 and hence let each feature equally contribute to the product similarity. We leave a weighting scheme for future work. Using the presented computation of the Gower distance, we derive the set of recommendation candidates for each user by computing the top-n similar new items. A new item is an item that is (i) newly added to the assortment in the current year and simultaneously an item that is (ii) not found in a user’s buying history. We rank this set of recommendation candidates using C2 as described in the remainder of this section.
C2 is responsible for estimating the saleability of a product (in a certain season) and hence computes the probability whether a certain product will be sold or not. As we observe that XGBoost delivers the best performance for this prediction task (cf. Section 5.3), we use XGBoost to compute the saleability of the recommendations candidates computed by C1. e saleability is computed by applying the pre-trained XGBoost model (trained with the turnover data of the last three years as described in Section 5) to the recommendation candidates. For each candidate, we get a saleability value s scaled between 0 and 1.
Using both, the product similarity based on the Gower coecient g and the saleability s, we compute the nal ranking of the new items for each user using the average of both values as depicted in Equation 4.
ri; j = w1gi; j + w2sj (4)
In Equation 4, we denote ri; j as the ranking coecient for an item i (contained in a user’s purchase history) and a new item j. Furthermore, we denote gi; j as the Gower coecient between item i and j and sj as the predicted saleability of an item j. For the real-life application, we set w1 = w2 = 0:5 and hence, consider the personalization aspect represented by the Gower coecient and the saleability aspect equally. 5
As already outlined in the previous section, we evaluate the classication accuracy of the predictive model using k-fold cross-validation. Before describing the experimental setup and discussing the results, we introduce the reader to the used dataset. 5.1
For evaluating the dierent classication methods, we use the turnover data of the previous three years (2015, 2016 and 2017) as training- and test data. Please note that we are not able to use the turnover data of the current year (2018), as we cannot estimate the success of the products yet. e dataset contains 538 main products and 1,939 product variations. A main product is available in dierent sizes, where each size is considered as a product variation. Hence, a product variant shares the same features besides the price and the height. As stated in Table 1, we characterize each product by 13 features.e features price and height are self-explanatory. Color, accent color 1 and accent color 2 specify the main color along with two accent colors of a product,i.e., silver or gold. Handle is a boolean feature, considering whether a cup has handles. Analogously, cap, emblem, and emblem holder are boolean features dening whether a cup or trophy features an emblem (holder) or a cap. Please note that an emblem can be mounted on an emblem holder or on a cap. Stand indicates the material of a cup’s stand, i.e., marble, wood or plastic. Decorative states whether there is a decorative element and the type, for instance, a colored orb.
Feature Type height numeric price numeric color factor accent color 1 factor accent color 2 factor handle boolean decorative factor emblem holder boolean emblem boolean design factor stand factor cap boolean material factor
Table 1: Product Features Overview 5.2
Utilizing the previously introduced dataset, we conduct a k-fold cross-validation to determine the most accurate model for the new product success prediction. For this, we randomly split the dataset into 5 folds of equal size where we use each fold as the test set once and the remaining folds for the training. For this evaluation, we use the packages’ default parameters but vary the number of latent features for the FM (k 2 f1; 5; 10; 25; 50g) and the number nodes per layer nl as well as layers l of the neural network (nl 2 1; 2; 5; 10; 20, l 2 f1; 2; 3g). To measure the classication performance, we rely on the accuracy measure and the Kappa statistic. While the rst measure solely considers the number of correctly classied instances, the Kappa statistic compares an observed accuracy with an expected accuracy. e expected accuracy is based on the inter-rater agreement. Due to this, the Kappa takes the possibility of correctly classifying a product to be successful by chance into account and hence is the more meaningful measure in our experiments. 5.3
e results of the conducted oine evaluations are stated in Table 2. For the FM and the MLP classiers, we only state the best result of our evaluations with dierent k and dierent n as well as l values respectively.
XGBoost 0.74 FM (k = 25) 0.69 MLP (n1 = 10, n2 = 5, n3 = 5) 0.60 Table 2: Prediction Accuracy
0.37 0.02 0.29
We observe that though the accuracies of XGBoost and the FM only diers by 6.76%, the Kappa value of the FM is only 0.03. Hence, we cannot see a substantial performance dierence to the random baseline or rather an approach always predicting a product to be
Carl Corporate Newsleer
20.61% 1.38% not successful. is is, as the FM predicts a success only for 0.72% of the products. In contrast, XGBoost classies 39.62% of the products as a success, a more realistic number.
For the MLP-based classier, using a grid search, we nd that a neural network with three layers containing 10, 5 and 5 nodes performs well with an accuracy of 0.60. However, though a good classication accuracy, according to the Kappa value, XGBoost works substantially beer. In particular, the Kappa value is 27.59% higher.
To conclude, we see that according to the Kappa statistic, both XGBoost and MLP classiers show a fair agreement in contrast to the FM which shows only a slight agreement. In addition, we observe that to predict whether a product will be sold or not, XGBoost works best in terms of prediction accuracy and the Kappa statistic. is is why we use XGBoost’s computed probability that a product will be sold for our proposed recommender system. We refer to this probability as the saleability of the product. In the next section, we show how the computed saleability is leveraged for sports award predictions. 6 REAL-LIFE APPLICATION e go-live of Carl was on January, the 4th 2018 and until April, the 31st, more than 2,000 e-mails have been sent. As our business is highly seasonal with a peak in the beginning and the end of a year, we consider the current analysis as late-breaking results and aim to perform a more detailed online evaluation using the sales data of a complete year in a future work. Nevertheless, in the current stage, we observe a very high interaction rate with the personalized recommendations in the e-mails sent via Carl compared to the corporate newsleer. e laer follows a simple most popular approach, which suggests the most popular products of the current season and the globally most popular sale articles to our customers. In Table 3, we state the relative number of users who have visited the website via a link in the sent e-mail, which is the relative portion of the recipients actually visiting the website, the corresponding average session duration in minutes, the average number of viewed pages as well as the relative number of conversions per user and per e-mail. For the laer two measures, we divided the number of conversions by the number of e-mails sent and by the number of users respectively. Please note that for this analysis, we only tracked orders made in the online shop as a conversion, no oine conversions as orders via telephone or an informal mail. We observe, (cf. Table 3) that the conversion rate of the personalized e-mail is a magnitude higher compared to the standard (unpersonalized) newsleer. e 8.94 times higher conversion rate per user is accompanied by a 2.77 times higher session duration with 1.89 as many page views per session. Moreover, we observe that the relative number of website visits is already a magnitude higher. However, we assume that this is not only rooted in the recommendations but also in the personalized the e-mail is sent as well as the precise timing. is will be a subject for further research in the next months.
Summing up, we see an excellent conversion rate in the e-mails sent via Carl. Our results show that for selecting products to be promoted in e-mail newsleers, combining the predictor for the saleability of products with a traditional content-based recommender system allows for a substantial improvement in a diverse set of quality measures. Hence, in a future work, we will run experiments on ne-tuning the recommender system and aim to implement a latent feature approach for the content-based part. Besides that, we aim to conduct a profound online analysis aer a whole year to capture all seasonal eects.
In this paper, we present Carl, a recommender system that aims to improve customer satisfaction by suggesting sports awards to customers of the Pichl Medaillen GmbH, an Austrian SME. In particular, the presented system aims to increase the customer satisfaction by sending the recommendations via e-mail to our regular customers who buy sports awards every season (i.e., yearly) and hence helps to nd suitable sports awards out of a set of more than 200 awards that change regularly. e recommendations are computed using a hybrid approach that leverages content-based ltering combined with a context-aware sales model. e laer is trained via XGBoost and estimates a general saleability coecient for each product based on product features and contextual information as the current season approximated by the current month. In an oine evaluation, we show that XGBoost delivers the best performance compared to Factorization Machines and multilayer perceptron neural networks. In a complementary online study can show that the conversion rate is substantially higher than the conversion rate of the unpersonalized corporate newsleer, promoting the most popular articles.