Due to the rise of the web, today's huge open source community and the numerous publications of industry as well as academia in the eld of computer science, nowadays even small and mid-sized companies can access state-of-the-art machine learning technologies, that can be leveraged for their businesses. In this paper, we present Carl, a hybrid recommender system utilizing content-based ltering combined with a context-aware sales model trained via XGBoost to recommend sports awards to customers. e computed recommendations are sent via e-mail to regular customers, who have already bought sports awards before. Hence, this systems aims to increase customer satisfaction by simplifying the decision which sports awards to buy every season. In o ine experiments we observe, that XGBoost compared to other state-of-the-art approaches as Factorization Machines and Neural Networks provides the best recommendation performance. However, more importantly, in the complementary online evaluation, we monitor that the interaction-and conversion rates of the e-mails sent via Carl are a magnitude higher compared to our corporate newsle er, relying on a non-personalized most popular approach.
e Pichl Medaillen GmbH is a family business founded in 1846 and specialized in producing custom medals and mints. Since the 1980s, Pichl also sells sports awards. Today, this segment is responsible for about 20% of the whole annual turnover. Due to the highly standardized products in this segment and the fact that customers demand those products regularly to have di erent awards for their events, Pichl decided to implement the company's rst recommender system in this segment as part of the digital innovation agenda. e system Carl, named a er Karl Pichl introducing the industrial manufacturing in the company, aims at helping the company to (i) design the work ow of selling sports awards more e ciently and (ii) increase customer satisfaction by suggesting products, as these suggestions ease the choice over ow by decreasing the search time for sports awards. Because customers demand new products for each season, this segment is moreover characterized by a regularly changing product assortment. Particularly, every year, about one-third of the complete assortment is replaced by new products. is is the reason why collaborative-ltering approaches or model-based approaches leveraging a user-item matrix as SVD [19], which are already available in shop applications and known to work well in other domains, fail in the presented use case: for collaborative ltering-based systems, the estimation of a useror item-similarity by leveraging user-item interactions is di cult with always changing items as no interactions for new items are available. is cold-start problem is referred to as the new item problem [26]. To avoid this problem, together with the Databases and Information System Group at the University of Innsbruck, the Pichl Medaillen GmbH decided to develop a hybrid approach recommendation facilitating content-and contextual information.
e developed approach is a hybrid recommender system facilitating both, content-based ltering and predictive modeling. e rst component based on content-based ltering covers the personalization aspect, whereas the second component is a global (not personalized) classi cation model that is capable to predict whether a certain product (with certain features) is likely to be sold in a certain period (i.e., winter or summer seasons) or not. We refer to this as saleability.
Using an o ine evaluation, we show that eXtreme Gradient Boosting [10] provides the best performance for the prediction task, compared to Factorization Machines [21] and Multilayer Perceptron Neural Networks [20]. Moreover, we are able to show that such a hybrid system overcomes the limitation of collaborative ltering for domains where the product assortment is changing regularly. Along with that, we show that the recommendations computed by the proposed system are not only highly precise if evaluated o ine but also deliver a high conversion rate in the real-life application. Hence, our proposed hybrid recommendation model is capable of recommending new products to customers and thus is applicable for domains with regularly changing product assortments. e remainder of this paper is structured as follows: We introduce the used machine learning methodologies in Section 3 and give the reader an overview about the whole recommender system in Section 4, where we also present the underlying recommendation model in more detail. Next, we introduce the dataset and the conducted o ine evaluations in Section 5. We present the real-life application in Section 6 and nally conclude this work Section 7.
accuracies [7,17,25] and to be useful for implicit feedback [14,22]. However, as outlined in the introduction, collaborative lteringbased approaches fail in our se ing due to the new item problem. To handle the new item problem, content-based approaches or hybrids facilitating content-based information to nd similar items to the new item are suitable [26]. In this work, we present a hybrid approach leveraging content-based information. Generally, content-based recommender systems focus on item characteristics to nd similar items. In particular, these systems recommend items that are similar to the items a user already interacted with in the past. is is why these are also called content-based ltering approaches: they lter items based on previous user-item interactions. ese approaches have their roots in the eld of information retrieval [4,9,24] and initially focused on recommending items containing text, for instance, news articles, websites or UseNet messages [1]. In this work, we use content-based ltering to derive an initial set of recommendation candidates. We rank the computed candidates using a context-aware classi cation approach similar to the approaches introduced next.
In the late 2000s, research shi ed towards hybrid approaches additionally integrating contextual information on top of the presented content-and collaborative ltering-based approaches: Several extensions of matrix factorization techniques have been introduced, e.g., time-aware SVD++ [18]. As context is a broad concept, subsuming any circumstances that in uencing the perceived usefulness of an item [2], a variety of additional contextual information has been exploited in the eld recommender systems, for instance, the current time [6,18], the current emotion and mood of a user [5,8,13,23] or the user's location [3,11,15,16]. Due to the success of context-aware approaches, we follow up this research and incorporate the current month as a proxy for the current season as contextual information into our recommender system, allowing us to estimate a product bias for certain seasons. To incorporate context along with content-based features, we rely on a classi cation approach (cf. Section 3).
To select the best methodology for classifying whether a product will be successfully sold or not, we evaluate three state-of-the-art classi cation approaches in an o ine evaluation. We evaluate eXtreme Gradient Boosting (XGBoost) [10], Factorization Machines (FM) [21] and Multilayer Perceptron Neural Networks (MLP) [20]. Using these three approaches, we cover a wide range of methodologies. In particular, we cover trees, factorization approaches leveraging latent features and neural networks. To contextualize the performance of the di erent classi cation approaches, we conduct an o ine evaluation (Section 5). In this evaluation, we require the classi ers to predict whether a certain product will be successful or not. For this two-class classi cation task, we consider products as successful if they exceed a certain turnover threshold.
As already outlined in the introduction, our proposed recommender system is based on two major components (C): C1 is responsible for nding the top-n new and similar products to the products that are found in the customers' purchase history. C2 is responsible for predicting whether a certain product is likely to be sold. Hence, it additionally ranks the top-n new products by the probability whether these will be sold. Finally, the top-1 recommendation is sent by a personalized e-mail to the customer periodically. In particular, we send the e-mails two weeks prior to a potential customer order. A potential customer order is computed using last order plus one period. For example, if a customer buys once a year sports awards, he or she will get the e-mail 351 days a er their last order. An overview of the system is given in Figure 1. We describe both components along with their interaction next. C1 nds the top-n nearest new products based on the products contained in a user's purchase history. For the computation of product similarity, we utilize a generalization of the Gower coecient [12]. We use this distance, as in contrast to Pearson correlation, a measure that is widely used in the eld of recommender systems, the Gower coe cient allows us to incorporate factor variables into the similarity computation. In Equation 1, we show the computation of the Gower similarity G between two products i and j. We denote i, j,k to the contribution provided by the feature k weighted by w i, j,k . e computation of the feature contribution i, j,k for numeric features as price or height is shown in Equation 2, where we denote r k to the range of feature k. For the factorial features in our dataset (color, design, …), i, j,k is computed as depicted in Equation 3.
For the nal similarity computation of products that is used in the real-life system, we set all weights w i, j,k = 1 and hence let each feature equally contribute to the product similarity. We leave a weighting scheme for future work. Using the presented computation of the Gower distance, we derive the set of recommendation candidates for each user by computing the top-n similar new items. A new item is an item that is (i) newly added to the assortment in the current year and simultaneously an item that is (ii) not found in a user's buying history. We rank this set of recommendation candidates using C2 as described in the remainder of this section.
C2 is responsible for estimating the saleability of a product (in a certain season) and hence computes the probability whether a certain product will be sold or not. As we observe that XGBoost delivers the best performance for this prediction task (cf. Section 5.3), we use XGBoost to compute the saleability of the recommendations candidates computed by C1. e saleability is computed by applying the pre-trained XGBoost model (trained with the turnover data of the last three years as described in Section 5) to the recommendation candidates. For each candidate, we get a saleability value s scaled between 0 and 1.
Using both, the product similarity based on the Gower coecient and the saleability s, we compute the nal ranking of the new items for each user using the average of both values as depicted in Equation 4.
In Equation 4, we denote r i, j as the ranking coe cient for an item i (contained in a user's purchase history) and a new item j. Furthermore, we denote i, j as the Gower coe cient between item i and j and s j as the predicted saleability of an item j. For the real-life application, we set w 1 = w 2 = 0.5 and hence, consider the personalization aspect represented by the Gower coe cient and the saleability aspect equally.
As already outlined in the previous section, we evaluate the classication accuracy of the predictive model using k-fold cross-validation. Before describing the experimental setup and discussing the results, we introduce the reader to the used dataset.
For evaluating the di erent classi cation methods, we use the turnover data of the previous three years (2015, 2016 and 2017) as training-and test data. Please note that we are not able to use the turnover data of the current year (2018), as we cannot estimate the success of the products yet. e dataset contains 538 main products and 1,939 product variations. A main product is available in di erent sizes, where each size is considered as a product variation. Hence, a product variant shares the same features besides the price and the height. As stated in Table 1, we characterize each product by 13 features. e features price and height are self-explanatory. Color, accent color 1 and accent color 2 specify the main color along with two accent colors of a product,i.e., silver or gold. Handle is a boolean feature, considering whether a cup has handles. Analogously, cap, emblem, and emblem holder are boolean features de ning whether a cup or trophy features an emblem (holder) or a cap. Please note that an emblem can be mounted on an emblem holder or on a cap. Stand indicates the material of a cup's stand, i.e., marble, wood or plastic. Decorative states whether there is a decorative element and the type, for instance, a colored orb.
Utilizing the previously introduced dataset, we conduct a k-fold cross-validation to determine the most accurate model for the new product success prediction. For this, we randomly split the dataset into 5 folds of equal size where we use each fold as the test set once and the remaining folds for the training. For this evaluation, we use the packages' default parameters but vary the number of latent features for the FM (k ∈ {1, 5, 10, 25, 50}) and the number nodes per layer n l as well as layers l of the neural network (n l ∈ 1, 2, 5, 10, 20, l ∈ {1, 2, 3}). To measure the classi cation performance, we rely on the accuracy measure and the Kappa statistic. While the rst measure solely considers the number of correctly classi ed instances, the Kappa statistic compares an observed accuracy with an expected accuracy. e expected accuracy is based on the inter-rater agreement. Due to this, the Kappa takes the possibility of correctly classifying a product to be successful by chance into account and hence is the more meaningful measure in our experiments.
e results of the conducted o ine evaluations are stated in Table 2. For the FM and the MLP classi ers, we only state the best result of our evaluations with di erent k and di erent n as well as l values respectively.
Accuracy Kappa XGBoost 0.74 0.37 FM (k = 25) 0.69 0.02 MLP (n 1 = 10, n 2 = 5, n 3 = 5) 0.60 0.29 Table 2
We observe that though the accuracies of XGBoost and the FM only di ers by 6.76%, the Kappa value of the FM is only 0.03. Hence, we cannot see a substantial performance di erence to the random baseline or rather an approach always predicting a product to be eCom ' 3: Recommender Key Metrics not successful. is is, as the FM predicts a success only for 0.72% of the products. In contrast, XGBoost classi es 39.62% of the products as a success, a more realistic number.
For the MLP-based classi er, using a grid search, we nd that a neural network with three layers containing 10, 5 and 5 nodes performs well with an accuracy of 0.60. However, though a good classi cation accuracy, according to the Kappa value, XGBoost works substantially be er. In particular, the Kappa value is 27.59% higher.
To conclude, we see that according to the Kappa statistic, both XGBoost and MLP classi ers show a fair agreement in contrast to the FM which shows only a slight agreement. In addition, we observe that to predict whether a product will be sold or not, XGBoost works best in terms of prediction accuracy and the Kappa statistic.
is is why we use XGBoost's computed probability that a product will be sold for our proposed recommender system. We refer to this probability as the saleability of the product. In the next section, we show how the computed saleability is leveraged for sports award predictions.
e go-live of Carl was on January, the 4th 2018 and until April, the 31st, more than 2,000 e-mails have been sent. As our business is highly seasonal with a peak in the beginning and the end of a year, we consider the current analysis as late-breaking results and aim to perform a more detailed online evaluation using the sales data of a complete year in a future work. Nevertheless, in the current stage, we observe a very high interaction rate with the personalized recommendations in the e-mails sent via Carl compared to the corporate newsle er. e la er follows a simple most popular approach, which suggests the most popular products of the current season and the globally most popular sale articles to our customers. In Table 3, we state the relative number of users who have visited the website via a link in the sent e-mail, which is the relative portion of the recipients actually visiting the website, the corresponding average session duration in minutes, the average number of viewed pages as well as the relative number of conversions per user and per e-mail. For the la er two measures, we divided the number of conversions by the number of e-mails sent and by the number of users respectively. Please note that for this analysis, we only tracked orders made in the online shop as a conversion, no o ine conversions as orders via telephone or an informal mail. We observe, (cf. Table 3) that the conversion rate of the personalized e-mail is a magnitude higher compared to the standard (unpersonalized) newsle er. e 8.94 times higher conversion rate per user is accompanied by a 2.77 times higher session duration with 1.89 as many page views per session. Moreover, we observe that the relative number of website visits is already a magnitude higher. However, we assume that this is not only rooted in the recommendations but also in the personalized the e-mail is sent as well as the precise timing. is will be a subject for further research in the next months.
Summing up, we see an excellent conversion rate in the e-mails sent via Carl. Our results show that for selecting products to be promoted in e-mail newsle ers, combining the predictor for the saleability of products with a traditional content-based recommender system allows for a substantial improvement in a diverse set of quality measures. Hence, in a future work, we will run experiments on ne-tuning the recommender system and aim to implement a latent feature approach for the content-based part. Besides that, we aim to conduct a profound online analysis a er a whole year to capture all seasonal e ects.
In this paper, we present Carl, a recommender system that aims to improve customer satisfaction by suggesting sports awards to customers of the Pichl Medaillen GmbH, an Austrian SME. In particular, the presented system aims to increase the customer satisfaction by sending the recommendations via e-mail to our regular customers who buy sports awards every season (i.e., yearly) and hence helps to nd suitable sports awards out of a set of more than 200 awards that change regularly. e recommendations are computed using a hybrid approach that leverages content-based ltering combined with a context-aware sales model. e la er is trained via XGBoost and estimates a general saleability coe cient for each product based on product features and contextual information as the current season approximated by the current month. In an o ine evaluation, we show that XGBoost delivers the best performance compared to Factorization Machines and multilayer perceptron neural networks. In a complementary online study can show that the conversion rate is substantially higher than the conversion rate of the unpersonalized corporate newsle er, promoting the most popular articles.