Recommender systems have evolved beyond the basic user-item filtering methods in research. However, these filtering methods are still commonly used in real-world scenarios, mainly because they are easier to debug and reconfigure. Indeed the existing frameworks do not adequately support algorithmic tuning. Moreover, they are primarily focused on the reproducibility of state-of-the-art accuracy rather than ease of algorithm development and maintenance. Therefore, rapid and iterative experimentation and debugging are considerably hindered. In this work, we propose an AutoML-based framework with a modular deep session-based recommender code-base and an integrated automated HyperParameter Tuning (HPT4Rec) component. The proposed framework automates searching for the best session-based model for a given data. Therefore it can help to consistently update the model based on potential changes in the type and volume of data that is prevalent for a real-world scenario. It is demonstrated that HPT4Rec provides extensible data structures, training service compatibility, and GPU-accelerated execution while maintaining training eficiency and recommendation accuracy. We have conducted our experiments on the benchmark RecSys 2015 dataset and achieved performance on par with state-of-the-art results. Achieved results of our experiments show the importance of continuous and iterative parameter tuning, particularly for real-world scenarios.
CEUR ceur-ws.org
It is often overwhelming to an e-commerce user to see so many products available for sale. Recognizing the burden of data overload, recommender systems (RSs) improve user experience substantially in various applications. Traditional RSs often rely on user profiles to provide personalized recommendations. Collaborative filtering approaches [1, 2, 3] could use history of purchases to determine user similarity, or use matrix factorization to establish latent factor vectors for each user. In both cases, it is essential to identify the user when making recommendations. However, this may not always be possible, such as not being logged in, having deleted their tracking information, or a new user not having profile.
user’s history sufer from cold-start issues.
Making session-based recommendations is another alternative to using historical data [4]. In this setup, recommendations are only made based on the behavior of users nEvelop-O (A. H. Karimi) (A. H. Karimi) data like GDPR [5].
Methods based on deep learning (DL) have shown great promise in the session-based recommendation and also in other communities [6]. As stated in various literature [7, 8, 9], they perform better than traditional baseline methods by around 20-30 percent. However, recent investigations have shown that many of these methods are not compelling enough [10], moreover, results are hard to reproduce in many of them [11], and the codes are not readily available. Recent publications have addressed reproducibility by implementing several
[12, 13, 14]. While these frameworks are efective and helped to alleviate the problem, two key factors should not be overlooked: 1. Iterative algorithm optimization: If these algorithms are intended for real-world use, they dataset (not the ofline benchmark datasets). The process emerge, and user preferences may change. 2. Modularity and ease of reproduciblity: Besides accuracy, several other factors must be taken into consideration, when implementing literature-approved methods in production, including non-complexity, fault tolerance, real-time pre- with the neighborhood-based method, Jannach et al. [10] diction, debuggability, resource consumption, and modu- combined sequential patterns and co-occurrence signals larity [15, 16]. The most advanced and well-performing to get the best of both worlds. Tuan et al. [17] fused models are often left behind in the business, because they session clicks with content features (namely, item titles are complex and challenging to debug. As a result, busi- and categories) to generate recommendations based on nesses still opt for more straightforward methods that 3-dimensional Convolutional Neural Networks (CNN). are less accurate, but easier to manipulate and debug. In Li et al. [21] have developed a neural attentive recomseveral papers [8, 10, 17, 18] (discussed in the background mendation machine (NARM) using an encoder-decoder section of prior work), various techniques were used to architecture. NARM can distinguish sequential behavior slightly improve performance, which not only may not and the primary purposes of users using the attention be useful for large-scale day-to-day use, but may also mechanism on RNN. In another study, a Short-Term Atcause problems in production and during debugging. It tention Priority model (STAMP) [18], which employs a would be more practical to implement a robust and mod- simple MLP network, and an attentive net has been proular core structure with clear interfaces and to give room posed for understanding users’ general interests as well to add more complex mechanisms based on the business as their current interests. In both NARM and STAMP, an demands. attention mechanism emphasizes the importance of the
Motivated by reasons mentioned above, in this paper, last click. we present HPT4Rec, an AutoML-based framework for Almost all of the aforementioned RNN-based SBR modhyperparameter self-tuning with a modular code-base els follow the same architecture as GRU4Rec [7]. They aimed at session-based recommendation. Our frame- have just incorporated new features and mechanisms to work simplifies the development and manipulation of improve performance on top of the core structure. Theredeep recommendation algorithms to meet business needs. fore, in HPT4Rec, a minimal code-base based on GRU4Rec PyTorch and Microsoft NNI 1 are used to develop the was built, with all the necessary tools and modules for code-base, both of which are well known in the DL and a methodologically simplified bottom-up approach to AutoML communities and receive continuous updates. model development. This can remove the barrier of en
Besides being open-source, this framework can be in- try for practitioners and allow them to add other features stalled easily, and all prepared data and trained mod- if necessary. els are available at https://github.com/amirreza-m95/ Related Frameworks In the modern RSs field, reHPT4Rec producibility is crucial. Recently, various researchers [10, 11, 22, 23] pointed out the need for fair evaluation of recommender models. Upon thorough hyperparameter 2. Prior Work tuning, their argument about the supremacy of latentfactor models over deep neural models made it necessary Background. The most commonly used deep model, to develop new recommendation frameworks. Beginwhen dealing with sequential data are Recurrent Neu- ning in 2011, Mymedialite [24], , RankSys [25], LensKit ral Networks (RNN). There is a type of RNNs known [26], LightFM [27], and Surprise [28] have established as LSTM [19] that are shown to work particularly well, a set of integrated tools for rapid prototyping and testincluding additional gates regulating, when to take into ing of recommendation models, using standard metrics account input and, when to reset the hidden state. These and an intuitive model execution. Deep learning (DL) models are not afected by the vanishing gradient prob- recommendation models achieved remarkable success lem usually associated with RNN models. A somewhat and attracted growing community interest, which led simpler alternative to LSTM, but still retaining all of its to the development of new tools. The first open-source properties, are Gated Recurrent Units (GRUs) [20], which frameworks for DL-based recommenders were LibRec we employ in this work as the core learning structure of [29], Spotlight [30], and OpenRec [31]. Although these the recommender for the experiments. frameworks provided plenty of models, they lacked fil
Hidasi et al. [7] suggested the RNN approach for tering and Automated hyperparameter tuning strategies. session-based recommendation (SBR) and then proposed The RecQ [32], DeepRec [33], and Cornac [34] framea parallel RNN architecture [9] to model sessions using works have made a significant contribution towards a the clicks and features of the clicked items. Further re- more comprehensive collection of model implementasearch was presented based on RNN methods in order to tions. DaisyRec [35], RecBole [36], and Elliot [12] raised improve the accuracy of this model. Performance of the the bar considerably after the reproducibility hype, makrecurrent model can be boosted by taking into account ing available a large number of models, data filtering and temporal changes in user behavior and data augmen- splitting operations, as well as hyperparameter tuning. tation techniques[8]. By uniting the recurrent method Nevertheless, we observed a deficiency of two increas1https://github.com/microsoft/nni ingly critical aspects of recommendation model develop
where, The logistic sigmoid function is a smooth function with a bounded input of , which is the unit input at time . Based on its actual state ℎ , an RNN provides a probability distribution for the subsequent element of the sequence.
GRU is a form of RNN that tends to cope with vanishing gradient problems better than vanilla RNN. In essence, GRU gates learn when to update their hidden state and by how much. GRUs are superior to Long Short-Term Memory (LSTM) units when it comes to the session-based recommendation. [7].
A linear interpolation between the prior activation and the candidate activation is used to determine GRU activation, ℎ : (2) (1) I n p u t D a t a
E m b e d d i n g L a y e r
G a t e d R e c u rr e n t U n it
G a t e d R e c u rr e n t U n it
G a t e d R e c u rr e n t U n it
F e e d fr o w a r d L a y e r s
S c o r e s o n I t e m s 3. HPT4Rec In this section, we describe HPT4Rec’s architecture and tuning pipeline. First, we describe the general architecture of the recommender. Next, we present the components and architecture of the framework. Finally, we discuss the available self-tuning methods and their best application scenarios.
RNNs. RNNs are characterized by the internal hidden state present in the units that make up the network, which sets them apart from conventional feedforward neural networks. A standard RNN updates its hidden zt = ( xt + ht−1) (3)
In a similar manner while the candidate activation function, ℎ , is also computed: (4) (5) ht̂ = tanh ( xt + (rt ⊙ ht−1)) and eventually, the reset gate is provided by: rt = ( xt + ht−1)
in Equations (3) and (4), but it is important to note that framework users can tweak the model by using other options, like using diferent final activations such as relu, leaky-relu, and softmax. 3.1.1. GRU4Rec Architecture The network core comprises the GRU layers, and further feedforward layers may be added between the GRU layer and the output. Each item’s predicted preference can be calculated to predict whether it will be the next item in the session. If more than one GRU layer is employed, the hidden state of each layer is used as an input for the next layer. An option is to connect the input to a higher layer of the network to improve performance [7]. We adjusted the base network to suit the task better since recommender systems are not the principal application area of RNNs. The SBR model architecture is demonstrated in Figure 1.
In addition, we also use trainable embeddings to represent all of our inputs. With backpropagation Through
Time (BPTT), we can train our neural networks using mini-batch gradient descent on multiple options for loss over a dynamic number of time steps.
Session-parallel mini-batches. Click sessions are often of varying length. It may take some users a long time to ifnd their desired item, while others find it within seconds. 21 { "dropout_rate": {"_type": "uniform", "_value": [0.1, 0.5]}, In the recommender system, accurate predictions should 3 "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, 4 "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, be provided regardless of the current session length. This 5 "lr": {"_type": "loguniform", "_value": [0.0001, 0.1]}, problem has been addressed by diferent methods like 6 "momentum": {"_type": "lognormal", "_value": [0.1, 1]} session-parallel mini-batches [9] and data augmentation 7 } [8]. Since we are seeking the least sophisticated approach, we have taken the former approach. by the variable name, sampling strategy, and parameters of a search space.
A search space definition can be expressed as follows:
configuration (e.g., a set of hyperparameters) to a model. The first step in constructing an experiment is to define the search space (i.e., parameters). The tuner will sample parameters/architecture according to the search space, which is defined as a JSON file. Search spaces are defined
According to this definition, the dropout rate is characterized by a uniform distribution within a range of 0.1 to 3.2. Architecture and Data Flow 0.5. This search space will be used by Tuner to build conifgurations, selecting a value from within the range for Automated tuning of hyperparameters is a key feature of each parameter. Besides defining the search space, the HPT4Rec. We provide 11 popular self-tuning algorithms. only requirement is to define a configuration file containExperiments can be run on a wide range of training plat- ing information like experiment log folder, self-tuning forms, including local machines, multiple servers on a algorithms, trial number, and duration threshold. The distributed network, and open-source platforms such as configuration file is in YAML format. Kubernetes and OpenPAI. In order to implement a new tuning algorithm or tweak the existing ones, the base tuner should be inherited. 3.2.1. HPT4Rec’s Data Flow Then, by following the interface of the module and returning the experiment results, passing the new parameters, and updating the search space, the tuning module will function properly. ber of trials. A wide range of experiments revealed that TPE outperformed random search. If the variables in the search space can be selected from a prior distribution, Anneal is useful. Likewise, it is recommended to use naive evolution, when your experiment code supports weight transfer, which implies that the experiment could inherit its parent’s converged weight from its predecessor.
Training can be substantially accelerated with the right tuning method, resulting in less time and money spent and higher revenue, as well as better recommenders, to enhance user experience.
4.1.1. Dataset 3.2.2. Architecture By executing the experiment_runner python script through Cli and passing the configuration file path, experiments are instantiated. The experiment manager parses the configuration file to determine the path to the search space and target the training service, and then runs the model code with the appropriate parameters from the search space. Preprocessing will be performed by the experiment manager (e.g., one-hot encoding, embedding dropout). Following the execution of the model with the first set of parameters, the self-tuner will examine intermediate results (i.e., after each epoch) to determine whether results are improving. Next, it will pass the model on to the evaluation module. Evaluation will be conducted by the evaluator, and results will be provided to self-tuning algorithm to update its inner state. Following the update, the self-tuning algorithm determines the next metric to use. The iterative process will be repeated until a certain time or number of experiments is reached.
Figure 2 illustrates this procedure. HPT4Rec will output results in a webUI interface and collect all metrics, intermediate results, best parameters, and system logs in a JSON format.
We conducted our experiments on the YOOCHOOSE e-commerce dataset for RecSys 2015 challenge 2. A sixmonth period of click-streams from an e-commerce site was included in this dataset. Click-streams are sometimes followed by purchase events. Following preprocessing, there are 7,936,469 sessions and 31,437,691 clicks on 37,403 items left for training and testing. Each clicking 3.2.3. Self-tuning event contains a session ID, an item ID and, if the item is The cycle of getting hyperparameters, carrying out exper- a buy-item, a price tag. A shopping session can contain iments, testing their results, and then tuning hyperparam- anywhere between 1 and 200 clicks, but most sessions eters is deemed as self-tuning. Recommender systems contain less than 30 clicks. We keep only the click events are used in various online websites with diferent lev- from the challenge’s training set. Sessions of length one els of user activity, which directly afects the volume of are filtered out. The Yoochoose dataset was chosen since data available for training models. Additionally, Training it is the most general dataset, based on the dataset’s feadeep models require substantial computational resources, tures compared to other well-known datasets in this field which is another crucial aspect since it directly impacts such as Diginetica3, Xing4, and Last.fm5. The default setrevenue. Thereby, diferent tuning strategies are needed tings of the framework can be used for all the datasets we based on available features, the volume of data, and avail- mentioned just by omitting some of their extra features. able computational resources. Based on the framework We employ a dataset characterized by minimalistic data review shown in table 1, HPT4Rec ofers several tuning features as a means to ensure the robust generalizability techniques tailored for diverse scenarios that occur in of the model to diverse datasets encompassing a greater real-world scenarios. abundance of data features.
After a series of experiments, we have gained an early intuition about the most suitable use cases of each selftuning algorithm. In that sense, Tree-structured Parzen 32hhttttpp:s/://2/c0o1m5.rpeectsityisocnhsa.clloednaglea.bc.oomrg/competitions/11161 Estimator (TPE) [42] is suitable when computation re- 4http://2016.recsyschallenge.com/ sources are limited, and you can only try a limited num- 5http://ocelma.net/MusicRecommendationDataset/lastfm-1K.html 4.1.2. Evaluation Metrics In order to match the user with the most relevant item on the list, recommender systems can recommend only a few items at a time. We, therefore, use recall@20 as our main evaluation metric, which counts the proportion of cases that have the targeted item in the top 20 items for all test cases. As long as an item is among the top-N, recall does not take its rank into consideration. The MRR@20 metric is the second metric used in the experiments. A reciprocal ranking of the desired items determines this value. A reciprocal rank above 20 is set to zero. 4.1.3. Implementation Details Thus, to make recommendations that reflect changes in user behavior over time, models must be continuously For demonstration purposes and to have a quantifiable and iteratively optimized. It is possible to have diferent search space, we optimized hidden size, batch size, learn- approaches when we have diferent quantities of data ing rate and the number of GRU layers and fixed others as and computation to find the best-optimized model, as follow. For our model, 50-dimensional embeddings were discussed in the self-tuning 3.2.2. Our experiments have used for the items, with a 20% embedding dropout. The been conducted using four proxy datasets that mirror optimization was conducted using Adam [43]. The GRU the RecSys benchmark data, which comprise diferent search space was set at 50 to 1000 hidden units for each quantities of data. HPT4Rec’s recommender model was model. A session ends with the GRU’s hidden state reset tuned using four self-tuning methods that used proxy to zero. Models are developed in PyTorch and trained on datasets as training data. Evaluation metrics and tuning an NVIDIA Tesla V100. The source code of the model, time were recorded to compare these methods. Table 2 checkpoints, and logs are available online. shows how we found the most efective model using 30
The comparison was made with four traditional recom- experiments. Results do not indicate the optimal use case mendations (POP, S-POP, Item-KNN and BPR-MF) and scenario for tuning methods, but rather demonstrate that with two well-performing configurations of GRU4Rec. each of these tuners performs well in diferent scenarios and that one of them does not outperform the others in all proxy datasets and evaluation metrics. • POP. In one of its simplest forms, the popular predictor predicts the items that are most popular in the training set. Even though it is simple, it often provides a good baseline for certain domains. • S-POP. This baseline recommends the items that are most popular during the current session. As the session progresses, the recommendation list grows. Global popularity values are used to break up ties. • Item-KNN. This baseline measures similarity by dividing the number of times two items appear together in sessions by the square root of the product of their occurrence rates.
4.2.1. Diverse Self-tuning Methods Efectiveness The most likely scenario for developing a recommender system in the real world is carrying out an experiment, where diferent levels of training data are collected. This may change as user activity increases and new users visit the website. Even in the ofline dataset of RecSys 2015, the results of training on a complete dataset are slightly worse than those of training on a recent region of the dataset, which shows changing user behavior [8]. 4.2.2. Consistency with Published Results A key element for any new tool is consistency with the previously published results since a wide range of results are possible due to a variety of implementation details, non-fixed seed values, and other domain-specific reasons. Our research also featured HPT4Rec’s self-tuning method for optimizing the base recommender model with the Original RecSys dataset. In Table 2 we show that HPT4Rec has outperformed baseline models by a fair margin and is almost on par with state-of-the-art models with this privilege that it has discovered parameters that lead to a simpler model, which results in less resource consumption in production mode. The pursuit of more streamlined models facilitates enhanced reproducibility, a fundamental tenet of our methodology, thereby engendering an essential advancement.
ommender system framework based on AutoML called HPT4Rec. We reviewed the recommended systems frameworks in the literature, showing HPT4Rec’s mer- networks meet the neighborhood for session-based its and shortcomings, and emphasizing the advantages recommendation, in: Proceedings of the Eleventh of modularity and automatic tuning. To the best of our ACM Conference on Recommender Systems, 2017, knowledge, HPT4Rec is the first recommendation frame- pp. 306–310. work that provides a thorough self-tuning experimen- [11] M. F. Dacrema, P. Cremonesi, D. Jannach, Are we tal pipeline supported by business scale training service really making much progress? a worrying analycompatibility. We expect HPT4Rec to simplify the tuning sis of recent neural recommendation approaches, efort of recommendation models, facilitate the devel- in: Proceedings of the 13th ACM Conference on opment and debugging process of new algorithms, and Recommender Systems, 2019, pp. 101–109. help migrate deep recommender algorithms to be used [12] V. W. Anelli, A. Bellogín, A. Ferrara, D. Malitesta, in real-world scenarios. Our immediate future work will F. A. Merra, C. Pomo, F. M. Donini, T. D. Noia, Elliot: emphasize automating other aspects of the recommen- a comprehensive and rigorous framework for redation pipeline, such as automated data augmentation, producible recommender systems evaluation, 2021. which has traditionally been done manually in literature. arXiv:2103.02590. [13] L. Yang, E. Bagdasaryan, J. Gruenstein, C.-K. Hsieh,
D. Estrin, Openrec: A modular framework for References extensible and adaptable recommendation algorithms, in: Proceedings of the Eleventh ACM In[1] Y. Koren, R. Bell, C. Volinsky, Matrix factorization ternational Conference on Web Search and Data techniques for recommender systems, Computer Mining, WSDM ’18, Association for Computing 42 (2009) 30–37. Machinery, New York, NY, USA, 2018, p. 664–672. [2] Y. Koren, Factorization meets the neighborhood: a URL: https://doi.org/10.1145/3159652.3159681. multifaceted collaborative filtering model, in: Pro- [14] S. Zhang, Y. Tay, L. Yao, B. Wu, A. Sun, Deeprec: ceedings of the 14th ACM SIGKDD international An open-source toolkit for deep learning based recconference on Knowledge discovery and data mining, 2008, pp. 426–434. [15] oPm.Kmoeunkdi,aIt.ioFno,u2n0ta1l9i.s,aNrX. iVva:s1il9o0g5lo.u1,0X53.6C.ui, E. Lib[3] R. Salakhutdinov, A. Mnih, G. Hinton, Restricted erty, K. Al Jadda, From the lab to production: A boltzmann machines for collaborative filtering, in: case study of session-based recommendations in Proceedings of the 24th international conference the home-improvement domain, in: Fourteenth on Machine learning, 2007, pp. 791–798. ACM conference on recommender systems, 2020, [4] J. B. Schafer, J. Konstan, J. Riedl, Recommender pp. 140–149.
systems in e-commerce, in: Proceedings of the 1st [16] D. Jannach, M. Jugovac, Measuring the business ACM conference on Electronic commerce, 1999, pp. value of recommender systems, ACM Trans. Man158–166. age. Inf. Syst. 10 (2019). URL: https://doi.org/10. [5] E. Commission, 2018 reform of eu data 1145/3370082.
protection rules, 2018-05-25. URL: https: [17] T. X. Tuan, T. M. Phuong, 3d convolutional net//ec.europa.eu/commission/sites/beta-political/ works for session-based recommendation with confiles/data-protection-factsheet-changes_en.pdf. tent features, in: Proceedings of the eleventh [6] A. Datar, C. Pan, M. Nazeri, X. Xiao, Toward ACM conference on recommender systems, 2017, wheeled mobility on vertically challenging terrain: pp. 138–146.
Platforms, datasets, and algorithms, arXiv preprint [18] Q. Liu, Y. Zeng, R. Mokhosi, H. Zhang, Stamp: arXiv:2303.00998 (2023). short-term attention/memory priority model for [7] B. Hidasi, A. Karatzoglou, L. Baltrunas, D. Tikk, session-based recommendation, in: Proceedings of Session-based recommendations with recurrent the 24th ACM SIGKDD International Conference neural networks, CoRR abs/1511.06939 (2016). on Knowledge Discovery & Data Mining, 2018, pp. [8] Y. K. Tan, X. Xu, Y. Liu, Improved recurrent neural 1831–1839.
networks for session-based recommendations, in: [19] S. Hochreiter, J. Schmidhuber, Long short-term Proceedings of the 1st workshop on deep learning memory, Neural computation 9 (1997) 1735–1780. for recommender systems, 2016, pp. 17–22. [20] K. Cho, B. Van Merriënboer, D. Bahdanau, Y. Ben[9] B. Hidasi, M. Quadrana, A. Karatzoglou, D. Tikk, gio, On the properties of neural machine translaParallel recurrent neural network architectures for tion: Encoder-decoder approaches, Fifth Workshop feature-rich session-based recommendations, in: on Syntax, Semantics and Structure in Statistical Proceedings of the 10th ACM conference on recom- Translation (2014).
mender systems, 2016, pp. 241–248. [21] J. Li, P. Ren, Z. Chen, Z. Ren, T. Lian, J. Ma, Neu[10] D. Jannach, M. Ludewig, When recurrent neural ral attentive session-based recommendation, in: Proceedings of the 2017 ACM on Conference on ACM/IEEE 47th Annual International Symposium Information and Knowledge Management, 2017, pp. on Computer Architecture (ISCA), IEEE, 2020, pp. 1419–1428. 982–995. [22] S. Rendle, L. Zhang, Y. Koren, On the dificulty of [34] A. Salah, Q.-T. Truong, H. W. Lauw, Cornac: A comevaluating baselines: A study on recommender sys- parative framework for multimodal recommender tems, 2019. arXiv:1905.01395. systems, Journal of Machine Learning Research 21 [23] D. Jannach, G. de Souza P. Moreira, E. Oldridge, (2020) 1–5.
Why are deep learning models not consistently [35] Z. Sun, D. Yu, H. Fang, J. Yang, X. Qu, J. Zhang, winning recommender systems competitions yet? C. Geng, Are we evaluating rigorously? bencha position paper, in: Proceedings of the Rec- marking recommendation for reproducible evaluommender Systems Challenge 2020, RecSysChal- ation and fair comparison, in: Fourteenth ACM lenge ’20, Association for Computing Machinery, Conference on Recommender Systems, RecSys ’20, New York, NY, USA, 2020, p. 44–49. URL: https: Association for Computing Machinery, New York, //doi.org/10.1145/3415959.3416001. NY, USA, 2020, p. 23–32. URL: https://doi.org/10. [24] Z. Gantner, S. Rendle, C. Freudenthaler, L. Schmidt- 1145/3383313.3412489.
Thieme, MyMediaLite: A free recommender system [36] W. X. Zhao, S. Mu, Y. Hou, Z. Lin, K. Li, Y. Chen, library, in: 5th ACM International Conference on Y. Lu, H. Wang, C. Tian, X. Pan, Y. Min, Z. Feng, Recommender Systems (RecSys 2011), 2011. X. Fan, X. Chen, P. Wang, W. Ji, Y. Li, X. Wang, [25] S. Vargas, Novelty and diversity enhancement J.-R. Wen, Recbole: Towards a unified, comprehenand evaluation in recommender systems and in- sive and eficient framework for recommendation formation retrieval, in: Proceedings of the 37th algorithms, 2020. arXiv:2011.01731. international ACM SIGIR conference on Research [37] P. Zhao, K. Xiao, Y. Zhang, K. Bian, W. Yan, Amer: & development in information retrieval, 2014, pp. Automatic behavior modeling and interaction ex1281–1281. ploration in recommender system, arXiv preprint [26] M. D. Ekstrand, Lenskit for python: Next- arXiv:2006.05933 (2020).
generation software for recommender systems ex- [38] Y. Chen, Y. Yang, H. Sun, Y. Wang, Y. Xu, W. Shen, periments, in: Proceedings of the 29th ACM Inter- R. Zhou, Y. Tong, J. Bai, R. Zhang, Autoadr: Autonational Conference on Information & Knowledge matic model design for ad relevance, in: ProceedManagement, 2020, pp. 2999–3006. ings of the 29th ACM International Conference on [27] M. Kula, Metadata embeddings for user and Information & Knowledge Management, 2020, pp. item cold-start recommendations, arXiv preprint 2365–2372.
arXiv:1507.08439 (2015). [39] T.-H. Wang, X. Hu, H. Jin, Q. Song, X. Han, Z. Liu, [28] N. Hug, Surprise: A python library for recom- Autorec: An automated recommender system, in: mender systems, Journal of Open Source Software Fourteenth ACM Conference on Recommender Sys5 (2020) 2174. tems, 2020, pp. 582–584. [29] G. Guo, J. Zhang, Z. Sun, N. Yorke-Smith, Librec: A [40] R. Anand, J. Beel, Auto-surprise: An automated java library for recommender systems., in: UMAP recommender-system (autorecsys) library with tree Workshops, volume 4, Citeseer, 2015. of parzens estimator (tpe) optimization, in: Four[30] M. Kula, Spotlight, https://github.com/maciejkula/ teenth ACM Conference on Recommender Systems, spotlight, 2017. 2020, pp. 585–587. [31] L. Yang, E. Bagdasaryan, J. Gruenstein, C.-K. Hsieh, [41] H. Liu, X. Zhao, C. Wang, X. Liu, J. Tang, AutoD. Estrin, Openrec: A modular framework for ex- mated embedding size search in deep recommender tensible and adaptable recommendation algorithms, systems, in: Proceedings of the 43rd International in: Proceedings of the Eleventh ACM International ACM SIGIR Conference on Research and DevelopConference on Web Search and Data Mining, 2018, ment in Information Retrieval, 2020, pp. 2307–2316. pp. 664–672. [42] J. Bergstra, R. Bardenet, Y. Bengio, B. Kégl, Algo[32] J. Yu, M. Gao, H. Yin, J. Li, C. Gao, Q. Wang, Gen- rithms for hyper-parameter optimization, in: 25th erating reliable friends via adversarial training to annual conference on neural information processimprove social recommendation, in: 2019 IEEE ing systems (NIPS 2011), volume 24, Neural InforInternational Conference on Data Mining (ICDM), mation Processing Systems Foundation, 2011.
IEEE, 2019, pp. 768–777. [43] D. P. Kingma, J. Ba, Adam: A method for stochas[33] U. Gupta, S. Hsia, V. Saraph, X. Wang, B. Reagen, tic optimization, arXiv preprint arXiv:1412.6980 G.-Y. Wei, H.-H. S. Lee, D. Brooks, C.-J. Wu, Deep- (2014). recsys: A system for optimizing end-to-end atscale neural recommendation inference, in: 2020