The Advertising Prediction Model Based on Machine Learning Technologies Anastasiia Ivanytska a, Liudmyla Zubyk a, Yaroslav Zubyk b and Dmitry Ivanov c a Taras Shevchenko National University of Kyiv, Gavrilishyna St., 24 B., Faculty of Information Technology, Kyiv, 04116, Ukraine b National University of Water and Environmental Engineering, Soborna str., 11, Rivne, 33028, Ukraine c Institute of Applied Mathematics and Mechanic, Generala Batyuka str., 19, Slavyansk, 84116, Ukraine Abstract The existing methods and models of forming predictions and forecasting in recommendation systems were analyzed for solving the problem of advertising selection. The expediency and efficiency of using machine learning technologies for the analysis of user preferences were substantiated. The formalization model of recommendations that based on technology of machine learning was proposed. After analyzing of different methods, the product-based collaborative filtering (CF) was chosen to solve the research problem. For calculations of the products similarities by the correlation it is proposed algorithm that based on matrix factorization. Its efficiency was verified on difficult test data and the estimation on adequacy of model on the basis of standard deviation was performed (RMSE). In contrast to the existing approaches, model showed an increase in the accuracy of personal recommendations. It implements in on-line stores of any complexity as a separate module for the supporting of advertising campaigns of the enterprise. Keywords 1 Forecasting recommendations, machine learning, collaborative filtering, matrix factorization 1. Introduction For now, there are a large amount of data with a complex structure that needs to be stored and processed with business logic. For end users it is difficult to process this amount of information. Web- sites and large media resources use recommendation systems with a powerful mathematical apparatus for fast data processing and information finding. Usually, these systems are used approaches that based on machine learning, Big Data technology, Internet of Things (IoT) etc. It is known [1, 2] that recommendation systems (RS) are one of the most powerful machine learning systems that online stores are implementing to increase sales. They are constantly improving and developing for current needs. The problem of selecting advertising based on user preferences is actual and important for small and medium-sized businesses, where funds for advertising have certain limitations. In this paper the state of development of RS, existing methods and models of data processing, artificial intelligence tools that used to form recommendations and forecasts purchases and their characteristics were studied. Existing systems for selecting advertising based on user preferences, depending on the method of working with data are divided into four types, namely content-system, systems with knowledge-based filtering, systems based on collaborative filtering (CF), hybrid systems [2-6]. Both types of methods (sequential and parallel) are used to build hybrid systems. To rating formation the parallel methods are used weighted, combined, switching and combined approaches. Consecutive methods of data analysis include a cascading approach, based on increasing the value based on characteristics and meta-learning [7-8]. In [9], to form recommendations for films selection the authors implemented a probabilistic latent Gaussian semantic analysis in the method of collaborative filtering. In these works, the PC was tested on open datasets, which does not allow to fully investigate the reaction of users to the proposed recommendations and the impact of botnets on these statistics. A web-application for working with neural networks based on the architecture of convolutional neural Information Technology and Implementation (IT&I-2021), December 01–03, 2021, Kyiv, Ukraine EMAIL: a.titova.wk@gmail.com (A. 1); l.v.zubyk19@gmail.com (A. 2); zubyk123@gmail.com (A. 3); dmitry.ivanov.iamm@gmail.com (A. 4) ORCID: 0000-0002-4803-2090 (A. 1); 0000-0002-2087-5379 (A. 2); 0000-0002-0802-3552 (A. 3); 0000-0001-9956-6589 (A. 4) ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 35 networks (VGG-16 architecture) was developed due to the best test results when working with graphic images. However, this does not reduce the impact of an external destabilizing factor - information attack by injection of profiles during the operation of the MS [10]. Modern methods and models for forming of recommendation have their advantages and disadvantages. The latter ones [3-7] include the problems of forecast accuracy, data processing speed and insufficient quantity and quality of input data for stable operation of the RS. Therefore, to solve the problem of selecting ads that meet the interests of the user, it’s necessary to create the improved model. It should be based on the collected data, offers related products to the customer and combines the methods of collaborative filtering and matrix factoring as variant. The system that based on this approach will help to the customers to choose a product on a site with adequate value for it, and return for further purchases, resulting to increase the business income. 2. Purpose of the study The aim of this study is to develop the formal model of forming recommendations that based on machine learning technologies and increase the accuracy of providing recommendations based on user preferences. To achieve this goal, it is necessary to solve the following problems:  analyze of the methods and models of forming recommendations in existing systems;  determine the stages of building of model for formation of recommendations based on machine learning technologies;  evaluate the adequacy of the proposed model on the test data. 3. Related work Modern recommendation systems perform recommendations or forecasts of goods, services and entertainment basing on the methods and models of varying complexity of input data processing. To build a movies rating, the estimations algorithm was proposed [5]. It is based on cosine similarity algorithms that form a weighted average of ratings. A cosine similarity criterion calculates the cosine of the angle between the vectors of digitized preferences of two users. At the studies [5, 6] the categorical, mixed and numerical clustering of data and modified the method of clustering similarity were used. They add the property of vagueness in the model of preparation of target proposals that based on user preferences and user activities in social networks. Combined methods of CF data [6] allow to calculate the required rating as a result of usage of two methods at the same time, namely filtering based on the similarity of the buyer and the product. When using collaborative data filtering that based on the models, the recommendation models are formed, which are implemented such approaches as Bayesian network, cluster analysis, Markov models, low-ranking matrix approximation [2, 7-9]. Studies [9-11] present RS and review of recommendation models based on various methods that used to predict student performance, such as CF k-nearest neighbors, MF and biased factorization, and predict student performance. The authors [12, 13] developed the system that used machine learning algorithms based on the CF model to predict the rating of users for elements without evaluation. The analysis of existing methods revealed that the mathematical apparatus is relevant to a selected problem, and accuracy indicators work well for specific data. In this paper it is proposed to evaluation existed approaches to data processing that based on machine learning technologies and develop a new model of recommendations. When developing the model, it is necessary to take into account the preferences of users and their personal data, choose modern software based on packages for rapid testing and testing of results on large and complex data. 4. The construction of recommendations model that based on machine learning technologies 4.1. Analysis of approaches to building of recommendations systems It is known that RSs function with two types of data: character information (data about items (keywords, categories) and users (settings, profiles)) and information about user-element interaction and the information includes estimates, number of purchases, reactions, and so on. Basing on this, four types of systems were separated and choice depends from sort of data processing: 1. Content systems (content-based systems) that use specific subject area information. 36 2. Knowledge-based filtering systems, which are based on previous user actions and features of their interactions with the site. 3. Collaborative filtering systems - based on user-site interaction. 4. Hybrid systems that combine both approaches of information processing to avoid problems that occur when working with only one species. According to the practical use, all the above approaches have certain advantages and disadvantages that affect on the choice of method of construction of the RS. For example, the allowable modes of calculation in expert and hybrid RS depend on the applied mathematical methods and requirements for the immediacy of the original data. On other hand, if the product range of the online store involves its infrequent updating and it’s need to buy many products together with others, that complement their functions or used in their selection, then it is preferable to use CF based on items as a method of building RSs. For an online store that houses specific products (such as tools and kits for a particular activity) that are of interest of specified stable target audience of customers, and new customers don't appear often, then the implementation of RC with a user-based CF method is suitable. In turn, in an on-line store where there is a lot of information about customers and recommended products or such data can be obtained in sufficient quantities, it used to content-based filtering methods. To summarize the analysis of the methods of advertising selection, presented analysis agenda in table. Table 1 Analysis of advertising methods in accordance with the solving tasks Problems Filtration on Filtration on Collaborative Hybrid filtration content knowledge filtration Cold start Non-available Non-available Available Non-available problem The complexity of Low Average Low High implementation Specifics of work Online stores, Online stores Online stores, Any area movie portals. movie portals. Accuracy of Average High Average High recommendations Potential speed High Average Average High Dependence on Non-available Non-available Available Depends on the other users implementation As seen from the table, CF approaches are not difficult to implement and they are used for different subject areas and have some averages for other problems. Fig. 1 present the categories of CF data schematically. For further development of the model of recommendation formation in our subject area it’s necessary to choose the indispensable methods among these categories. After analyzing of different methods such as collaborative filtering, the product-based collaborative filtering (CF) and matrix factorization some of them were chosen to solve the research problem. The CF algorithm based on objects implements in three stages:  Construction of a goods matrix to determine the degree of similarity between them.  Calculation of new products on the basis of the selected similarity criterion.  Offering products in the top-level system. Therefore, the set of elements is considered depending on which ones the user has evaluated, and the similarity to a specific element 𝑖 is calculated, and then 𝑘 the most similar products are selected {𝑖1 , 𝑖2 , … , 𝑖𝑘 }. During this process it is necessary to calculate the corresponding similarities {𝑠𝑖1 , 𝑠𝑖2 , … , 𝑠𝑖𝑘 }. Then, when the most similar products are found, the forecast is calculated by taking the weighted average amount of user estimates for similar items. The next important step in CF based on product similarity is to calculate the similarity between two elements 𝑖 and 𝑗 to isolate users that have evaluated both of these elements, and then use the chosen 37 method of calculating similarity to determine similarity 𝑆𝑖,𝑗 . Figure 2 shows how the rows of the matrix expose the users and the columns – the products. Figure 1: Approaches to collaborative data filtering Figure 2: Image of the process of calculating the similarity of the estimated elements After analysis [9-11, 12], it’s clear that the following methods are used to calculate similarity: based on cosines, Pearson correlation coefficient, Jacquard coefficient and similarity based on adjusted cosines. Determining similarity that based on a basic cosine measure when filtering data based on products has a drawback: the rating scale between different users doesn’t take into account the overall rating and similarity search in pairs of sets that may correspond to one user. The adjusted cosine similarity compensates for this shortcoming by subtracting the average value of the 𝑢 -th user for each of the overall ratings. That’s why, there is a normalization of data by elements. To implement the selected method, the similarity between the elements 𝑖 and 𝑗 is calculated as follows: 38 𝛴𝑢∈𝑈 (𝑅𝑢,𝑖 − ̅̅̅ 𝑅𝑢 )(𝑅𝑢,𝑗 − ̅̅̅ 𝑅𝑢 ) (1) 𝑐𝑐𝑠𝑖𝑚(𝑖, 𝑗) = 2 2 √𝛴𝑢∈𝑈 (𝑅𝑢,𝑖 − ̅̅̅ 𝑅𝑢 ) ∗ √𝛴𝑢∈𝑈 (𝑅𝑢,𝑗 − ̅̅̅ 𝑅𝑢 ) where ̅̅̅̅ 𝑅 𝑢 – the average value of the rating of the 𝑢-th user. According to the method of matrix factorization user's preferences determined by a small number of hidden factors. Then the space of characteristics where it's possible to represent both users, products and interactions between them, received by means of a scalar product between corresponding vectors in this space. It’s need to find the values of matrices 𝑈 (users) і 𝑉 (products) so that the difference of square errors between the points of the product and the known rating (rating) in the matrix of users is run to the minimum. It shown as follows: 𝑚𝑖𝑛(𝑢, 𝑣) = ∑ (𝑅𝑢,𝑗 − 𝑣𝑗 ∗ 𝑢𝑖 )2 (2) (𝑖,𝑗)∈𝑘 To track the quality of the recommendations of viewed methods, it is proposed to choose metrics from the following: root mean square error (RMSE), mean absolute error (MAE), normalized value of the mean absolute error (NMAE). According to the training part it is carried out on the basis of correlation of grades, and according to the test part the accuracy is checked. It is necessary to assess the adequacy of the model basing on the standard deviation of the expression: 𝑚 𝑛 (3) 1 𝑅𝑀𝑆𝐸 = √ ∑ ∑(𝑟𝑖.𝑗 − 𝑟̂𝑖.𝑗 2 ) 𝑝 𝑖=1 𝑖=1 where 𝑟𝑖.𝑗 – real value of subject estimate 𝑗 by user 𝑖 ; 𝑟̂𝑖.𝑗 – prognostic value of subject estimate 𝑗 by user 𝑖 ; 𝑝 – total count of values in text in test part; 𝑟𝑚𝑎𝑥 і 𝑟𝑚𝑖𝑛 – max and min value of subject estimate in test matrix. 4.2. Coverage of the stages of building a data model After the analysis of methods and models of CF data it's necessary to form stages of construction of recommendations formation model. Tables from Amazon.com were used for the study. It contains products of the beauty segment, where the information is presented in the following rows: User, Product, Rating, Timestamp (UserId, ProductId, Rating, Timestamp). For forming of recommendations, the following stages of model construction are proposed: 1. Read the input data. 2. Identify the similarity of users based on their ratings and personal data. 3. To form the list of the products that have received more estimations than threshold value. 4. Calculate the similarities of products by the correlation algorithm based on matrix factorization. 5. Build and display user recommendations. As presented, the algorithm consists of five main steps ending the product recommendations in the form of images and ID_PRODUCT for the end user. The input of the algorithm provides information about products and evaluations made by users about these products, the requirements of the online store. At the first stage it is important to understand: what user gave the assessment, what product and how many points is this assessment. To calculate the similarity of users, by random coincidence of interests based on estimates, it was accepted to use the formula for determining similarity based on the basic cosine measure (1). This CF method calculates the weight of all matches for further compilation of a list of similar interests of users. Then, to form a list of products that received more ratings than the threshold value (80), it's necessary to filter products for which more than 80 users responded. The threshold value was obtained as a result of a practical experiment, because for products that received less estimates, it's difficult to build a correlation matrix. 39 To calculate the similarity of products basing on matrix factorization the next formula factorization (2) is used, and the formation of the matrix for correlation between all selected products is as follows: (4) 𝑋𝑇 𝑋 С= 𝑁 where 𝑋 − the matrix, in the 𝑖 -th line of which there are evaluations of goods, which were evaluated by the 𝑖 -th user; 𝑁 − number of users. It should be noted that the matrix 𝑋is pre-normalized by columns by expression: 𝑥𝑗 − 𝑚𝑒𝑎𝑛(𝑥𝑗 ) (5) 𝑥𝑛𝑒𝑤 𝑗 = 𝑠𝑡𝑑(𝑥𝑗 ) where 𝑥𝑗 − is 𝑗-th column of the matrix; 𝑚𝑒𝑎𝑛(𝑥𝑗 ) − the average value of 𝑗-th column of the matrix; 𝑠𝑡𝑑(𝑥𝑗 ) − standard deviation of 𝑗-th column of the matrix. After that, Pearson's method was used for the set of products contained in the correlation table. The core of the method is that the rating is assigned to the product based on ratings of other similar products that are already included by the buyer in the order. The combination of collaborative filtering based on product and matrix data factorization is easy to implement, so the introduction of the model in the subsystem of advertising selection takes less time and money. The result is the formation of a recommendation and demonstration of images of related or similar products to the end user. 4.3. Evaluation of the adequacy of the proposed model on test data The ratings_Beauty.csv file, which the recommendation model works with, contains about 2,023,000 rows, with 36 unique customers and more than 3,700 products on the site. To assess the accuracy of the forecast and the recommended product to the user, depending on the him preferences (there is feedback), the calculation of the standard deviation (RMSE) by expression 3. Training and testing of data based on a neural network such as perceptron, was presented after testing of model by an independent user. The products obtain ratings, and then the rating of this product is calculated for all users, i.e., the average value. For example, a face cream received 20 ratings from 3 to 5, and as a result, the average value for this product is 4.3. After constructing the correlation field for the product rating, the following graph was obtained (Fig.3). As presented on Figure 3, rated products don’t form a very strong correlation, but obviously products with a lower rating are less rated by site users. Then the statistical parameters are calculated (average value for the corresponding products, std, min, etc.). Building a frequency histogram of the distribution of products by rating based on average user ratings and present in Figure 6. To apply CF, needs to choose only those products that are rated at least by 80 users. Next, building a matrix using the function pd.pivot_table, perform normalization and correlation of data. To test the model formation of recommendations by the independent user, it is necessary to distinguish two products - sunscreen and toner. A table showing the first five products with a rating of more than 80 users, will get after sorting of the resulting data. It's presented below. This table is used to search out which products correlate with the first product that the user recorded in the system. That is, products with the following ID_PRODUCT are best correlated with B009FKNGGQ: B009FKNGRA, B002MO3CG6, B008U2Y9BQ, B00AO379NE, B00604MSSC. Products hidden from ID_PRODUCT seen as images in the following pattern of matching the first product, shown in Figure 5. As shown on Figure 5, the model of forming recommendations allows to choose among products with similar ratings, which allows the user to quickly decide on a purchase. To verify the correct operation of the model, it is necessary to conduct another stage of testing the operation on different input data. For example, it's need to select the following three products to be preferred, ID_PRODUCT: B000NWDSAE (rating - 5.0), B001E96MZ8 (5.0), B001410OG4 (4.0). Next, it is need to select a column from the matrix, which is stored in a variable, and remove the zero ratings from this vector. Products matching ID_PRODUCT and rating are shown in Figure 6 below. 40 Figure 3: Image of the correlation field of the input data Figure 4: Rating distribution by frequency of user reactions Next step - get the products with the same estimates in the correlation matrix and scale the similarity according to user estimates. That is, the ratings are actually updated where the user's preferences were zero. The Pearson's correlation is used in conjunction with scalar multiplication of a series of correlation matrices corresponding to the product for which the rating is updated on the vector containing the ratings of all affiliated preferences. It allows to get a list of results - products (ID_PRODUCT), as in Table 4. Table 3 Number of users rating products with appropriate correlation ProductId Correlation Count B009FKNGGQ 1.000000 240.0 B009FKNGRA 0.415435 237.0 B002MO9BW0 0.334538 75.0 B002MO3CG6 0.333367 89.0 B008U2Y9BQ 0.330986 211.0 B00AO379NE 0.305995 135.0 41 Figure 5: Display of product selection results to the end user Figure 6: Display of test products Table 4 List of resulting products sorted by rating ProductId Real rating B001E96MZ8 4.980668 B000NWDSAE 4.973054 B000ZMBSPE 4.967862 B004NZDY42 4.922096 ... ... B00027EG9C 4.823046 B0043TURKC 4.812491 The holdout method is used to assess the accuracy. The data stack is divided into two sets - training (2/3 of the data is 130 0000 rows) and a set of tests (1/3 of the data is 600 000 rows of the table). The RMSE criterion was obtained for several advertising selection methods, namely CF, MF and CF + MF. After classification (see table 5) the division into data with the highest estimations of users and with falling of the minimum value of an error (Err) and the number of epochs is formed. As demonstrated on Table 5, the combination of CF and MF methods based on user preferences have smaller standard deviations between the predicted estimate and the actual product estimate and the learning rate among those selected for analysis. This indicates the accuracy and speed of the proposed method. The implemented model of forming recommendations based on machine learning technology – a combination of two methods of CF and MF based on user preferences was used as an opportunity to evaluate products during testing. In this regard, consider the main identified benefits. 42 Table 5 RMSE criterion and training time for different methods of recommendation formation Methods of recommendation formation RMSE Err Train time Collaborative filtration 0.0977 0.00031 12615 Method based on matrix factorization 0.0993 0.00028 13759 CF+MF 0.0864 0.00027 11054 4.4. Software implementation of the model of forming recommendations in Python and its implementation To attract customers and increase profitability, business owners need to build advertising campaigns so that customers return and become loyal. A combination of data filtering techniques is used to generate personalized recommendations and display it. That is, the company (the owner of the site) connects the recommendation model (subsystem selection of advertising based on user preferences) to the existing platform according to existing architecture. Further, the system performs the formation of recommendations by selected algorithms (namely the method of collaborative filtering by products with the cosine similarity algorithm) and matrix factorization with the Pearson correlation algorithm. After that, the data are stored in a file or in the current database. Data uploads at different intervals, for example, of 12 hours (usually 24 hours), then the subsystem customer data is copied to the cloud. Before the implementation of the proposed subsystem of advertising selection, it is necessary to implement a server for issuing and displaying recommendations to the end user. The main objects of the subsystem of advertising selection are site users, products and their evaluations. Next, it is necessary to determine the functionality and behaviour of the system with input and output data, to describe the subsystem being developed. The input data of the system used to not only data about a particular user, but general information about user's behaviour on the site, quantitative and qualitative characteristics of products and context also. In addition, when using recommendations for online stores, most of the available data are the data that allows to visit the website in a hidden form (imperfect data, hidden personal user data, etc.). When designing the software module, the following options for using the system were identified: 1. building a model for the formation of recommendations - the user selects the appropriate option; the system builds a software model of recommendations based on a combination of two algorithms; 2. obtaining parameters - the user enters input data to search for products that meet the recommendations, the system receives data through controls; 3. providing information about the recommendation - the system provides information related to the results of product selection for a particular user; 4. data storage - the file system provides the user with information about TXT type files, the user selects or creates a file to save product icons and ID_PRODUCT, the system stores data in the appropriate file. The software module was developed using the Python programming language in the Jupyter Notebook editor using the Anaconda service and the XAML user interface development language. The program is intended for use on personal computers running Windows, Linux or Mac OS X. In turn, the proposed software module for the selection of advertising plays a key role in the project, and for further demonstration and implementation of the subsystem, the following modules are noted: input data processing module; model of forming recommendations; user interface module. When assessing the adequacy of the model, the results of the software solution for each point of the algorithm of the model of forming recommendations were considered. 5. Conclusions As a result of the research, a model of forming recommendations basing on user preferences, that uses machine learning technologies, was developed and implemented. The following stages of research were performed. 43 Firstly, the existing methods and models of data processing, artificial intelligence tools that used to build advertising selection systems were analyzed. The methods of collaborative filtering and matrix factorization were selected to develop a recommendation model that, in contrast to existing approaches, have shown an increase in the accuracy of personal recommendations. Secondly, the structure of recommendation system was formed and new model on the basis of a combination of few methods of data processing was developed that, in contrast to the available tools, allowed to take into account the personal preferences of users, reduce the negative effect of the complexity of the initial data and simplify the software implementation of the proposed model. Thirdly, the model was tested on test data and the adequacy of this model was evaluated. It based on the standard deviation, that showed high accuracy of the proposed model on complex data with not less than 91.39%. This indicates the reliability of results and the adequacy of this model that used to an on-line stores of any complexity, as a separate module for the implementation of advertising campaigns of the enterprise. 6. References [1] L. Onokoy, J. Lavendels. "Modern approaches to building recommender systems for online stores." Applied Computer Systems, 2019, 24, pp. 18–24. doi:10.2478/acss-2019-0003 [2] A. Hernando, J. Bobadilla, F. Ortega. "A non negative matrix factorization for collaborative filtering recommender systems based on a Bayesian probabilistic model." Knowledge-Based Systems, 2016, volume 97, pp. 188–202. doi:https://doi.org/10.1016/j.knosys.2015.12.018 [3] Item-Based Collaborative Filtering in Python. The practice of making the item-based collaborative filtering in python. URL: https://towardsdatascience.com/item-based-collaborative-filtering-in- python-91f747200fab. [4] M. Lobur, М. Shvarts, Y. Stekh. "The method of sequential clustering for predicting recommendations." CAD in Machinery Design-Implementation and Education Problems: Proc. of the XXV Polish-Ukrainian Сonference: 20–21 October 2017, Bielsko-Biała, 2017, pp. 19–20. [5] Y. Koren. "Factorization meets the neighborhood: a multifaceted collaborative filtering model." International conference on Knowledge discovery and data mining: Proceedings of the 14th ACM SIGKDD, 2008, ACM, 2008, pp. 426–434. doi:10.1145/1401890.1401944 [6] H. Thanh-Nhan, H. Nguyen, N. Thai-Nghe. "Methods for building course recommendation systems. " Eighth International Conference on Knowledge and Systems Engineering (KSE), 2016, pp. 163–168. ISBN 978-1-4673-8929-7, IEEE. [7] R. Burke, P. Brusilovsky, A. Kobsa, and W. Nejdl. "Hybrid web recommender systems." The Adaptive Web - Methods and Strategies of Web Personalization, Springer-Verlag Berlin Heidelberg, 2007, pp. 377–408. doi:10.1007/978-3-540-72079-9_12 [8] V. Martsenyuk. "Indirect method of exponential convergence estimation for neural network with discrete and distributed delays." Electronic Journal of Differential Equations, 2017, 246, pp. 1–12. [9] O. Bychkov, O. Ivanchenko, K. Merkulova and Y. Zhabska. "Mathematical Methods for Information Technology of Biometric Identification in Conditions of Incomplete Data." Proceedings of the 7th International Conference "Information Technology and Interactions" (IT&I-2020), Kyiv, Ukraine, December 02-03, 2020, pp. 289–291. [10] H. Steck. "Training and testing of recommender systems on data missing not at random." International conference on Knowledge discovery and data mining, Proceedings of the 16th ACM SIGKDD: 2010, ACM, 2010, pp. 713–722. doi:10.1145/1835804.1835895 [11] N. Thai-Nghe, L. Drumond, L. Schmidt-Thieme. "Recommender system for predicting student performance." In Proceedings of the 1st Workshop on Recommender Systems for Technology Enhanced Learning: 2010, volume 1, Elsevier's Procedia CS, 2010, pp. 2811–2819. [12] T-N. Nguyen, L. Drumond, T. Horvath. "Using Factorization Machines for Student Modeling." Proceedings of FactMod 2012 at the UMAP 2012, volume 872, CEUR-WS. [13] B. Sarwar, G. Karypis, J. Konstan, and Riedl J. "Item-based Collaborative Filtering Recommendation Algorithms." Proceedings of ACM World Wide Web Conference: 2001, volume 1, 10, pp. 285–295. 44