Recommender systems are quickly becoming a part of our daily digital life. Mainly found in applications such as e-commerce, social media, and online entertainment services. They help users overcome the information overload problem by improving the browsing and consumption experience. Mandarine Academy is an Ed-Tech company that operates more than a hundred online e-learning platforms. They creates online pedagogical content (videos, quizzes, documents, etc.) on daily basis to support the digitization of work environments and to keep up with current trends. Suggesting items that are relevant to both users and visitors is challenging, the company is looking for ways to improve the learning experience by providing content that adheres to specific conflicting requirements. These requirements include similarity with user profile, the novelty of proposed content, and diversity of recommendations. Mandarine Academy is looking to implement an approach that can handle multiple conflicting goals with the possibility to adjust which to use in each browsing scenario. In this article, we propose a solution for Mandarine Academy Recommendation System ( ) problem by using Evolutionary Algorithms based on the concept of Pareto Ranking. After modeling objectives (Similarity, Diversity, Novelty, RMSE, and nDCG@5) as an optimization problem, we compared diferent algorithms (NSGA II, NSGAIII, IBEA, SPEA2, and MOEAD) to study their performance under diferent test settings. Extended data analysis of real-world user interactions showed drawbacks of many graphical issues that prevented users from learning eficiently and we proposed enhancements to the overall user experience and interface. We discuss initial findings under various objectives which show promising results considering production mode scenarios. A proposed custom mutation operator was able to outperform the classical swap mutation. A multi-Criteria Decision-Making phase that uses by default pseudo weight is responsible for providing results for end users after training our model. CCS Concepts: • Applied computing → Multi-criterion optimization and decision-making; E-learning; • Information systems → Retrieval models and ranking. Mandarine Academy is an Ed-tech company that supports the digital transformation of work environments by facilitating the handling and use of new technologies by all employees. It ofers a new way of training more efectively in terms of skills, capacity, time, and budget due to an exclusive approach that combines both digital learning and personalized support. With over half million users and more than 100 platforms, the company operates several products for multiple partners in varying sectors. Mandarine Academy saw how Massive Open Online Course (MOOC) impacted the traditional higher education market as well as the increasing rate of industry digitization and proposed custom MOOCs focused on the corporate sector to save employees from obsolescence.
∗Copyright 2022 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Presented at the MORS workshop held in conjunction with the 16th ACM Conference on Recommender Systems (RecSys), 2022, in Seattle, USA.
One of their most popular products is the Mooc-ofice365-training 1, a public bilingual (French & English) MOOC destined for learning Microsoft ofice 365 tools and soft skills related to workplaces. Having around four thousand monthly active users and more than hundred and thirty thousand registered users, the platform includes diferent types of learning materials such as: • Resources: Tutorials & use cases (short format videos), quizzes, documents, recorded live conferences, SCORM (Shareable Content Object Reference Model). • Courses: A collection of unordered learning resources.
• Learning Paths: A predefined set of courses to master certain skill/job/specialization.
In this work, we focus on video resources (tutorials & use cases) because they make up the majority of the
Moocofice365-training’s catalog. The company is providing up-to-date content that matches changes in work environments
and current trends. Unfortunately, this impacts users as they have to spend more time selecting the appropriate learning
material which can lead to known problems such as information overload, distraction, disorientation, lack of motivation
[
To reduce the time spent searching for relevant information, scientific literature proposed multiple approaches that matches users with relevant content through the use of recommender systems. Unfortunately, most of the works deal with academic recommender systems which is diferent from corporate objectives.
In this paper, we investigate and solve problems encountered in a public e-learning platform operated by Mandarine Academy. After reviewing the literature and analysing user behavior by collecting real-world interactions (explicit and implicit), we proceed to mathematically represent company objectives and constraints into an optimization problem. In addition, we perform a critical analysis of the user interface and experience to understand its impact on a user’s learning journey. Our approach combines both traditional recommender system techniques with metaheuristics in order to recommend relevant content to users. The established experimental protocol aims to compare the performance of many evolutionary algorithms and provides an in-depth analysis of their results. Finally we give insights about production mode findings and future research directions. The rest of the paper is organized as follows: Section 2 provides an overview of related works. In section 3, we discuss the findings of data analysis done on Mooc-ofice365-training and list diferent graphical issues. In section 4, we showcase our proposed approach. Section 5 provides experimentation design and results analysis. Section 6 concludes the paper and gives directions for future work. 2
In this section we give a brief description of Multi-Objective Optimization (MOO) and common recommendation system techniques before diving into related scientific studies.
Real-world optimization problems are rarely mono objective, instead we end up with many conflicting goals. This can be defined as optimizing ( ) = ( 1 ( ), 2 ( ), ..., ( )) with ∈ , is number of objectives ( ≥ 2), being a 1https://mooc.ofice365-training.com/
Manuscript submitted to ACM
vector of decision variables, a set of feasible solutions and ( ) represents each objective that we want to minimize
or maximize [
Metaheuristics represent a family of approximate optimization strategies that ofer acceptable solutions to complex
problems in reasonable time. Unlike exact optimization algorithms, metaheuristics do not guarantee that the results are
optimal [
Before we can assess "how good" any single run of a Multi-Objective Evolutionary Algorithm (MOEA), we must first
grasp two concepts. The first is convergence, which indicates how “close” we come to find the best solution. The second
metric is diversity which measures if solutions are fully spread throughout the set or are clustered together. The following
quantitative metrics can provide an evaluation mechanism for both convergence and diversity [
Recommender Systems (RS) are algorithms aimed at suggesting relevant items to users. This is made possible by ifltering massive amounts of data that can be obtained from users, content, or other sources (e.g. context).
Recommender systems are becoming a part of our daily digital life. Mainly found in online entertainment services,
e-commerce, and social networks [
Generally, the most used types of recommender systems are: Collaborative Filtering (CF) and Content-Based filtering
(CBF). The main diference between these two techniques is the type of data employed. The CBF approach requires
items metadata to recommend content having similar attributes with the user profile [
We found that existing recommendation algorithms (CF and CBF) do not perform well when evaluated in terms
of accuracy, novelty, and diversity. Approaches that exploit the combination of such recommendation algorithms are
known as Hybrid Recommenders [
Xie et al. [
Zuo et al. [
Our work addresses diferent aspects that are missing or under-exploited in the aforementioned works: • The use of multiple recommendation engines to initialise our solution population and provide more diversity. • Working with real-world implicit ratings to train our model. • We propose a customized mutation operator to improve diversity of recommended items. • Performance comparison of various Multi-Objective Evolutionary Algorithms (MOEA). • Optimizing five conflicting objectives in the context of a real-world problem. • The use of parameter tuning to optimize algorithms depending on user behaviour and selected objectives. • Work is being integrated in production-ready environment. 3
The more you know about your users, the better equipped you’ll be to make informed decisions about your service. In order to gain a richer understanding of how users interact with content, events are used to independently track users journeys. A typical method of providing feedback is in the form of rating methods that captures users preferences in explicit ways (like button, social sharing, course/learning path registration and bookmarks). The disadvantage is that users tend to avoid the burden of explicitly stating their preferences. To overcome the shortage of explicit ratings, platforms tend to collect users behavior through multiple ways (page views, percentage of videos watched, etc). This is called implicit feedback. The advantage is that users can trigger a lot of actions when using a service. This generates a lot of data that can be significant in some cases but shows a major inconvenient, which is not having a ground truth. When running short on ratings (explicit or implicit), content descriptors like (subtitles, title, description, number of views, duration, etc.) are used as additional input to recommender systems.
We conduct our data analysis using the Mooc-ofice365-training platform (French version) with the following catalog: • 41 Learning Paths. • 142 Courses.
• 1294 Tutorials and 113 Use Cases. that other reasons beside avoiding to express their opinion might be possible. To confirm our hypothesis we observed the graphical interface available for both registered users and visitors. We list below our findings per page. (1) Home page: The current homepage ofers a list of newest courses and tutorials. Users / visitors are limited if they are looking to learn about certain tools, required skills for specific jobs or certification. What we propose for visitors is a list of items (courses and resources) with options to select popular or newer items. Furthermore, categories (skills, jobs, certificates) should be shown in top of the page to guide visitors eficiently. For registered users, a multiple personalized lists of recommended items provided by our approach and others (CF, CBF) will help users find relevant content easier. (2) Content page: Learning paths, courses and videos (tutorials and use cases) are presented to both users and visitors without similar items, visible interactions or feedback options. The like and share buttons are provided without text, only a small icon. In case a video doesn’t correspond to a user’s needs, they must go back to the previous page and spend additional time looking for another one. We propose for both users and visitors a more appealing interface with visible interactions (like, dislike, social share), addition of a "save to watch later (Bookmark)" and "feedback" options. Multiple recommended lists (CF, CBF, Popularity) of similar items to minimise the burden of content search and provide guidance.
Initial propositions address the way content as well as interactions are displayed and insists on improving both
visibility and readability for users. One of the proposed features (Bookmark) has already been integrated into the
platform and its shown in Table. 1, since it’s relatively new, additional data need to be collected before assessing its
significance. Furthermore, the search process is upgraded with more filters to empower users looking for specific
information. Finally, we highlight recommended content by improving its graphical positioning for users [
(1) (4)
The performance evaluation of our approach is highly dependent on the data it processes and the task it has to perform. In our particular context, this task is complex since it must satisfy diferent goals that are far from being complementary. We had to find a compromise satisfying both the need to match user taste, highlight diverse content and also focus on unpopular items. These goals represent what the company aspires to achieve. These requirements were updated with two additional objectives widely used in the literature and considered as a standard in recommender systems evaluation metrics.
(Objective 1) Maximize similarity with user profile. This is done by calculating the overall cosine score between items in user’s profile and items in the proposed recommendation.
= Í=0 Í=0 (, ) Where is the recommended list (solution) and is the item number from . The user profile is expressed as where is the item number from . is the item-item cosine distance score. is the length of the solution.
(Objective 2) Maximize diversity which is responsible for how dissimilar items are in the solution. This can be
achieved by using the Intra-List Similarity metric (ILS) [
= Í=−01 Í=1 (, ) (2) Where is the recommended list (solution) and is the item number from . is the item-item cosine distance matrix. is the item pairs count.
(Objective 3) Maximize novelty. In this objective, we aim to recommend less popular items and focus on content having both low number of views and recently added to the catalog. A scoring function that sums the number of views and number of days since release, returns the median. The smaller the median, the more novel the items are.
Í=0 ( ) = (3) Where is the recommended list (solution) and is the item number from . is novelty scoring function.
To add more flexibility to the proposed recommender system, two additional objectives were added. Root Mean Square Error and the Normalized Discounted Cumulative Gain . Both are well known metric and widely used in recommender systems research field.
(Objective 4) Minimize . What this metric does essentially is finding diference between a predicted rating and a real rating. Having lower error value, means our model is predicting ratings similar to what the user gave.
tv 1 ∑︁ = ( ) =1 ( − )2 Where is the actual rating, is the predicted rating and is the number of ratings.
(Objective 5) Maximize . This is a measure of ranking quality where highly relevant items are more useful when ranked first. Also this metrics follow the assumption that highly relevant items are more useful than marginally relevant items, which are in turn more useful than non-relevant items. We will be using nDCG@5 which corresponds to the number of relevant results among the first 5 recommended items. (5) With = Í|=1 | log2(+1) and = Í=1 lo2g2(−+11) . Where is a list of top relevant items (ordered by relevance). is the graded relevance of the result at position .
Even though both metrics reflect relevance, is crucial for ranking problems while isn’t relevant for that case. The intuition behind adding these objectives is to provide the ability for platform managers more options to tune from selecting only a single objective to combining many. For example, a platform wants to highlight on content similarity to users profile can use both (Objective 1) and (Objective 4).
Note that initial objectives don’t capture learning objectives of users. This is due to available events (explicit and implicit). Work is being conducted to incorporate learning performance tracking events to better suggest items for users with specific learning goals (job requirements or mastery of certain tools).
Since we are working on an optimization problem we must define our constraints in order to determine if a solution is feasible or not. Constraints can be defined as conditions that a solution must satisfy in order to be feasible. Two constraints are considered in this work: • Recommended list must be unique and contain no duplicates.
• Recommended items must not exceed the fixed length .
To define our problem, objectives and constraints, we chose JMetalPy [
Genetic algorithms begins with the choice of the chromosome encoding (solution representation). This depends on the
problem in-hand. Following the works of [
After defining a structure that represents our solutions, we proceed to generate an initial set or population of solutions
as input for our approach. Generally initial populations can be produced through many approaches (random, heuristics,
etc.) [
• Association Rules (FP-Growth).
We try to get a recommendation from each approach listed above for each user we test. In case an algorithm is
unable to provide personalized recommendations, we have defined a fallback method, which is the "Random" approach
that provides a list of randomly selected items. To implement most of these algorithms, the python library "Surprise"
[
Genetic algorithms usually applies a crossover operation to two solutions in order to produce new chromosomes
(solutions) called "children chromosomes". Operators like One-Point and Two-Point crossover are popular in the
literature [
Mutation is a genetic operation used to maintain genetic diversity. It introduces changes inside solutions in an attempt
to avoid local minima. Random mutation is frequently found in the literature [
To make use of initial population generators, platform managers will be able to create these models using an administration dashboard. We identified a total of three model types each with diferent associated algorithms: • Popularity: Provide popular content based on either view portions (default) or number of page visits. • Personalized: Based upon one method (User Based CF, Item-Based CF, Content-Based (default), Model-Based CF,
Metaheuristic, FP-Growth).
• Similar Items: Based upon one method (Item-Based CF, Content Based (default), FP-Growth).
Manuscript submitted to ACM Algorithm 1 Pseudo-code of custom mutation operator Require: , ℎ ← ℎ ∈ (, , ) ← ∈ ← ∈ [0.0, 1.0] if ≤ then for each Item ∈ Replaced Items do
← ℎ () end if
Additional advanced settings can be fine tuned for example, selecting objectives associated to each model and what interactions to use. This makes it possible to experiment with diferent parameters across diferent interface placements. This dashboard is still in work and will also provide monitoring features to track performance of each model in real-time by using metrics such as click-rate (), watch time and click through rate ( ). 5
Along with the implementation of the metaheuristic and diferent solution generators, we conducted a series of tests in a more experimental setting in order to compare diferent approaches for solving the recommendation problem with diferent objectives combinations.
Technically, platform managers can select any combination of objectives they desire. However, in our test settings we tested a subset of possible combinations to observe the impact of having multiple objectives on performance.
Each test setting has a parameter tuning phase to ensure each algorithm is using the best configuration for the task. The data in this experiment was collected from real-world users and were adapted to our approach. 5.1
Moving on to initial experiments where the first step was selecting the right data to work with. We have seen that explicit ratings in Table 1 sufer from low user engagement compared to implicit ratings in Table.2. Unfortunately for page views, there isn’t a ground truth that indicates if viewing a content page multiple times leads to increased user satisfaction. However, view portions (Watch time) might be a good fit for our approach as it can be used to measure user interest. Table. 3 details the diferent attributes and values present in view portions.
The dataset has a sparsity of 99.312%. With a total of 822 users, 776 items, and 3699 ratings. The company uses the following scale to describe viewing events: • (1) Not considered: viewings from 0% to 10% of the video. • (2) In progress: viewings from 11% to 69% are considered equal. (a) The current rating scale.
(b) The proposed rating scale.
• (3) Finished: viewings from 70% to 100% consider the user has finished watching the item.
We believe that the degree of viewing has an impact on the overall impression and that the previous scoring function does not reflect user interest. When plotting the old scoring scale shown in Fig. 1, the majority of users are in the "In progress" state followed by "Finished". This implies that users are still learning or have completed their videos. A new scoring system is proposed. It builds on the previous approach by incorporating more levels of appreciation. The assumption is that longer viewing times indicate a higher user interest and satisfaction. Considering the following scale: • (1) No interest: viewings from 0% to 20% of the video. • (2) Small interest: viewings from 21% to 40% of the video. • (3) Medium interest: viewings from 41% to 60% of the video. • (4) High interest: viewings from 61% to 80% of the video.
• (5) Finished: viewings from 81% to 100% of the video.
This introduces 5 diferent levels that have varying degrees of importance and reassembles the classical 5-star rating system. When plotting the new scoring system, a diferent narrative is found. The majority of users fall into the "No interest" category, followed by "Finished" and "Small interest". Diferent assumptions are made here, starting with the highest group count "No interest" which indicates that users watched a maximum of 2 seconds on a scale of 10. This can be interpreted as users stumbling into wrong content and returning to search for something more suitable. Perhaps the title wasn’t clean enough since descriptions aren’t always provided, or the video content was advanced for user’s skills. Additional insights are gathered after further dataset analysis. When grouping seen elements per user we obtained an average of 5 items. With 80% of users have seen less than or equal to the average. This might indicate that most users have abandoned their learning path or are unable to locate appropriate content. We only consider videos with over 50% of watch time in users profile. The assumption is that longer watch times indicate a higher user interest and satisfaction. 5.2
Instead of choosing fixed parameters empirically and applying them to all algorithms indiferently, another protocol is
used to provide a fair performance comparison. It uses the irace package [
Manuscript submitted to ACM
Population size at each generation.
Crossover Genetic Operator
Crossover Probability Mutation Genetic Operator
Mutation Probability
Kappa (IBEA) Neighbourhood Selection Probability (MOEAD) Max Number of Replaced Solutions (MOEAD) Neighbor Size (MOEAD)
Random, 0.1 - 1.0 0.1 - 1.0 0.1 - 1.0 10, 50, 100, 200, 500, 1000 10, 50, 100, 200, 500, 1000
Elite configurations are returned based on their average best Hypervolume ( ) metric [
Our first test experiment will focus on the three initial objectives ( , , ) which were proposed initially by the company. The second test experiment will add both and @5 to the other three objectives. This will make it a many-objective optimization task and will test how the parameters and performance adapts. Note that both and @5 requires a portion of user history to validate predictions. Since 80% of users have less than or equal to 5 items in their history we will be using the remaining 20% of users as they have more watched items. Both test experiments are samples of the possible choices that platform managers can select. For example another setting can be focused on relevance by having three diferent objectives ( , and @ ).
Starting with Table 5 which shows the elites configurations provided by irace for each algorithm (NSGA2, NSGA3, SPEA2, MOEA/D, and IBEA) using implicit interactions for three objectives (, , ). Observations indicate that 1 − crossover operator has been chosen over the 2 − crossover operator by the majority of algorithms. This can be explained by the fact that the crossover operator in this test setting does not impact objectives performance, so selecting a simpler operator could be the reason. Only / chose as the mutation operator, while the rest of the algorithms used the random mutation operator. This can be attributed to a variety of factors, including the allowed computing time and length of solutions aside from the number of objectives. When looking at Table 6 for the many-objectives irace runs, most elite configurations have chosen the 2 − crossover operator over 1 − . This confirms our previous assumption, that crossover operators, are chosen depending on their role in improving the objectives. Since the additional objectives in this experiment have an interest in item ordering, a change in elite configuration was anticipated. Similarly, all algorithms selected as a mutation operator, indicating that this operator has superior performance, particularly in complex settings. 5.3
The following experiments are based on the results provided by irace. For each algorithm, 30 independent executions are launched using the same settings as seen in parameter tuning: stopping criteria of one hour, objectives (3 &5 ) and = 10 recommended items). We used these metrics discussed in Section 2 to measure the quality of our solutions: , , and . However, the latter metrics do not guarantee better recommendation results for the end user, for Manuscript submitted to ACM we seen in Section 4, metrics like the click-rate (), watch time and click through rate ( ) will be implemented to measure how users are handling recommended items.
Starting with 3 column found in Table 7, results show that 3 has a maximum score of (0.91). In second place followed by 2 with a score of (0.85) and (0.80) respectfully. The previous findings are compared with the results of 5 Objectives experiments. Since objectives , and are already included, we aggregate their values. This is indicated by 3 column in Table 8. Both 2 (0.85) and 3 (0.84) kept a robust performance with 2 (0.84) outperforming its previous score. didn’t perform well compared to the 3 experiment. Shifting our focus to 5 Objectives results shown in 5 column. 2 and 3 achieved good scores of (0.81) and (0.79) respectively.
Taking into account other performance indicators (, , ) which must be minimized and starting with column shown in Table 7. Most algorithms obtained similar scores with achieving the lowest score (1.22). However, looking at column, 2 was able to obtain a value of (1.002) and creating a gap with the rest of algorithms. Note however that both and are easier metrics to meet compared to . For which, was able to achieve the lowest score with (0.093) followed by both 3 and 2. Discussing the same metrics for 5 experiments shown in Table 8. 2 obtained the lowest score (1.41) not far from other algorithms. Surprisingly thought, 2 was also able to obtain the lowest score of (1.06) followed by 3. When considering the column, 2 achieved a performance similar to 2 with scores of (0.06) and (0.07) respectfully. Setting aside the 5 results, 2 have shown good results considering the many-objective problem setting. But, 2 and 3 continued to perform marginally better in which make them more fit for our future experiments.
Moving on to discuss the evolution of the hypervolume indicator for each algorithm. We start with 3 performance charts shown in Fig. 3a. Each algorithm , , 3, 2, and has its respective colors (Red, Grey, Blue, Black, Green). Same as our findings in Table 7, 3, , and 2 are in the lead when looking at the end of the graph. 3 was able to maintain its superiority from the beginning, while both and 2 lacked behind in the first third of the experiments. This behavior changes when looking at 5 graph in Fig. 3b. 2 keeps the lead from the start compared to other algorithms. From both experiments, we can see that most algorithms are not improving at the same level as the beginning of the experiments. This indicates a stagnation state and algorithms aren’t likely to improve considerably.
Considering production scenarios and after examining Fig. 3 findings show that around five minutes of computing time, good results are obtained. While indeed most approaches continue to improve after that period of time but it doesn’t justify the additional computing. The company can quickly update the recommendations for users. However, one major drawback of our approach is the fact that modeling solutions as lists of recommendations for each user does not guarantee that our model will always converge in five minutes with the increase of users or items in the catalog. This scalability issue can be improved by clustering users and providing recommendations per groups instead of individual users, this is considered in future work. At last, these initial findings indicate that for both experiments (3 & 5 ), two algorithms 3 and 2 possess a robust performance and are the best fit for this task compared to the rest.
Selecting the best solution can be complicated specially since these objectives are conflicting with each other.
MultiCriteria Decision Making (MCDM) deals with such decision problems. Among many methods we selected Pseudo
Weights (PS) which calculates the normalized distance to the worst solution regarding each objective [
= Í(=f1(f− − ())(/()f)/(f−f−f) ) (6)
The steps are rather simple, first we get nadir (ideal) points from Pareto front. We proceed to calculate the normalized
distance to the worst solution for each objective . Finally we find the closest solution to the normalized distance. Other
methods aside from Pseudo Weights can be used, such as the high trade-of method [
Platform managers are provided a graphical interface to either selected already defined profiles (balanced objectives, high relevance, etc) or tune the weights assigned to each selected objective using a simple slider. The end-user will receive the first solution returned after the MCDM phase. 6
In this article, we solve a many-objective recommendation problem at Mandarine Academy. The approach is applied on e-learning platforms and took advantage of real-world interactions to better understand user behavior and identify key points for user interface/experience improvement.
Studying related works and following company guidelines, we mathematically formulated our objectives into a Multi-Objective Combinatorial Optimization Problem (MOCOP). With five objectives ( , , , , and @5). Our proposed approach focuses on personalization of recommendations by providing each user Manuscript submitted to ACM the items that match their profile and ratings while emphasizing on novelty and diversity. By evaluating diferent evolutionary algorithms using real-world user data we were able to find best performing approaches considering diferent test settings.
Existing users may benefit from models that generates diversified or novel items to further explore the catalog, while new users may receive recommendations created by emphasizing ratings and ranking. The freedom of selecting what goals to prioritize is provided for platform managers in a user-friendly interface. Future graphical improvements insists on the same principles of readability, ease of use and availability of interactions.
The most time-consuming part of our approach is performed entirely ofline, where chosen objectives are trained on user data and served online through Application Programming Interfaces (APIs). One major drawback is the scalability (a) 3 Fig. 3. Evolution of the indicator (Y-axis) for both 3 and 5 over 1 hour of computing time (X-axis) using all algorithms (30 Executions). of the proposed system when the user base and catalog are expanding, training times can be significantly afected. Also, exploring the possibility for users to indicate their objective preferences which will be taken into account when updating the model is considered in future work.
The integration of Mandarine Academy Recommender System is currently underway and will include an administration dashboard specific to each platform owner for managing and monitoring performance.