Effective Learning Recommendations Powered by AI Engine Xiaodong Dang Ioana Ghergulescu Adaptive Learning Research, Adaptemy Adaptive Learning Research, Adaptemy 27 Lower Mount Street, Dublin 27 Lower Mount Street, Dublin Ireland Ireland xiaodong.dang@adaptemy.com ioana.ghergulescu@adaptemy.com ABSTRACT alternative collaborative filtering approaches, TEL RecSys that consider learning contextual information, TEL RecSys that assess This paper presents the learning effectiveness evaluation of a the educational impact of the recommendations, and TEL RecSys recommender system powered by Adaptemy's AI Engine in terms that focus on recommending courses. As the authors presented in of average lesson success rate and improvement per lesson. The their review, the recommender system and engine could be data from over 80k lessons were used in this analysis. Three main informed by complex information from learner model, domain cases are considered based on the level of teachers’ guidance. The model and personalisation model. Furthermore, other research first case is when the system makes recommendation with no input studies presented the need for the recommender systems to use as from the teacher, the second case is when the system input complex information such as: learning ability, learner needs recommendations are loosely-guided by teacher input through [3], affective and motivational state [4], ontologies about the assignment in a topic, and the third case is when the lessons are learner and the content [5], learner context and domain [6], or to done on concepts that are specified by teachers while the system- use complex engines such as neuronal networks or Bayesian given recommendation is ignored. In each case the results are networks [7]. However, there are few solutions for recommender compared between the lessons done on system-recommended systems in TEL that make use of complex information and that concepts and the lessons done on other concepts. The results have were implemented in a system used in a real- world scenario. shown that both the learning success-rate and the improvement per This paper introduces the recommender system of the lesson are higher if the system-based recommendations are Adaptemy platform that is powered by an Artificial Intelligence followed, in all the three cases. Engine. The Adaptemy platform performs several layers of adaptation and personalisation for students such as: personalised CCS CONCEPTS feedback, personalised content sequence, tailored interventions •Applied computing → Education → Interactive learning when disengagement and demotivation was detected as well as environments; •Information systems → Information retrieval learning paths recommendations. The recommender system makes → Retrieval tasks and goals → Recommender systems use of complex information involving the user, content, domain and context. The recommender system recommends to learners what is KEYWORDS the next most suitable concept to study, the recommended action Recommender systems, AI engine, learning experience, together with personalized guidance and evaluation. The system technology-enhanced learning, learning effectiveness gives flexibility to teachers in terms of how they want to use it: system-independent recommendation with no input from them, system recommendations loosely-guided by teacher, teacher direct 1 INTRODUCTION recommendations to the class where they overwrite the system’s With the amount of learning material available on the Internet, recommendations. there is bigger uptake of recommender systems in Technology- Following a layered evaluation approach as suggested by the Enhanced Learning [1], and a bigger need for effective literature [8], this paper evaluates the recommendation layer with recommender systems and their evaluation through real-life testing its recommender system powered by the Adaptemy’s AI engine [2]. In order to support learning, recommender systems for TEL when the system was used in real-life contexts and at large scale. need to consider specific learning aspects which differ from The evaluation is focused on the educational impact of the recommender systems from other domains such as e-commerce [2]. recommendations to the learning effectiveness. Success-rate and In their review, Drachsler et al. [1] reviewed and classified the improvement per lesson [1] are the two metrics that are used in the recommender systems in TEL in terms of relevant contributions to learning effectiveness evaluation depending if students are the field, categorising them in 7 clusters: TEL RecSys that follow following or not following the system recommendations. The collaborative filtering approaches as in other domains, TEL RecSys evaluation study made use of data corresponding to 4257 students that propose improvements to collaborative filtering by taking into from secondary schools and 80266 learning lessons finished account specifics of the TEL domain, TEL RecSys that take into between September 2017 and March 2018, covering 211 unique consideration educational constraints, TEL RecSys that explore the concepts of a Maths course. © 2018 Copyright held by the owner/author(s). Effective Learning Recommendations powered by AI Engine PALE 2018 at AIED, London The paper is structured as follows: section 2 provides an with track information about previous work and behaviour. The AI overview of the Adaptemy system, section 3 presents the evaluation Engine is responsible for updating the 3 models (user, content and methodology, section 4 analyses the results and discusses the curriculum), and for performing the adaptivity across various layers research findings, while section 5 concludes the paper. such as: content difficulty adjustment, learning loop, motivation detection, learning path recommendations. 2 ADAPTEMY SYSTEM In a previous study [9], the feasibility of integrating adaptive learning powered by the Adaptemy system in the classroom was 2.1 Overview analysed with 62 schools and 2691 students. The results showed Adaptemy system is an intelligent personalized learning that 97% of teachers believe that students enjoy using the environment that is developed based on existing research in the Adaptemy system and want to use it at least once per week. A areas of Intelligent Tutoring Systems and Adaptive E-Learning [9]. further study with over 10,000 students using the system for more It follows the classical architecture of an adaptive and intelligent e- than 6 months in over 1,700 K12 math classroom sessions was learning system that makes a separation between curriculum model, carried out to analyse Adaptemy system’s learning effectiveness. content model, user model, and adaptation engine. The students’ math ability improved by 8.3% on average per The curriculum model includes concepts and the relationships concept for an average of 5 minutes and there was a statistical between them (see Fig. 1 for an example a topic map). The list of significant improvement across various ability ranges [11]. prerequisite links allows for misconception detections, complex Moreover, a 25% problem solving speed increase was observed for user model and enables multiple layers of personalization and the first revision, and 38% increase for the second revision [11]. adaptation. The content model contains all the metadata about the content and up to date analytics. The rich information from the 2.2 Learning Path Recommendations content model enables the AI engine to personalize the learning In the Adaptemy system, each learner receives recommendations loop and to accurately update the user model. For example, each by default. To promote student’s autonomy, the system also allows question has attached information such as: difficulty level, learners to reject the given recommendations and to select discriminant, probability of guessing, probability of having a slip themselves the concept to work with. The system’s and expected time to solve the question. recommendation has 3 parts: a specific concept to study, a specific action (i.e., learn, attempt, revise, practice), and tailored encouragements. The aim is to both present the students with specific goals and to prepare them for the given sessions. The centre of the recommendation is the specific concept to work with. The recommendation is given based on the updated ability profile of the student at the time before a lesson is started. The engine considers which concepts have been worked on, the student’s ability profile in each of the worked and unworked concepts, as well as the positions of the concepts in the knowledge graph of the course map. The engine aims at maximising the learning gain in the student’s next lesson and getting the student better prepared for more advanced concepts, while not making the work too demotivational. There are 3 types of recommendation strategies available for students and directed by teachers. The first case is when the teacher does not provide input to the recommender system and the Figure 1: Example of topic map for maths curriculum. recommendation is done by the system judgment (see Fig 2 A). The second case is when the teacher provides loose input to the Students learn via Adaptemy by doing lessons, each of which is recommender system through a form of assignment by providing on a single concept consisting of a group of questions. For each as input the topic where students should work and the number of student, an ability profile on all the concepts in the curriculum is concepts they should work. The third case is when the teacher maintained in the system, which is updated by the lesson outcomes. overwrites the system recommendations and provides the learners On each concept, the ability profile is represented by a vector of in the class with specific concepts though assignments. 100 elements giving the probability densities of the ability level In the first two cases, the Adaptemy system makes use of a being from 1 to 100. As they finish a lesson, the ability profile of hybrid knowledge-based recommender algorithm. The algorithm the concept that is worked on during the lesson is updated based on makes use of information from the learner model and curriculum the direct evidence using a customised Item Response Theory model. Information from the leaner model includes learner ability (IRT) model. The profile on the other concepts is also updated and previous learning experiences with each concept, as well as based on the lesson outcome as indirect evidence through Bayesian learner motivation index. The algorithm makes use of the Networks update [10]. Overnight, student forgetting is modelled, curriculum model to identify concepts that are misconceptions for and the profile is updated. Additionally, the user model is enhanced 2 Effective Learning Recommendations powered by AI Engine PALE 2018 at AIED, London students and concepts that would have a high prerequisite in a lesson. The lesson is labelled as success if the estimated ability activation in their memory. after lesson on the worked concept is higher or equal than 60 on a 1 to 100 scale. The improvement per concept in a lesson is defined A as the difference between estimated ability at the end of the lesson and estimated ability at the beginning of the lesson. The lessons done by students correspond to one of the 3 cases depending on whether each lesson is involved in an assignment given by the teacher, and whether the assignment is made with specified concepts: 1) Lessons with no assigned concept or topic by the teacher 2) Lessons in an assigned topic by the teacher 3) Lessons on specific concepts assigned by the teacher For all three cases, each time when a student is doing a lesson, the Adaptemy system records a log of the lesson details as well as what was the recommended concept by the AI engine right before B the lesson. However, the system-recommended concepts may be of different indications in the three cases. In the first and second cases, the students have their autonomy in choosing to follow the recommendations given by the system or not. This enables to compare the effectiveness of the recommendations when students are following or not the system’s recommendations. In the third case, the students are in fact following the recommendations given by the teacher. The system-recommended concept in such a lesson means only what would have been the system recommendation at that time given the student’s ability profile updated by the adaptive engine right before the lesson. This enabled to study how students are doing when the teacher recommendations matched the system’s recommendation in comparison with when the teacher’s recommendations did not match the system’s recommendation. Based on the course map used in the Adaptemy system and the prerequisite links, each concept is given a rank position depending Figure 2: A) Screenshot of an example recommendation when on its position in the map representing its level from the very basic there is no input from the teacher (case 1), and B) when the concepts to the most advanced concepts. The difference in this rank teacher assigned a specific concept and overwrote the system position between two concepts can be used to represent if one recommendation (case 3). concept is more basic or more advanced (easier or harder) than the other. In this study, the rank position difference between the The algorithm’s strategy is to reduce misconceptions, increase worked concept and system-recommended concept is used to coverage and to increase engagement by keeping students in flow. represent whether the student is doing an easier, harder or equal- In the first case, the system takes all concepts into consideration, level concept than the recommended concept. while in the second case, the system will filter and use only the concepts from the recommended topic. When the teacher overwrites the system recommendations, the student will receive 4 RESULTS AND DISCUSSION the concept recommended by the teacher and the system This section presents the results analysis structured into several recommendation will be logged for offline analyses (see Fig. 2 B). cases based on the type of the learning activity (i.e., study vs. assignment), and recommendation (i.e., adaptive system vs. teacher 3 METHODOLOGY recommendations). This section details the methodology of the data processing and analysis study conducted to evaluate the recommendation 4.1 Case 1: Study Activity, no Teacher component of the Adaptemy system. The data used in the study Recommendation corresponded to 4257 students and 80266 learning lessons. The This first case corresponds to student activity with no teacher lessons were finished between September 2017 and March 2018, recommendations given through assignments. In this case, the covering 211 unique concepts in the Maths course. system will recommend to students the concept to work with Two main metrics were used to evaluate learning effectiveness through a lesson. Students have the choice to follow the system of the recommendations: percentage of students successfully recommendations or to choose another concept to work through the finishing the lessons and average improvement per concept studied lesson. The lessons were divided in 2 categories: lessons where the 3 Effective Learning Recommendations powered by AI Engine PALE 2018 at AIED, London students followed the system recommendation and lessons where 4.2 Case 2: Assignment Activity, Teacher the students did not follow the system’s recommendations and they Recommends Overall Topic chose themselves other concepts. This case corresponds to 28416 The second case corresponds to student activity when the teacher lessons (involving 3597 students), out of which 16964 lessons gives students an assignment and recommends the overall topic of (59.70%) followed the system recommendations. activity and the numbers of concepts to be worked. In this case, the As shown in Fig. 3, the success rate when following the system will recommend to students one at a time a concept to work recommendation was 78.77% and the success rate when not with through a lesson until they successfully finished the number following was 57.41%. This difference is statistically significant of concepts recommended by the teacher. Students have the choice based on Chi-squared = 1488.9, p < 10e-16. The overall success to follow the system recommendations or to choose another rate was 70.16% in student activity with no teacher concept to work within the topic specified for the assignment. The recommendations given through assignment. lessons were divided in 2 categories: lessons where the students followed the system recommendation, and lessons where the students did not follow the recommendation and they chose themselves other concepts. All concepts (recommended by the system or not) will contribute to the assignment. This case corresponds to 7756 lessons (involving 1138 students), out of which 4956 lessons (63.90%) followed the system recommendations. The success rate when following the recommendation was 71.19% and the success rate when not following was 37.50% (see Fig. 5). Overall success rate in assignments with recommended topic was 59.03%. The difference is statistically significant with Chi-squared = 838.08, p < 10e-16. Figure 3: Percentage of students succeeding the lessons based on them following or not the system recommendations. Figure 5: Percentage of students succeeding in lessons based on them following or not the system recommendations when the teacher recommended the overall topic. Figure 4: Improvement in student ability based on them The average improvement per lesson when following the following or not the system recommendations. recommendation was 17.96 ability points and the average improvement per lesson when not following was 2.08 ability points The average improvement per lesson when following the (see Fig. 6). The difference was statistically significant with t = recommendation was 21.65 ability points and the average 28.422, p < 10e-16. The overall average improvement in the second improvement per lesson when not following was 8.14 ability points case was 12.23. (see Fig. 4). The difference was statistically significant, with t = 48.537, p < 10e-16. The overall average improvement for student activity with no teacher recommendations given to students through assignment in the system is 16.21. 4 Effective Learning Recommendations powered by AI Engine PALE 2018 at AIED, London This difference is statistically significant with Chi-squred = 3273.4 and p < 10e-16. The overall success rate is 58.72% in this case. As shown in Fig. 8, when the concept specified by the teacher matches the recommended concept by the system, the average improvement is 21.08 ability points, and when they do not match the average improvement is 6.13 points. The difference is statistically significant with t = 63.701 and p < 10e-16. The average improvement per lesson for this case is 11.83. Figure 6: Improvement in student ability based on them following or not the system recommendations when the teacher recommended the overall topic. 4.3 Case 3: Assignment Activity, Teacher Recommends Specific Concepts The third case corresponds to student activity when the teacher gives students an assignment with specified concept(s) to work on. In this case the worked concept is the one that is recommended by the teacher. The lessons are divided into two categories: first is when the concept worked is the same as the concept recommended Figure 8: Improvement in student ability based on the lessons where the concept specified by the teacher matches or not the by the adaptive learning system, and the other is when the concept concept recommended by the system. worked is different from the system-recommended concept. In total 43417 lessons worked by 2649 students are included in this case, From the results it is shown that in every case of the lessons, the where for 16546 of the lessons (38.11%) the concept specified by learning effectiveness is higher if the worked concept that is also the teacher matches the system recommended concept. the concept specified by the teacher in the 3rd case, matches the concept recommended by the system. 4.4 A closer look into all the three cases To further investigate the effect of concept difficulty levels on the success rate of lessons, a further analysis is done by dividing the lessons not matching the system recommendations into three categories: 1) lessons done on easier concepts, 2) lessons done on same level concepts, and 3) lessons done on more difficult concepts. Fig. 9 and Fig. 10 provide a closer look into the three cases when the system-given recommendation is not followed. The dashed blue lines in the two figures represent the value when the recommendation is followed for easy visual comparison. The results from Fig. 9 show that in all three cases the highest success rate is when the worked concept is easier or at a more basic level than recommended, and the lowest success rate is when the worked concept is harder or at a more advanced level than Figure 7: Percentage of students succeeding in lessons based on recommended. However, even if the worked concept is at the same the lessons where the concept specified by the teacher matches level as the recommended concept, the success rate is still lower or not the concept recommended by the system. than that when the system recommendation is followed. Therefore, the higher success rate when the recommendation is followed is not As shown in Fig. 7, the lesson success rate is 75.95% when the only due to the difficulty levels of the concepts chosen by the teacher assigned concept matches the system recommended recommendation. concept, while the success rate is 48.11% in non-matching lessons. 5 Effective Learning Recommendations powered by AI Engine PALE 2018 at AIED, London reinforce their abilities on, than the easier concepts chosen by the students themselves. However, regarding the more advanced concepts worked in the lessons, the teachers’ selection may be more over-challenging than what the students choose by themselves, and thus the improvement is even lower (2.5 compared to 5.54). The results shown in Fig. 10 indicates that choosing the right difficulty levels of concepts to be worked on is part of the reason why working on the concepts recommended by the engine would gain higher improvement per lesson. It provides a strong evidence that the recommendation engine is giving the right difficulty levels of concepts to be worked on. Figure 9: Lesson success rates compared across different cases 5 CONCLUSIONS when the worked concept is easier or harder than, or at the same level as the system-recommended concept. In summary, the study shows that the learning recommendations provided by the Adaptemy’s AI Engine, when followed by the The results from Fig. 10 show that in all three cases, no matter users, will give both a higher success rate and a higher average if the concept is of an easier, same or higher level than the ability improvement than if they are not followed, which shows the recommended concept, the improvement per lesson (represented by effectiveness of the personalised learning path recommendation. the three bars) is still lower than that if the lesson is worked on the Moreover, looking into more details regarding to concept advance system-recommended concept (represented by the dashed line in levels shows that the higher effectiveness of the recommendation blue). This is seen as a corroborative evidence that the is not solely due to the right advance levels, corroborating that the recommendation engine does not only take into account the learning path consideration takes its part of effect in the difficulty levels of concepts, but also the prerequisite relationships recommendation engine. between concepts in the knowledge map. The effectiveness of the recommendation engine regarding the best learning paths is thus REFERENCES supported by the results here. [1] H. Drachsler, K. Verbert, O. C. Santos, and N. Manouselis, “Panorama of recommender systems to support learning,” in Recommender systems In all three cases, the average improvement is the lowest when handbook, Springer, 2015, pp. 421–451. the lesson is on a harder concept than recommended. In Case 1 and [2] M. Erdt, A. Fernández, and C. Rensing, “Evaluating Recommender Systems for Technology Enhanced Learning: A Quantitative Survey,” IEEE Trans. Learn. Case 2, the highest average improvement is seen when the lessons Technol., vol. 8, no. 4, pp. 326–344, Oct. 2015. are done on a same-level concept as recommended. However, in [3] I. Hulpuş, C. Hayes, and M. O. Fradinho, “A Framework for Personalised Case 3, where the lessons are done on concepts assigned by the Learning-Plan Recommendations in Game-Based Learning,” in Recommender Systems for Technology Enhanced Learning, N. Manouselis, H. Drachsler, K. teachers, the highest average improvement is seen when the worked Verbert, and O. C. Santos, Eds. New York, NY: Springer New York, 2014, pp. concept is easier than the system recommended one. 99–122. [4] O. C. Santos, M. Saneiro, S. Salmeron-Majadas, and J. G. Boticario, “A Methodological Approach to Eliciting Affective Educational Recommendations,” 2014, pp. 529–533. [5] J. K. Tarus, Z. Niu, and A. Yousif, “A hybrid knowledge-based recommender system for e-learning based on ontology and sequential pattern mining,” Future Gener. Comput. Syst., vol. 72, pp. 37–48, Jul. 2017. [6] A. Klašnja-Milićević, B. Vesin, M. Ivanović, Z. Budimac, and L. C. Jain, “Recommender Systems in E-Learning Environments,” in E-Learning Systems, Springer, Cham, 2017, pp. 51–75. [7] T. K. Paradarami, N. D. Bastian, and J. L. Wightman, “A hybrid recommender system using artificial neural networks,” Expert Syst. Appl., vol. 83, pp. 300– 313, Oct. 2017. [8] A. Paramythis, S. Weibelzahl, and J. Masthoff, “Layered evaluation of interactive adaptive systems: framework and formative methods,” User Model. User-Adapt. Interact., vol. 20, no. 5, pp. 383–453, Dec. 2010. [9] I. Ghergulescu, C. Flynn, and C. O’Sullivan, “Adaptemy – Building the Next Generation Classroom,” presented at the EdMedia: World Conference on Figure 10: Average improvement per lesson compared across Educational Media and Technology, 2015, vol. 2015, pp. 79–88. different cases when the worked concept is easier or harder [10] T. Lynch and I. Ghergulescu, “Large Scale Evaluation of Learning Flow,” in 2017 IEEE 17th International Conference on Advanced Learning Technologies than, or at the same level as the system-recommended concept. (ICALT), 2017, pp. 62–64. [11] I. Ghergulescu, C. Flynn, and C. O’Sullivan, “Learning Effectiveness of Looking at Case 3 in comparison with Case 1 one can note that Adaptive Learning in Real World Context,” presented at the EdMedia: World Conference on Educational Media and Technology, 2016, vol. 2016, pp. 1385– the improvement values are similar on the lessons where the 1390. worked concepts match the recommendation (21.65 compared to 21.08) or are on the same-level concepts as recommended (13.76 compared to 13.62). The indication is that the teachers may provide a better selection of easier concepts for the students to revise and 6