Improving student inclusion through learning analytics: a Step-Wise approach Dr Hildo Bijl 0 Utrecht University of Applied Sciences , Padualaan 99, Utrecht , The Netherlands

At the Utrecht University of Applied Sciences, students from a non-standard background often have deifciencies in their mathematics/physics/engineering skills. To improve inclusiveness, a practice support app called Step-Wise has been set up that automatically detects these deficiencies and advises students on which skills to practice. Through student interviews using the CIMO-logic, it has been established that this app mainly improves the efectiveness of practice, although it also has a beneficial efect on the amount of exercises practiced and on the student motivation.

 eol>Inclusion Learning analytics Bayesian user modeling Engineering education 1. Introduction

prevent these equal opportunities: if a student does not have the prior knowledge and skills that are expected, education will inevitably be less efective.

A variety of methods have been discussed in literature to combat this. There are tools to support general study planning [ 3 ], but these have not been proven efective. Other tools focus on deciding when teachers need to perform an intervention. [ 4 ] These methods have shown more promise, but in the absence of suficient teacher availability, it would help to have said intervention also be automated. It is hard to accurately judge the efectiveness of such automatic programs, mainly due to the lack of large-scale long-term studies and the dificulty of objective evaluation. [ 5, 6, 7 ] However, initial studies have shown promise, leading to the experiment described in this paper.

When setting up an SLE, there are various important concepts to keep in mind. The main goal is to use diferentiation between students to improve learning outcomes. This diferentiation has shown promise, but also complexities. [ 8, 9, 10 ] Direct feedback has shown positive efects [ 11 ] but so have the principles of Just-In-Time Teaching [ 12 ], High-Impact Learning [ 13 ] and 4C/ID [ 14 ]. In addition, principles of gamification have also shown to contribute to student motivation. [ 15 ] All these ideas have been taken together and implemented in a new SLE: Step-Wise.

This paper is set up as follows. In Section 2 the set-up of the Step-Wise practice platform is discussed, including the didactic background. Section 3 discusses the learning analytics side of the platform, detailing the functioning of the algorithms. Afterwards, Section 4 explains the application of the platform in various courses and the lessons learned from it. This paper is closed of by conclusions and recommendations in Section 5.

 2. The practice system: physics engines and direct feedback

The Step-Wise practice platform was designed with three main goals in mind. 1. Provide students with practice experiences that are as close as possible to what they need to do in their final assessments. 2. Support students during practice, giving feedback on work and guidance/support when they get stuck. 3. Coach students in their general learning process, detecting deficiencies and helping them tackle it.

The least innovative is goal 1. To tackle this, the Step-Wise platform has input fields that allow the easy entering of numbers with units, shown in Figure 1. Behind these input fields is an intelligent physics engine that checks the numbers, ensuring that diferent ways of writing the right answer thing are all considered correct. The input fields also provide customized feedback to the student based on the provided answer, conform goal 2. (Again, see Figure 1.) In addition, the app also supports interactive plots/diagrams that require user interaction. And just like with the regular input fields, these interactive diagrams provide feedback to given solutions as well.

However, the direct feedback on student work is not the only way goal 2 is obtained. The main idea behind Step-Wise is that each exercise consists of steps. If a student understands the exercise, they are immediately allowed to enter the final answer. If they do not, they can request to solve the exercise Step-Wise. In this case, they are presented the steps needed to solve the exercise, one by one, and subsequently get feedback on those steps as well. For some exercises, there are multiple ways to solve the exercise. In that case the student can indicate which method they will apply – see Figure 2 – and the rest of the exercise adapts.

Experience has shown that this Step-Wise approach is already strongly appreciated by students. However, this is not what makes Step-Wise innovative. That is the learning analytics system behind the exercises.

 3. Learning analytics: skill trees and Bayesian modeling

In Step-Wise, each course is dissected into a large number of skills. Typically, every 15-20 minutes of lecture time represents one skill. These skills are then hierarchically linked in a large skill tree. A small example is shown in Figure 3.

The innovative part of Step-Wise is that every step of every exercise is coupled to a skill from the respective skill tree. Whenever a student submits anything, their success or failure at the respective step is immediately noted and registered. Of course a single success does not directly imply mastery. Instead, the system uses a specially designed Bayesian user modeling algorithm to keep track of the chance that a student will perform a given skill correctly the next time. Contrary to Bayesian Knowledge Tracing, this algorithm does not take into account the (virtually non-existent) chance of correct guesses, but instead models how the probability of success changes throughout the learning process. The result is an estimate, for every skill at every point in time, of the future success rate. This idea is visualized in Figure 4, with a thorough discussion of all the mathematics behind the algorithm available in [ 16 ].

Through this Bayesian user modeling, the Step-Wise platform always has up-to-date information on the level of each student at each skill. This allows the platform to counsel the student in which skill to practice next. Whenever a student is practicing a skill, without having mastered all the prerequisites, they are recommended to practice said prerequisites first. Or, if the student is practicing a skill that they have already mastered, they are recommended to practice the ifrst follow-up skill that is not mastered yet. In practice this means that, when a student is practicing an exercise and consistently fails at a certain subskill, they get a pop-up "You are recommended to practice [this subskill] first." Do keep in mind that Step-Wise is meant as a practice support platform: students are always free to ignore advice and practice what they deem to be appropriate for them at the given time.

The Step-Wise platform does not stop at only skill recommendations. Whenever a student practices a skill, Step-Wise also selects the optimal practice exercise. Every skill has a large number of exercises coupled to it. However, some exercises have more steps or more complex steps than others, and as a result are more complicated. Whenever a student wants to practice a certain skill, Step-Wise calculates the chance that the student will do each exercise correctly. It then filters out exercises with a very high estimated success rate (too easy) and with a very low estimated success rate (too dificult). This ensures that a student always receives exercises on their own level.

There are various other ways in which Step-Wise uses the available user data, but it is beyond the scope of this paper to elaborate on that. Further details can be found in the Step-Wise explainer in [ 17 ].

4. Application to education

The Step-Wise platform has so far been applied to three courses.

• Alpha test: a second-year thermodynamics course for 60 full-time students. The goal was to filter out bugs. • Beta test: a second-year part-time thermodynamics course for 20 part-time students. The goal was to improve the user experience. • Main test: a first-year thermodynamics course for 80 full-time students. The goal was to gauge the efectiveness of the system at improving the student’s learning experience.

A numerical analysis of the efectiveness of the platform could not be performed. Due to the limited number of students and due to ethical reasons, it was not possible to split the student body up into a part with access to the app and a part without. In addition, due to the corona crisis, a comparison of this group with last year’s students would be inappropriate.

Instead, the evaluation was done through the CIMO-logic [ 18 ]: in a certain Context, check if a given Intervention activates a Mechanism that results in an adjusted Outcome. Because the app has a variety of functionalities, this analysis is done per feature. What was the mechanism activated by each functionality? And what were the resulting outcomes? To answer these questions, a dozen interviews have been held with volunteering students, focusing on how the app changed their behavior, their understanding and their motivation. • Automatic checking of exercises: Nearly all students noted1, "The app is very strict! One small mistake and the exercise is incorrect." Students indicated that this decreased motivation, but at the same time did force them to be more thorough in their work. Since performing fault-free calculations is an important part of engineering education, this is a useful learning outcome. One student even mentioned, "This is the first university physics course I passed, ever!2 And it’s because the app always pointed out my mistakes." • Automatic feedback: Students indicated that most of the time the automatic feedback gave them some hint on why their answer was wrong. This allowed them to try and find their mistake. It encouraged them to give the exercise another try. • Interactive input fields and diagrams : A small majority of students spontaneously noted the ease-of-use of the app and specifically the method of solution input. They said that other apps they have used in the past (Canvas, MapleTA) do not allow for the intuitive use of units, but this app did. It made practice more like real life. This increased the efectiveness of practice and the motivation to practice. • Step-Wise approach to exercises: The most appreciated feature of the app was the Step-Wise approach. Students mentioned that splitting an exercise up into steps and checking each step one by one provided them with a much-desired structure. And by seeing this consistent solution method behind exercises, students were subsequently motivated to practice more. • Skill tracking: When practicing a skill, students continuously see a small globe in the top right of the app, which fills up when they do well. This skill globe was a continuous source of both desire and frustration to many students. Seeing it fill up encouraged them to practice more and fill it up further, but seeing it empty a bit on every tiny mistake also occasionally resulted in bouts of anger. Nothing conclusive can hence be said about the efect on student’s motivation, but it certainly resulted in an increased amount of practice. • Skill recommendations: Various students indicated that the course overview (a clear image of which course skills they had mastered) helped show them what they still needed to practice, improving the efectiveness of practice. Students also, almost without exception, indicated that they initially followed the recommendations of the app. However, as they grew more accustomed to using the app, they did start to deviate from the recommendations. Especially after the app "Completely destroyed my score after one tiny pointless mistake" students were tempted to continue with the next skill, having decided for themselves they had obtained suficient mastery. • Exercise selection: None of the students were aware of any intelligent exercise selection script that gave them easier exercises at the start and harder exercises as they improved. Students did notice, however, that the app was quite monotonous in providing them with exercises. They often received the same exercise, albeit with diferent randomly generated numbers, twice in a row. (This is an inconvenient by-efect of the exercise selection strategy, which should reduce/disappear once more exercises are added to the 1All quotes mentioned are student quotes. Quotes were originally in Dutch and have been translated. 2The Step-Wise trial was run on the fourth physics course the students had. The student involved came from a labour school, anecdotally showing that the app can be useful for students with an alternative background. Functionality Automatic checking of exercises Automatic feedback Interactive input fields and diagrams Step-Wise approach to exercises Skill tracking Skill recommendations

Exercise selection app.) As a result, student motivation dropped and a few students indicated they practiced slightly less as a result.

If we summarize the outcomes, based on three categories "Amount practiced", "Efectiveness of practice" and "Motivation", then we find Table 1. The results conform to the main student sentiment about the app. "The app definitely made practice more useful, and encouraged me to work harder. It even made it a bit more fun."

 5. Conclusions and recommendations

Overall, it can be concluded that the Step-Wise app helped students practice and improved the learning outcome, especially from students from an alternate background. This is mainly due to an increased quality of practice. It must be noted here that some of these efects could also be obtained without any app. After all, especially the Step-Wise approach of the app was appreciated, and it is also possible to write a PDF solution manual in a similarly structured fashion. Nevertheless, the interactive elements of the app also had demonstrated positive efects, proving that an app can have an added value. Combining the interactive elements and the Step-Wise approach in a single app of course combines the best of both worlds.

The app did have a few interesting downsides. A main philosophy of Step-Wise is that all students will eventually master all skills. However, there are many students in university that only want to loosely master three quarters of the course and then get a bare passing grade. Especially those students struggled with the strict exercise checking algorithms of the app. The app is hence not completely inclusive to students who, as a student said, "just want to wing it." It’s not clear yet if it’s better to adjust Step-Wise to this student culture, or adjust the student culture to the philosophy behind the app.

Another downside of every student constantly getting personalized randomly generated exercises is that it prevents students from working together. This problem is inherent to randomly generated exercises, as was also experienced by [ 19 ]. A follow-up project can focus on expanding the app to include a collaboration mode.

o + o + ++ + −

 Acknowledgments

This project was funded by the Dutch ministry of OCW through Comenius grant 405.20865.254.

 [1] A. K. Chmielewski , The global increase in the socioeconomic achievement gap , 1964 to 2015, American Sociological Review 84 ( 2019 ) 517 - 544 . URL: https://doi.org/10.1177/ 0003122419847165. doi: 10 .1177/0003122419847165. [2] W. Crenna-Jennings , Key drivers of the disadvantage gap: Literature Review , Education in England: Annual Report 2018 , Education Policy Institute, 2018 . [3] T. D. Laet , M. Millecamp , M. Ortiz-Rojas , A. Jimenez , R. Maya , K. Verbert , Adoption and impact of a learning analytics dashboard supporting the advisor - student dialogue in a higher education institute in latin america , British Journal of Educational Technology 51 ( 2020 ) 1002 - 1018 . URL: https://doi.org/10.1111/bjet.12962. doi: 10 .1111/bjet.12962. [4] M. Hlosta , C. Herodotou , V. Bayer , M. Fernandez , Impact of predictive learning analytics on course awarding gap of disadvantaged students in stem , in: I. Roll , D. McNamara , S. Sosnovsky , R. Luckin , V. Dimitrova (Eds.), Artificial Intelligence in Education , Springer International Publishing, Cham, 2021 , pp. 190 - 195 . [5] J. M. Spector , Conceptualizing the emerging field of smart learning environments , Smart Learning Environments 1 ( 2014 ) 1 - 10 . URL: https://doi.org/10.1186/s40561-014-0002-7. doi: 10 .1186/s40561-014-0002-7. [6] C. Herodotou , B. Rienties , M. Hlosta , A. Boroowa , C. Mangafa , Z. Zdrahal , The scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study , The Internet and Higher Education 45 ( 2020 ). URL: https:// www.sciencedirect.com/science/article/pii/S1096751620300014. doi: 10 .1016/j.iheduc. 2020 . 100725 . [7] Z. Papamitsiou , A. Economides , Learning Analytics for Smart Learning Environments: A Meta-Analysis of Empirical Research Results from 2009 to 2015 , 2016 , pp. 1 - 23 . doi: 10 . 1007/978-3- 319 -17727-4_ 15 - 1 . [8] K. Scalise , Diferentiated e-learning: Five approaches through instructional technology , Int. J. Learn. Technol . 3 ( 2007 ) 169 - 182 . [9] J. Laskaris , 5 tips for applying diferentiated instruction in eLearning, 2015 . URL: https: //www.talentlms.com/blog/5 -tips-for-applying-diferentiated-instruction-in-elearning/. [10] C. Pappas , Diferentiated instruction in elearning: What elearning professionals should know , 2015 . URL: https://elearningindustry.com / diferentiated-instruction-in-elearning-what-elearning-professionals-should-know. [11] J. Laskaris , The elearning feedback power: Personal, specific , and timely, 2016 . URL: https://www.talentlms.com/blog/elearning-feedback-power/. [12] G. Novak , E. Patterson , A. Gavrin , W. Christian, Just-in-Time Teaching: Blending active learning and web technology , Prentice Hall, Saddle River, NJ, 1999 . [13] F. Dochy , I. Berghmans , A. Koenen , M. Segers , Bouwstenen voor High Impact Learning, Boom uitgevers , Utrecht, 2015 . [14] P. A. K. Jeroen J. G. van Merriënboer , 4C/ID in the Context of Instructional Design and the Learning Sciences, Routledge , 2018 , pp. 169 - 179 . URL: https://www.routledgehandbooks. com/doi/10.4324/ 9781315617572 - 17 . doi: 10 .4324/ 9781315617572 - 17 . [15] E. McLaughlin , 6 reasons why gamification enhances the learning experience , 2017 . URL: https://elearningindustry.com / gamification-enhances-the-learning-experience-6-reasons-why. [16] H. Bijl , Automatic skill tracking, 2021 . URL: https://github.com/HildoBijl/stepwise/blob/ master/frontend/public/SkillTracking.pdf. [17] H. Bijl , Step-Wise: an interactive practice platform , 2021 . URL: https://step-wise.com/ Explainer-EN.pdf. [18] J. Holmström , T. Tuunanen , J. Kauremaa , Logic for design science research theory accumulation , in: Proceedings of the Annual Hawaii International Conference on System Sciences , 2014 , pp. 3697 - 3706 . doi: 10 .1109/HICSS. 2014 . 460 . [19] M. Meijers , P. Verkoeijen , The relationship between ICT based formative assessment and academic achievement in a Mechanics of Materials course , in: Proceedings of the SEFI 47th Annual Conference , 2019 , pp. 1753 - 1762 . URL: https://www.sefi.be/wp-content/uploads/ 2019/10/SEFI2019_Proceedings.pdf.