Personalising e-Learning Systems: Lessons learned from a vocational education case study Lie Ming Tang and Kalina Yacef Computer Human Adapted Interation Reserach Group School of Information Technology University Of Sydney http://chai.it.usyd.edu.au ltan8012@uni.sydney.edu.au Abstract. Vocational education refers to the training of specific skills or trades. It is often done part time or in personal time over a lengthy period (months to years). As such, it requires persistence, self motivation and self regulatory skills including goal setting, planning and time man- agement. A growing body of evidence suggests that these self-regulating skills are a key determinant in learning performance and can be im- proved with support. We report in this paper our experience with a leading vocational education provider in Australia who is transitioning from classroom-based training to a pilot e-learning system. We present the key lessons learned and the prototype interface we designed to im- prove user self-regulation in planning and time management. Keywords: Vocational Education, Personalization, Self-Regulated Learn- ing, e-learning 1 Introduction Vocational education, which refers to the training of specific skills or trades, dif- fers from academic education as the focus lies in skills and knowledge for specific industries or job roles and often requires proof of practical competency to com- plete. However, similarly to academic learning, self-regulated learning [11] and the ability to direct one’s own learning is seen as an essential part of success in vocational education [5, 9]. Important self-regulation skills include setting appro- priate proximal goals [8], resource management (e.g., allocation of resource, time management, managing ones learning environment), self-monitoring and adjust- ing behaviour based on performance. There is a growing body of evidence that suggests personalised and computer aided support for self-regulated learning im- proves student engagement and performance in e-learning systems [2]. However, such studies has generally focused on academic learning. Self-regulation skills are very relevant in vocational education and help in making students aware of their own role in education and developing their learning are key components [5]. It is important to understand the key challenges for e-learning systems aimed at vocational education and how their personalisation and scaffolding features can support self-regulation. We performed a study with an Australian leading vocational education provider who transitioning from a classroom focused train- ing model to an online and self-managed training model using a pilot e-learning system. We analysed student usage patterns over a 6 month period to under- stand how this e-learning system is used and interviewed trainers and trainees to get qualitative feedback on how students managed their learning in the system. In this paper, we present the key lessons learned from our study and a pro- totype with user interface designed to demonstrate the key features for future e-learning systems to personalise and support for self-regulation of learning. 2 Vocational Education & Training Vocational education and training represents a critical sector of education where skills for a particular industry, trade and career are created. Over 11 percent of the Australian population between 15 and 64 undertake vocational education and training and the age spread is wide [7]. Moreover, the percentage of adults with professional education as the highest qualification is also very high (be- tween 5 and 34 percent [1]). Learning topics are broad and include industry accreditation and certification for fields in health care, accounting, engineering, law and information technology and many others. Vocational education is usu- ally competence based: where training and assessments are assessed on whether a participant is skilled and competent in a particular job or trade rather than measuring theoretical knowledge alone. This may involve collecting evidence and reports over many weeks or months as part of the assessment. For example, flight training requires that trainee pilots log the number of hours of actual and sim- ulated flying. Similarly to academic learning, vocational education can span a lengthy period of time from months to years. Both require students to maintain self-motivation and persistence. However, over 88 percent of vocational learners are part time in Australia. This suggests a higher competing presence of other priorities for most vocational learners compared to academic students, with the unavoidable challenges in attention focusing, organisation and time management skills. 3 Case Study: e-learning system We worked with a leading vocational education provider in Australia who has operated a nationally accredited certification program for their employees for over 10 years. Recently, they started transitioning from a classroom training model to a pilot e-learning training program requiring self directed learning and assessments. A key motivating factor is motivate and encourage students to regulate their own learning and to reduce the contacts needed with trainers. The pilot e-learning system is used for accessing online learning materials, perform assessments and uploading evidence for practical experience and com- petence. In the pilot program, participants first read or view online learning ma- terial, gain practical experience and reinforce their learning and then physically attend a classroom learning workshop. They then complete online assessments via the e-learning system. While the online learning material is not mandatory, they contain the knowledge needed to pass the online assessments. The workshop offers an opportunity for trainers to reinforce the online learning material, pro- vide discussion and simulated practical experience to prepare learners for their formal online assessment. We performed a 6 month study to analyse student usage patterns based on data from the e-learning system for over 600 trainees. We also interviewed 3 participants and 3 trainers to gain an qualitative view of the challenges with self-regulation and performance. 3.1 Planning & Time Management Planning and time management is a significant challenge for many students. A commonly cited problem by students, including the trainees in our study, is the lack of time. However, interviews with trainers indicate that this is not the case because trainees are allocated time or are getting paid for their time spent on learning. Rather, the key issues cited by trainers are attention focusing, planning, and time management rather than time constraints. In many cases, work priorities conflict with the planned learning times and students do not adjust their planning or they forget. While the majority of trainees complete their learning on their own, trainers needed to organise separate workshops specifically for certain groups of trainees to concentrate and complete their online learning away from their workplace which can be busy and not conducive for learning. Trainers found that the trainees’ lack of self-regulation skills in time management, planning, prioriti- sation, and remembering to perform tasks were key challenges. The difficulty level of the learning material was rarely an issue in this program. 3.2 Environment management According to the social cognitive theory, the social and work environment are a key determinant of behaviour [3]. As part of the program, most trainees are paired with a coach who helps and supports their learning progress. Feedback from both trainers and trainees were very positive in terms of the support pro- vided. Trainer feedback suggests that when coaching support is not very strong, the trainee is less motivated and requires more trainer engagement. Trainees also highlighted that pairing with a study partner provided mutual support and improved their motivation and persistence. Feedback from both trainer and trainee suggests that those who managed their social and learning environment well had little trouble completing the course. For instance, one successful trainee, who managed his environment by performing his study during his day off, had a study partner in the program. He also ensured that he completed the planned task on schedule through either performing them on time or adjusting his plan. This trainee was able to complete the learning tasks well ahead of schedule. 4 Our Approach Our hypothesis is that trainees who are struggling in the vocational program can be supported through scaffolding or computer aided support to improve their planning and time management. We built a prototype system, augmenting the existing e-learning platform, with user interface elements designed to promote self regulation. The prototype system lets trainees set time schedules for their learning objectives, monitor their performance and adjust them when necessary. It is also available via mobile application and sends them reminders of upcoming tasks. The prototype system also allows trainers to monitor the performance of their trainees and identify those who need personalised attention. 4.1 Planning & Time Management To address the feedback above, we made planning and time management a core skill for the prototype system to support and scaffold. There have been very few user interfaces designed to scaffold time management and planning. A previous approaches used Zimmerman’s cyclic model of self-regulated learning as the basis to detect and model the learner states [10]. This approach used a calendar-like interface where users define their learning schedules with recommendations and help support. However, we found that the trainees of our vocational program can benefit from an initial engagement to setup a simple schedule and keeping track of their learning task and maintaining their plans. We targeted the more general skill of scheduling of tasks and following through with that plan through moni- toring and reminders. In our interface, when users first login to the application, they are first presented with a wizard where they are prompted to set a plan or schedule for when they expect to complete a task, as shown in figure 1. The user can also add this task to their Google calendar and activate an email or SMS notification when their planned task is due. The wizard does this for the first task only and users can access the wizard later if needed. After the wizard exits, they can set schedule for tasks via the plan button (see figure 2). The objective of this wizard is to scaffold users taking control and managing their plan and time. 4.2 Self Monitoring As part of the interface, the user is allowed to monitor his/her progress in the program. They are also reminded about their upcoming planned tasks, what they have completed and what they still have pending, see figure 3. This allows them to monitor their progress, reflect on their planning and scheduling. In addition, users can monitor their progress compared to their peers for each of the learning objectives in their program, see figure 4. Studies have shown that behaviour can be modified through comparing one’s own performance against peers [6], [4]. Fig. 1. Wizard to help the user get started with planning their learning schedule. Note: a schedule is referred to as a ”goal” in the interface Fig. 2. Add learning schedule to Google calendar and set reminder. Fig. 3. Learning progress of each learning objective: 1) planned and started but not completed, 2) started but no plan, 3) completed Fig. 4. Monitor learning progress versus other students. 4.3 Monitoring Trainees We have also provided trainers with the ability to monitor the progress of each of their participants and the status of each e-learning modules i.e., not started, started or completed. This allow trainers to see which students are lagging behind their peers and require personalised attention. See figure 5. Trainers can also see which students have not accessed their e-learning materials so they can send them reminders to maximise the workshop outcomes. Fig. 5. Trainers see student progress and activity. Further details can be obtained by clicking on the interested status bar drill down (highlighted by red circle). 5 Conclusion & Future Work We found a key challenge for vocational education students is time and envi- ronment management. We designed a prototype system to support students in becoming better planners and time managers. We believe such goal setting and time management interface designs can also be integrated into other e-learning systems where the learning profile is similar to vocational education (e.g., self redirected professional learning, part time academic studies). Peer and trainer engagement and support appears to be important to trainees and trainers and future systems should investigate how scaffolding can be applied. This can poten- tially reduce withdrawals, increase engagement and motivation for the trainees. 6 Acknowledgement This work was funded by Smart Services Cooperative Research Centre. References 1. OECD 2014. Skills beyond School: Synthesis Report, OECD Reviews of Vocational Education and Trainning. 2014. 2. Roger Azevedo and Allyson F. Hadwin. Scaffolding Self-regulated Learning and Metacognition Implications for the Design of Computer-based Scaffolds. Instruc- tional Science, 33(5-6):367–379, November 2005. 3. Albert Bandura. Human agency in social cognitive theory. The American psychol- ogist, 44:1175–1184, 1989. 4. Albert Bandura. Social cognitive theory of self-regulation, 1991. 5. Elly De Bruijn and Yvonne Leeman. Authentic and self-directed learning in vo- cational education: Challenges to vocational educators. Teaching and Teacher Education, 27(4):694–702, 2011. 6. Sten Govaerts, Katrien Verbert, Erik Duval, and Abelardo Pardo. The student activity meter for awareness and self-reflection. In Proceedings of the 2012 ACM annual conference extended abstracts on Human Factors in Computing Systems Extended Abstracts - CHI EA ’12, page 869, New York, New York, USA, May 2012. ACM Press. 7. Kathrin Hoeckel, Simon Field, Troy R Justesen, and Moonhee Kim. Learning for Jobs. OECD Reviews of Vocational Education and Training. Education And Training, (November):220, 2009. 8. Gary P Latham and Edwin a Locke. Self-regulation through goal setting. Or- ganizational Behavior and Human Decision Processes, 50(2):212–247, December 1991. 9. Paul R. Pintrich and Elisabeth V. de Groot. Motivational and self-regulated learn- ing components of classroom academic performance. Journal of Educational Psy- chology, 82(1):33–40, 1990. 10. KP Shih and HC Chen. The Development and Implementation of Scaffolding- Based Self-Regulated Learning System for e/m-Learning. Educational Technology & Society, 13:80–93, 2010. 11. Barry Zimmerman. Becoming Learner: Self-Regulated Overview. Theory into Practice, 41(2), 2002.