Learning Analytics Driven ARC-Tutoring for Individual Study Success

Abstract

Students have to face challenges in applying scientific research skills during their internship and thesis writing at the universities. For this purpose, they receive some static web information and in the best cases holistic mentoring support from supervisors. However, they often require additional assistance in getting suggestions, immediate responses to errors, scaffolding, and reminders of their own learning goals. In this doctoral study, the concept of ARC tutoring guided by learning analytics has been realized as a proposition to address the aforementioned need for assistance in study success in higher education. It advocates leveraging learning experience data by employing learning analytics to develop the assessment, recommendation, and conversational agent (ARC) integrated tutoring workbench featuring distinct learner and tutor perspectives. This will enable the student to gain access to performance metrics and semi-automated individualized tutoring support, while tutors can observe group and individual performance, facilitating required interventions.

Keywords

• eol>learning analytics
• tutoring
• assessment
• recommendation
• tutoring agent
• study success1

1. Introduction

In higher education, study success is the graduation from the degree program at the institution level whereas, it is assumed as completion to the attained mastery of the specific learning objectives of the individual student [ 5 ]. This acquired mastery leads to individual learning success. One of the mastery goals at universities around the world is to equip students with research competencies. From the natural and social sciences to engineering and humanities, students are expected to have the ability to engage in scientific inquiry, analyze data, and draw informed conclusions. As a result, the study success of the students depends on the successful completion of their research internship and/or thesis where they need to demonstrate their skills in conducting and reporting scientific research. Different research methodology seminars are offered before they commence their final assignment of their degree program at the undergraduate and postgraduate levels. Despite this, most of the students require individual learning support during their internship report and thesis writing. They are supported by some static web information from the universities on how to conduct and report their research. Moreover, they get mentoring support from their supervisors who may have time and work constraints to holistically support their students. This leads to students' need for more individual learning support during their application phase of scientific research methodology. This need has been addressed as the research problem in this doctoral research. Consequently, this empirical research aims to provide individual learning support to higher education students in conducting and reporting scientific research.

2. Theoretical overview of Tutoring support and Learning analytics (LA)

Mentoring has been recognized as the pedagogical approach to support students' perseverance in academic success. While the broader concept of mentoring nurtures the psychosocial and career growth among the students, the derived concept of tutoring following the didactical strategies of mentoring can narrow down the focus on students’ domain-specific learning needs. The concept of the zone of proximal development (ZPD) from Vygotsky’s social learning theory is the fundamental basis of the proposed concept of tutoring. This indicates the state of the students having the ability to master the knowledge with the needed guidance or support from the tutor to reach the status of mastering it independently. This guidance or gradual release of support is embedded in the instructional design theory of Scaffolding which emerged in the context of ZPD. Accompanied by selfregulated learning and reflective learning strategies, technology-enhanced tutoring can independently support individual students with minimal human-tutor involvement from time to time.

Due to the technology integration and learning data generation in the tutoring environment, learning analytics (LA) is essential to gain actionable insight. In this era of Web 2.0 technology, LA is a potent tool for anticipating and improving student success by utilizing actionable insight from the generated data during the learning process. This practice entails the systematic measurement, collection, analysis, and reporting of data about learners and their contexts [ 6 ]. The objective is to offer meaningful feedback and scaffolding support when necessary for individual study success [ 5 ]. While statistical analysis serves as a common thread unifying all types of LA, the purposes behind these analyses dictate four primary categories. Descriptive LA involves the exploration and summarization of historical patterns of behaviors and performance in online learning environments. Diagnostic LA seeks to pinpoint the root causes of problems or challenges encountered in the learning process, fostering a deeper understanding of areas that may require intervention or improvement. Predictive LA leverages various methods and technologies to model and foresee future learner outcomes, enabling proactive measures to enhance overall educational effectiveness. Lastly, Prescriptive LA involves the generation of recommendations and decision-making based on computational findings derived from algorithmic models, offering actionable insights to guide and optimize the learning experience.

The Five Steps Learning Analytics (LA) Model, introduced by Campbell and Oblinger in 2007, was designed to address academic analytics and improve student retention. Notably employed in the Signals Project at Purdue University, the model involves capturing data, reporting data patterns, predicting outcomes using statistical regression, implementing interventions to improve the learning process, and refining the model based on the obtained results, serving as a baseline for empirical research and application in education. The Learning Analytics Cycle, introduced by Doug Clow [ 1 ], integrates learning theories and the five-step Learning Analytics model by Campbell and Oblinger [ 2 ]. The cycle encompasses four linked stages: beginning with learners in various learning environments, followed by generating and capturing diverse data, processing the data to develop metrics and analytics, and concluding with interventions based on the developed metrics to impact learners and improve learning practices. Clow emphasizes that effective learning analytics projects involve closing the loop with interventions based on the generated metrics. The Reference Model of Learning Analytics is structured around four dimensions—"What?," "Who?," "Why?," and "How?"—with the goal of identifying challenges and research opportunities in Learning Analytics (LA). It addresses data sources, stakeholders, objectives, and methods, emphasizing tailored approaches for effective interventions, ethical considerations, and stakeholder expectations.

2.1. Virtual and Intelligent Tutoring Systems

The state of the art Intelligent and AI-independent virtual tutoring approaches are articulated via synchronous and asynchronous formats in online learning platforms as well as in learning management systems (LMS). The scope of analysis has been delimited to blended, asynchronous, and Intelligent tutoring approaches. These transformative approaches to learning leverage the strengths of technology to create a more inclusive, interactive, and effective learning environment for students of all backgrounds and levels [ 6 ]. Employing a wide array of artificial intelligence (AI) techniques including natural language processing (NLP), machine learning (ML), and expert systems, Intelligent Tutoring Systems (ITS) go beyond conventional tutoring systems. They exhibit a remarkable ability to discern and adapt to diverse learning styles, pacing preferences, and individual inclinations [ 9 ]. Through this adaptive approach, ITS offers targeted instruction along with invaluable feedback and guidance, ensuring a finely tuned and enriching learning journey for every student. ITS encompasses four fundamental components, each playing a crucial role in the learning process. At the core lies the domain model, which encapsulates the breadth of knowledge and skills required to proficiently navigate a specific subject area. The second module is the student model which intricately maps out the unique knowledge, aptitudes, and capabilities of each student. This personalized profile serves as the compass guiding instructional adjustments and feedback delivery, ensuring a tailored learning experience [ 7 ]. The pedagogical model stands as the strategic backbone of the system, orchestrating the methodologies and approaches employed to effectively impart knowledge. It encompasses a spectrum of techniques, from elucidative worked examples to insightful hints and informative feedback loops. This dynamic array of instructional strategies adapts in real-time, aligning with the student's progress and learning pace. The fourth module is the user interface that serves as the gateway for students to engage with the system. Beyond its functional role, it creates an immersive learning environment, facilitating seamless interaction with the material. Additionally, it acts as a conduit for the timely delivery of constructive feedback and personalized instructions, further enhancing the learning journey [ 7 ]. This personalized approach can integrate formative assessments, learning recommendations, and interactive support via tutoring agents [ 7 ] and significantly contributes to enhanced learning outcomes by providing immediate feedback. This timely guidance empowers students to swiftly recognize and rectify any misconceptions or errors in their comprehension, fostering a deeper grasp of the subject matter. Furthermore, these systems equip educators with invaluable insights into their students' learning trajectories and specific needs. Equipped with this information, human tutors are better positioned to customize their teaching methods, ensuring a more effective and targeted educational experience for each student [ 9 ].

While ITSs hold immense promise, they do come with their share of challenges. These systems operate within the confines of predetermined rules and responses, lacking the capability to dynamically adjust to the unique needs and preferences of individual learners. While proficient in providing foundational support, they fall short in delivering personalized, tailored learning experiences that cater to the diverse requirements of each user [ 3 ]. There is a critical requirement for the development of highly effective pedagogical models and instructional strategies capable of not only proficiently imparting knowledge but also seamlessly adapting to the unique learning needs of individual students. These challenges collectively underscore the complexity and depth of considerations involved in the development and utilization of virtual and intelligent tutoring systems [ 7 ]. These have been shown as impactful approaches for supporting study success either in combination or in segregation [ 9 ].

2.2. Research Question

According to Ifenthaler [ 4 ], the comprehensive approach of LA is geared towards near realtime modeling, prediction, and optimization of learning processes, as well as the environments in which learning occurs. The learning environment includes a range of technologies, from learning management systems (LMS) to more sophisticated automated systems like automated feedback-based online assessment systems, and conversational agents like chatbots. The term does not inherently imply the sole use of advanced adaptive AI technologies. This enables the formal face-to-face higher education system to use the advantages of LA by employing the data from LMS.

Overall, the problem is that students require learning support while writing their research reports for seminars, internships, and thesis. The proposed solution is to provide them with individual tutoring support with digital formative assessments, recommendation system, and tutoring agent. The activation of students’ learning initiation, the processing of their learning status and progression, and acknowledging them are done utilizing learning analytics. Accordingly, the research questions are, RQ1: How to design and develop ARC-Tutoring environment by integrating formative assessments, recommendation system, and conversational agent? RQ2: How to integrate learning analytics functionalities (by integrating the tasks of descriptive, diagnostic, and predictive LA) to guide the proposed “ARC-Tutoring” environment? RQ 3: What is the perception of the primary stakeholders on LA-guided ARC-tutoring support concerning study success? 3.1: What is the perception of the students on the ARC-Tutoring environment? 3.2: What is the perception of the teachers on the ARC-Tutoring environment?

Henceforth, the research objective of this empirical research is to design and develop a prototype of an ARC-tutoring workbench with dual perspectives (Student and Tutor) by incorporating descriptive, diagnostic, and predictive LA to facilitate individual learning success.

3. Concept of ARC-Tutoring model

Technology-enhanced tutoring can actively utilize LA within a specific learning context to discern tutoring needs among students, whether acknowledged or unrecognized by the students themselves. In realizing the research objective, this thesis introduces the Model of ARC-Tutoring facilitated with LA. LA is the main driving force in this model because, at the same time, it is the input and main source of representation of students’ learning intervention, optimization, and progression [ 8 ].

A conceptual framework is designed from the inspiration of Campbell and Oblinger's five-step LA model [ 2 ], with an emphasis on Clow's coherent cyclic process [ 1 ]. In the mentioned ARC-Tutoring model, A stands for formative assessments, R stands for recommendation and reminders, and C stands for Conversational agents (Chatbot). Formative assessments are used for learning to identify the learning gaps and with feedback to optimize it. Recommendations are the suggestions and advice that the students seek to act for their learning progression. Reminders are repeated cues for time management to cope with the recommendations and learning goals. Last but not least the conversational agent is the Tutoring avatar that initiates reflection through its’ question-based interaction, providing self-regulatory level feedback on the student's response and supporting by delivering organizational information.

Capture datafrom the tutoring environment through regression analysis, acting through ARC-tutoring support, refining the learning process based on tutoring input, and connecting to the initial phase for subsequent LA iterations shown in figure 1. The central actionable tutoring phase, informed by investigations from prior phases, implements interventions like synchronous and self-study support, formative e-assessment facilities, and consultations facilitated by recommender systems and virtual tutoring agents. Incorporating these tutoring strategies, the motivation of this doctoral thesis is to develop a tutoring environment embedded with LA visualization.

4. Status of implementation and plan of evaluation

The final phase of this doctoral research is currently underway, focusing on the implementation of the technological components necessary to realize the learning analytics (LA) tasks integrated with the ARC-Tutoring model in a testbed-specific scenario. This phase is crucial for assessing the practical application and effectiveness of the proposed learning environment.

The plan of empirical research for this implementation employs a convergent parallel mixed method design, as outlined by Creswell and Plano Clark [ 10 ]. This approach involves the simultaneous collection and analysis of both qualitative and quantitative data to provide a comprehensive understanding of this learning environment for writing scientific reports for academic purposes.

Data collection will include qualitative data from five expert interviews and quantitative data from an online survey administered to targeted students working with scientific research seminars, internships, and thesis. The instruments for data collection consist of a questionnaire featuring Likert scale items to measure technical usability, learning support, and overall satisfaction, along with a semi-structured interview protocol to guide expert interviews. This mixed-method approach is planned to conduct the acceptance and applicability investigation of the ARC-Tutoring model, capturing diverse perspectives and detailed insights into its usability and learning support following the principles of SRL.

5. Conclusion

This doctoral study systematically addressed the prevalent challenges faced by higher education students in applying scientific research skills during their internships and thesis writing. Recognizing the inadequacies of static web information and often limited tutoring support due to supervisors' constraints, this research introduced the ARC tutoring model, guided by learning analytics, as a robust intervention to enhance study success. The ARC tutoring model integrates assessment, recommendation, and conversational agent (ARC) features, providing students with access to their own performance metrics and individualized tutoring support while enabling tutors to monitor group and individual performance for timely and targeted interventions.

Study success in higher education is a multifaceted construct. At the institutional level, it is typically defined by graduation rates, whereas at the individual level, it is characterized by the mastery of specific learning objectives, particularly in scientific research competencies. Mastery in scientific inquiry, data analysis, and informed conclusion-drawing is critical across disciplines, from natural and social sciences to engineering and humanities. The successful completion of research internships and theses is essential for demonstrating these competencies. Despite the availability of research methodology seminars, many students require additional individualized learning support during the practical application phase of scientific research.

The final phase of this research involves integrating the technological components to realize the learning analytics tasks within the ARC-Tutoring model in a testbed-specific scenario. The evaluation plan includes expert interviews and an online student perception survey to assess the developed tutoring environment’s technical usability, learning support, and overall satisfaction. This comprehensive approach aims to substantiate the efficacy of the ARC tutoring model in enhancing individualized learning support and overall study success in higher education.

References

[1] D. Clow , The learning analytics cycle: closing the loop effectively , in: Proceedings of the 2nd International Conference on Learning Analytics and Knowledge , LAK '12 , ACM Press, New York, NY, ( 2012 ), pp. 134 - 138 . doi: 10 .1145/2330601.2330636.

[2] J. P. Campbell , P. B. DeBlois , D. G. Oblinger, Academic analytics: A new tool for a new era , EDUCAUSE Review 42 (4) ( 2007 ) 40 .

[3] M. A. Hasan , N. F. M. Noor , S. S. B. A. Rahman , M. M. Rahman , The transition from intelligent to affective tutoring system: a review and open issues , IEEE Access 8 ( 2020 ) 204612 - 204638 . doi: 10 .1109/ACCESS. 2020 . 3035963 .

[4] D. Ifenthaler , Learning analytics , in: J. M. Spector (Ed.), The SAGE Encyclopedia of Educational Technology , Vol. 2 , Sage , Thousand Oaks, CA, ( 2015 ), pp. 447 - 451 .

[5] D. Ifenthaler , J. Y. K. Yau , Utilising learning analytics to support study success in higher education: a systematic review , Educational Technology Research and Development 68 ( 2020 ) 1961 - 1990 . doi: 10 .1007/s11423-020-09788-z.

[6] P. Long , G. Siemens, What is learning analytics , in: Proceedings of the 1st International Conference on Learning Analytics and Knowledge , LAK '11 , ACM Press, New York, NY, 2011 .

[7] E. Mousavinasab , N. Zarifsanaiey , S. R. Niakan Kalhori , M. Rakhshan , L. Keikha , M. Ghazi Saeedi, Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods , Interactive Learning Environments 29 ( 1 ) ( 2021 ) 142 - 163 . doi: 10 .1080/10494820. 2018 . 1558257 .

[8] B. A. Motz , Y. Bergner , C. A. Brooks , A. Gladden , G. Gray, C. Lang, ... J. D. Quick , A LAK of Direction Misalignment Between the Goals of Learning Analytics and its Research Scholarship, 2023 .

[9] N. Singh , V. K. Gunjan , A. K. Mishra , R. K. Mishra , N. Nawaz, SeisTutor: a custom-tailored intelligent tutoring system and sustainable education , Sustainability 14 ( 7 ) ( 2022 ) 4167 . doi: 10 .3390/su14074167.

[10] Creswell , J. W. , V. L. Plano Clark , Designing and Conducting Mixed Methods Research , 2nd ed., Sage , Thousand Oaks, CA, 2011 .