Self-regulated learning (SRL) plays an important role in academic success. However, many students struggle to effectively self-regulate their learning and they need support to improve their SRL as well as their learning outcomes. Research shows that SRL supports are generally effective but often do not benefit the students who need them the most. One reason is that the support is rarely personalized to their individual needs. With the advancement of technology and, more recently, the proliferation of generative AI-powered technologies (e.g., chatbots and large language models), there is a potential to better meet students' needs, and at the same time, a greater call to examine ways to personalize SRL support using AI. In this workshop presentation, we introduce two work-in-progress empirical studies to explore the use of generative AI chatbots, specifically OpenAI's ChatGPT, as a peer feedback tool and as a study tool to enhance SRL and learning performance in writing and reading, respectively, in the setting of higher education. Preliminary results of the empirical studies will be shared in the workshop. The presentation will contribute to the pressing discussion on opportunities and considerations in using generative AI tools to support SRL.
Advancement in technology has played an important role in shaping higher education [1]. More recently, the proliferation and increased accessibility of generative AI (GenAI) powered technological tools has gained the attention of practitioners and researchers (e.g., [2]). One example is the development of chatbots (e.g., Open Ai's ChatGPT, Microsoft's Bing Chat, Google's Bard) based on large language models (LLMs) that leverage deep learning and advanced algorithms to perform language-related tasks [3]. Applications of such chatbots are wide-ranging in education. UNESCO listed ten possible ways to use ChatGPT for teaching and learning [4], for instance as a 'study buddy' to help students reflect on the learning material or a 'personal tutor' (i.e., AI tutors each student and gives immediate feedback on the learning progress).
Given that GenAI-powered chatbots are a recent development, research is needed to examine their effect on teaching and learning, and ways to implement them effectively to improve students' selfregulatory learning skills and processes, and ultimately their learning outcomes. The open accessibility of such GenAI tools calls for a much-needed understanding of how students can leverage support from them to self-regulate their learning and the support needed to effectively use them during self-regulated learning (SRL). In this workshop, we present two work-in-progress empirical studies conducted at European universities. The preliminary findings of these studies contribute to the discussion on the opportunities of GenAI in supporting SRL, and whether and how GenAI can be used to enable personalized learning experiences and feedback in higher education.
SRL refers to the extent to which students are motivationally, behaviorally, cognitively, and metacognitively engaged in their learning [5]. Self-regulated learners set goals and plans, monitor their progress, and reflect on their learning. Research across all educational levels shows that effective selfregulated learners achieve higher academic success [6]. However, students' ability to self-regulate their learning varies. They typically do not spontaneously engage in SRL processes (e.g., plan, monitor, reflect) and can be relatively poor at it even if they do (e.g., set suboptimal goals and inaccurate monitoring) [7]. The demand for SRL is even greater in learning environments where students are expected to have greater autonomy, such as in higher education and online learning environments. Consequently, there is an SRL paradox: students are poor at SRL but need to be good at it to succeed in their studies.
To help students succeed, there is a need to support and improve students' SRL. Jansen et al. [8] conducted a meta-analysis on 51 SRL-intervention studies in higher education. Results revealed positive medium-sized effects of SRL interventions on SRL activities (d=.50) and academic performance (d=. 49). However, the effect sizes are smaller than those found in primary and secondary education [9]. Moreover, SRL activities only partially mediated the effect of SRL interventions on academic performance, suggesting that other factors, such as task motivation (e.g., self-efficacy), brought about the positive effect of SRL interventions on academic performance. While various SRL support is generally effective, the smaller effect sizes found in higher education suggest a need to tailor SRL support to the characteristics of learners and learning in higher education. One-size-fits-all approaches might be less effective or even distracting if students already have their repertoire of SRL strategies [10,11], and effective SRL support in one context may not generalize to another [12]. Therefore, personalizing SRL is a promising research direction. The proliferation of GenAI tools can potentially help meet students' needs when studying in different contexts [13].
The emergence of GenAI and coupled with the open accessibility of GenAI chatbots like ChatGPT has sparked concerns regarding students' potential misuse of GenAI tools of these tools in education, particularly in assessment [40] and jeopardizing academic integrity [14]. However, the utility of GenAI tools extends beyond mere assessment practices, with diverse applications emerging [15,16]. In a study by Chang et al. [17], it was suggested GenAI chatbots could augment and enhance SRL by fostering self-assessment through reciprocal questioning, prompting students to reflect on their reading process. Another promising avenue involves leveraging GenAI chatbots to deliver personalized feedback on individual's learning process. As the use of GenAI tools becomes more prevalent and influential among students in higher education, there is a pressing need research is needed to understand how students incorporate these tools in their self-regulatory learning practices and identify the crucial skills for students to optimally benefit from such tools. For example, to harness GenAI chatbots effectively for learning, students need to be adept in SRL and have the skills to accurately monitor and evaluate the quality of their learning experiences while engaging with texts [18]. Considering the evolving landscape of GenAI chatbots, it is imperative to conduct further research to explore their potential in higher education, particularly in enhancing SRL [19,20]. Understanding how these tools can be leveraged to support students and enhance their SRL and learning outcomes is vital as we navigate the nascent stage of GenAI-powered chatbots and complexities around human and AI collaboration.
The first study investigates using GenAI-powered chatbots for peer feedback during academic writing. Whereas some scholars have shown that GenAI can help learners when writing essays or creative texts [21], others found the use of ChatGPT does not enhance students' essay-writing performance [22]. Feedback plays a key role in the process of mastering writing in several academic subjects [23]. Previous research shows that ChatGPT's feedback was more detailed and readable than the instructor's feedback and maintained high levels of agreement with the instructor feedback on selected aspects of writing [24]. In this study, the ability of ChatGPT to act as a personal tutor, namely, to provide feedback to students on their produced texts, is compared to the feedback offered by peers. The main research question is What are the students' views on the feedback provided by ChatGPT as compared to human peer feedback on their written texts in the setting of academic writing?
Study design. The study is performed in the setting of an academic writing course offered at a large university in Sweden. All the students (N =70) in the course were asked to 1) write a text (1-page plan written in pairs) that would set the ground for their thesis project that they will carry out over one academic semester, 2) to individually provide structured written feedback on the texts written by their peers (i.e., each student was asked to provide feedback on the two automatically assigned texts), and 3) to prompt ChatGPT to provide feedback on their own written text that was completed in pairs. Thus, each pair of students working on one project received peer feedback from ChatGPT and the two human peers. After receiving the feedback, the students, guided by a teacher, discussed the feedback provided by ChatGPT and their peers in the three two-hours long face-to-face seminars (all the students were divided into 3 groups). Finally, after the seminars, each student was asked to complete a survey, focusing on the ChatGPT's ability to act as a personal tutor. The survey's items were adapted from the seven principles of good feedback practice, introduced by Nicol and Macfarlane-Dirk [41]. Preliminary results show that, overall, students found ChatGPT to be useful to support their SR. However, it was perceived to provide less explainable answers, as well as it was seen to be less "critical" and "analytical" as compared to the human peers' feedback. The study contributes to an improved understanding of the ability of ChatGPT to assist students in the academic writing process in higher education.
Metacognitive monitoring is an important component of SRL [25]. Self-monitoring (i.e., judging one's level of understanding) is required for self-regulation (i.e., deciding what to study). Therefore, to effectively self-regulate one's learning, it is essential to accurately self-monitor [26]. Accurate monitoring helps students differentiate between well-learned and less-learned material; they can better allocate their study time and dedicate more time to the less-learned material [27]. However, it is wellestablished that students' monitoring judgments are highly inaccurate [28]. The same applies to the monitoring judgments when learning from texts (i.e., metacomprehension) [29]. Learning from textbased material is a common and highly important activity across all educational levels, requiring students to read text passages and comprehend the content [30]. An explanation for poor metacomprehension is that the cues students base their judgments on are not diagnostic of their comprehension performance [31]. For example, how well they can read the text (i.e., processing fluency) is not diagnostic of how well they have comprehended it.
To improve metacomprehension, researchers examined various interventions, such as summary writing, self-explanations, and concept mapping. A recent meta-analysis [32] showed that delayed summary writing was the most effective technique to improve comprehension accuracy among the different standalone interventions. Moreover, while delayed summary significantly improved metacomprehension, immediate summary was not considerably better compared to non-intervention conditions. The delayed-summary effect is supported by the situation model hypothesis, which states that metacomprehension is more accurate when based on cues reflecting the quality of one's situation model representation [33].
While interventions built on the situation-model approach aimed at helping students generate, focus on, and select situation-model cues (e.g., the experience of generating summary after delay) are generally effective at improving metacomprehension, the need to exert additional effort to implement them might hinder students from doing them during SRL [34]. In addition, students may differ in their abilities to carry out generative activities. Research suggests that students need help to produce highquality summaries [35], and providing pre-defined summaries is more beneficial than self-generated summaries [36]. Alternatively, providing feedback to students after an immediate generative activity may allow students to restudy and focus on more relevant information while filtering irrelevant information [36]. With the advances in GenAI technologies, new opportunities arise in using genAI chatbots to support students' SRL when learning from texts, such as by generating summaries for students or providing feedback on student-generated summaries. The aim of the current study is twofold in the context of text comprehension: 1) to examine the use of GenAI chatbot to enhance metacomprehension and performance, and 2) to examine the role of self-efficacy when using GenAI chatbot as a study support tool. Two main research questions are:
1. What is the effect of delayed summary and GenAI supported immediate summary on metacomprehension accuracy, performance, and mental effort? 2. Does self-efficacy mediate the effectiveness of delayed summary and GenAI supported immediate summary on performance?
Method. The second study employed a between-subject design with four conditions (see Table 1 for an overview of the four conditions). An a priori power analysis was conducted using G*Power version 3.1. [38] to determine the minimum sample size required to test the study hypothesis. Results indicated required sample size to achieve 80% power for detecting a medium effect of Cohn's d = .5 [32,33], at a significance criterion of α = .05, was N = 180. Bachelor's and master's students from a university in the Netherlands are recruited to participate in the online experiment via course announcements and flyers posted on the campus.
All materials in the study were delivered via an online survey platform, Qualtrics. Table 1 illustrates the procedure of the study. After consenting to the study, students were randomly assigned to one of the four conditions. The delayed-summary and immediate-summary conditions were adapted from [31]. Participants in the delayed-summary condition were asked to read the first text, followed by the second text, before writing a summary for Text 1, followed by Text 2. In the immediate-summary condition, participants were asked to read Text 1 and immediately write a summary for Text 1. They repeated the process for Text 2. There were two additional conditions to examine the use of a GenAI chatbot, i.e., ChatGPT, as a study support tool to provide feedback on self-generated summaries or to provide AIgenerated summaries. Participants in these two conditions were asked to read the first text, and they either had to write a summary and ask ChatGPT to provide feedback on their summary, or they asked ChatGPT to generate a summary that they evaluated and study AI-generated summary. The process was repeated for Text 2.
In all four conditions, self-efficacy was measured before participants read the first text and after, they have completed the summary for the second text. Self-efficacy was measured using a four-item survey adapted from Motivated Strategies For Learning Questionnaire (MSLQ; [39]). Mental effort and metacomprehension were measured using a one-item survey respectively. Participants were prompted to report their mental effort after each summary phase "How easy was it for you to comprehend the passage whose title is listed above?" At the end of both summary phases, they were prompted with the title of each text to elicit their metacomprehension "How well do you think you understood the passage whose title is listed above?". All survey items were measured on a 7-point Likert scale. After providing the comprehension judgment for both texts, participants proceed with a comprehension test. The experiment takes around 45 minutes.
Data collection starts in February 2024, and preliminary results will be shared in the presentation.
The authors would like to acknowledge the master students who are helping in the data collection and the participants of the studies. Further, ChatGPT 3.5 has been used for editing few selected sentences.