Developing and Evaluating an Interactive Reading Tool with Teachers in the Loop: Action Research Approach Mihwa Lee Hector Research Institute of Education Sciences and Psychology, Universirty of Tübingen, Walter-Simon-Str. 12, 72072 Tübingen, Germany LEAD Graduate School & Research Network, University of Tübingen, Europastr. 6, 72072 Tübingen, Germany Abstract Reading is an essential life skill and crucial for students’ academic success. Especially, there has been an increasing necessity for students to read in English as a second language (L2) due to its global importance. However, teachers in schools often face challenges in providing interactive L2 reading experiences for a large number of students due to limited time and highly heterogeneous students, leading to L2 readers having few opportunities for meaningful, interactive reading practice with instant support. The rapid advancements of artificial intelligence (AI) in education have given rise to a number of opportunities for interactive and adaptive learning. Despite significant advancements in AI-powered educational tools, many language educators continue to view them with skepticism. This may stem from a perceived misalignment between teaching methods that educators find effective and the features or approaches offered by these technologies. As a result, the gap between educators’ expectations and the capabilities of AI-driven solutions remains a point of concern. It is crucial to ensure that educational systems align with established theories and pedagogical insights, and to investigate them from multiple perspectives, including perceptions of the system, learning outcomes, motivation, and learning behavior to better design educational products. This article introduces a pedagogically grounded web-based intelligent computer-assisted language learning (ICALL) system, designed to enhance L2 reading experiences, developed using the Action Research design with teachers in the loop. The article details the system’s development and provide an overview of ongoing and planned studies, which focus on different aspects of the ICALL system, examining learners’ behaviors through interaction logs to further L2 learning research and improve educational tools. Keywords Reading comprehension, Language learning, Intelligent computer-assisted language learning (ICALL), Process data 1. Introduction To address this gap between research on language ed- ucation, foreign language teaching insights, and real-life In today’s increasingly globalized world, the increasing ne- classroom usage, an ICALL system that systematically and cessity for students to read in English as a L2 underscores automatically provides various interactive support for L2 the importance of proficient L2 reading skills. Learning to reading has been designed and developed, targeting learners read in L2 is complex, as learners must grasp literacy in an of English as a foreign language (EFL). The design and de- unfamiliar language [1]. Thus, there is an urgent need to velopment of the system is grounded in theories in Second support L2 reading from the early school years. However, Language Acquisition (SLA), educational sciences, and ped- teachers face challenges in providing interactive and adap- agogical insights from school practitioners, and leverages tive learning experiences for a large number of students the affordance of the Natural Language Processing (NLP) with limited time. Digital environments, such as ICALL sys- tools and Large Language Model (LLM). In this article, we tems, offer unique opportunities for new ways of learning introduce the design rationale of the system and present the and teaching [2]. These systems have been shown to en- plans and status of studies assessing the effectiveness of the hance learning engagement [3] and achieve better language system in promoting L2 reading from various dimensions acquisition [4] through features such as automatic feedback and examining learners’ learning behaviors using interac- [5], intelligent tutoring [6], and personalized support [7]. tion logs stored in the system. Specifically, the first goal Despite these advancements, there remains a significant gap is to design and develop an ICALL system that supports of the use of such tools in school settings, possibly because and enhances L2 reading comprehension based on the SLA of the skepticism among practitioners [8] due to not only theories and teachers’ insights. The second goal is to ex- people’s lack of knowledge of the field and its capabilities amine the effectiveness of the ICALL system in promoting but also the fact that a lot of AI-based education applications students’ learning outcomes and motivation compared to do not meet educators’ expectations of how effective lan- traditional online reading practice. Lastly, the third goal guage teaching and learning should be conducted [9]. Given is to investigate learners’ self-regulated learning behavior the complexity of challenges in AI in education (AIED) and from by combining interaction logs with self-report data. the field’s traditional emphasis on technical aspects, many By exploring these dimensions, we aim to advance L2 learn- AI-driven educational tools and studies struggle to align ing research and refine educational tools to better support with the most recent advancements in learning theories, reading development in school contexts. empirical research findings, and pedagogical insights [9]. Proceedings of the Doctoral Consortium of the 19th European Conference 2. Background on Technology Enhanced Learning, 16th September 2024, Krems an der Donau, Austria Envelope-Open mihwa.lee@uni-tuebingen.de (M. Lee) 2.1. Linguistic knowledge in reading GLOBE comprehension https://uni-tuebingen.de/fakultaeten/wirtschafts-und-sozialwissensch aftliche-fakultaet/faecher/fachbereich-sozialwissenschaften/hector-ins Reading is a complex cognitive task that necessitates the titut-fuer-empirische-bildungsforschung/institut/personen/lee-mihwa/ integration of textual information with prior knowledge. (M. Lee) Effective comprehension relies on the reader’s ability to effi- Orcid 0009-0000-0101-3549 (M. Lee) ciently process the visual information presented in the text © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings [1]. Current theories of reading comprehension generally like the desire to learn [18, 21, 22]. [22] emphasizes that depict it as involving multiple interconnected layers of con- effective feedback should guide students to consider both ceptual representation. These layers include a lower-level cognitive and motivational aspects in their learning process, representation that draws on text-based elements such as particularly when using computer-assisted learning tools. vocabulary and grammar, and a higher-level representation Therefore, one can assume that providing adaptive and scaf- where the textual content is incorporated into the reader’s folding feedback for learners potentially triggers changes broader conceptual framework (e.g., combining informa- of learners’ attitudes (motivational component) and read- tion across sentences) [10, 11, 12]. The reader’s vocabulary ing strategies (cognitive component), which consequently and grammar knowledge significantly shape the formation improves reading comprehension [21]. of these semantic structures throughout the reading pro- Despite the significant role feedback plays in reading com- cess [10]. Specifically, vocabulary and grammar steer the prehension, teachers are usually the only reliable source of parsing process, which builds meaning from local text seg- feedback for learners in real-life classroom settings. How- ments. If the local-level representations are inaccurate or ever, their time and the amount they can spend with each incomplete, overall text comprehension can be significantly student in class are very limited, resulting in few opportu- hindered [10, 13]. Lexical-syntactic knowledge is essential nities for learners to receive individual formative feedback. for constructing the local-level representation, which forms This is especially important given the substantial individual the foundation for higher-level text comprehension [11, 14]. differences in aptitude and proficiency [23]. Another issue Thus, vocabulary and grammar facilitate the building of is the lack of research on how feedback enhances L2 reading text-based propositions and contribute to deeper compre- comprehension. A recent meta-analysis [21] indicates that hension. research on feedback has predominantly concentrated on SLA researchers have also focused on the role of vocabu- learning outcomes related to reading comprehension, where lary and grammar knowledge in understanding L2 reading few studies have explored the cognitive and affective pro- comprehension. Numerous studies have explored how these cesses triggered by feedback aimed at text comprehension. factors influence L2 reading comprehension, with findings Furthermore, most of this research has focused on the effec- consistently underscoring the importance of morphosyn- tiveness of feedback in reading comprehension in the first tactic knowledge [14]. Recent meta-analyses on L2 read- langauge (L1), with relatively little attention given to its ing comprehension [14, 15] highlight that vocabulary and impact on L2 learners’ reading comprehension. Therefore, grammar knowledge are the two strongest predictors of further empirical research is needed to investigate if and L2 reading comprehension. Consequently, vocabulary and how such feedback, especially in the context of a computer- grammar knowledge have a significant impact—whether assisted learning environment, can enhance learners’ learn- directly or indirectly—on reading comprehension. ing processes and, consequently, influence their learning From the instructional perspective, however, it is almost behavior and overall reading comprehension. impossible for teachers to pinpoint vocabulary and gram- matical knowledge that each learner does not understand 2.3. Self-regulated learning (SRL) in a while students are reading. One way to support teachers in this process is by utilizing supportive computer envi- computer-based learning environment ronments. Despite the importance of such fundamental Self-regulated learning (SRL) broadly refers to an educa- linguistic skills in reading comprehension, however, there tional process in which learners proactively engage in aca- are a relatively small number of technological applications, demic tasks [24, 25]. [24] provides a widely accepted defi- though these (e.g., [16, 17]) involve only minimal use of nition: “active, constructive process whereby students set technology. Therefore, it is crucial for researchers to ac- goals for their learning and then attempt to monitor, regu- count for both vocabulary and grammar when developing late, and control their cognition, motivation, and behavior, language learning applications. Additionally, further re- guided and constrained by their goals and the contextual search is needed to explore whether and how support for features of their environment” (p. 453). In academic litera- these aspects can enhance learners’ L2 reading processes, ture, there is a consensus that SRL is essential for students’ potentially shaping their learning behaviors and improving reading development. Proficient readers are typically highly overall reading comprehension. motivated self-regulated learners who use various reading strategies effectively [26]. Motivation drives learners to use 2.2. Feedback in reading comprehension learning strategies, which helps regulate learning behav- iors and improve outcomes [25, 27]. SRL strategies include Feedback is information communicated to learners to mod- planning, critical thinking, peer learning, effort regulation, ify their thinking or behavior to close the gap between their and goal orientation. Classroom based research indicates actual performance and target performance [18], thus aim- that SRL strategies lead to higher learning performance ing to improve learning [19], as well as enhance emotions [24, 25, 27]. Therefore, supporting students’ motivation and motivation during a learning situation [20]. In the field through, for instance, help options and feedback is cru- of both SLA and educational sciences, feedback is recognized cial for promoting their use of learning strategies, which as an important factor in supporting learning, particularly eventually enhances learning outcomes [28, 29]. Digital when it helps overcome insufficient or false hypotheses technologies have the potential to directly influence learn- [18]. Feedback serves a cognitive function by informing ers’ motivation, strategy use, and outcomes by providing readers of misunderstandings, filling gaps, and increasing interactive and adaptive learning environments that cater awareness of their understanding [21]. This awareness of to individual needs [30]. Hence, investigating the effects of one’s understanding level is crucial for teaching students these technologies on motivation, learning strategies, and to self-regulate their learning from texts, which involves outcomes is urgent. both (meta)cognitive strategies, such as making inferences Previous studies indicated that SRL is the crucial skill for and monitoring comprehension, and motivational processes, success in computer-based learning environments as well 1. What are the characteristics that are considered ben- [31]. However, learners cannot always regulate themselves eficial for supporting and enhancing L2 reading com- successfully because of reasons such as lack of good strategy prehension? What are the students’ and teachers’ use, lack of metacognitive knowledge, failure to control of perceptions of our L2 reading system? metacognitive processes, or lack of experience in learning 2. To what extent is the ICALL system for reading effec- environments with multiple representations. Thus, how to tive in promoting students’ learning outcomes and foster SRL ability has become a central issue in the field of motivation compared to traditional online reading education research and practice. In order to support SRL in practice? the computer-mediated learning environments, instruments 3. What insights do the interaction logs reveal about that capture students’ self-regulation are critical. the learners’ usage of the system and their SRL be- havior? How students with high self-regulation and 2.4. Trace-based measurement of SRL low self-regulation behave differently? The following sections present the plans and status of Offline instruments like self-reported questionnaires and the studies addressing those research questions in detail. At semi-structured interviews have long been used to measure the time of submission, the first research question is being students’ SRL processes in both educational sciences and addressed in a study that is currently taking place. SLA. However, these traditional methods face criticism due to their subjectivity, obtrusiveness, and limited ability to capture the dynamic nature of learning [32]. Such instru- 4. Methodology ments are often unable to reflect all the elements learners attend to during their learning processes [33]. To address Involving teachers or stakeholders in education research these limitations, researchers advocate for the integration whose results will be used in schools is considered very of multiple data types, such as digital-trace data, which important because schools and teachers should not only includes real-time interaction log data [34, 35]. This digital- be treated as consumers of the research results [42]: suc- trace data offers a more granular and continuous insight into cessful research that has a practical impact in schools is SRL, allowing both researchers and practitioners to mon- always the outcome of bi-directional efforts. This the bi- itor students’ learning behaviors and strategic decisions directional effort will not be a one-off process, but a process in online environments with remarkable detail and in real that will involve multiple iterations of interactions between time [36]. These online measures are particularly valuable the research team and the teachers. Consequently, a multi- because they capture cognitive processes as they unfold cycle Action Research paradigm was chosen to guide the during learning, offering a temporal perspective on cogni- research process. The Action Research Model (see Figure 1) tive change and presenting a moment-by-moment view of is a systematic, collective, collaborative, and self-reflective students’ processing behaviors [37]. scientific inquiry aimed at improving educational practices One challenge that is often addressed by researchers is and ad-dressing the practical concerns of teachers [43, 44], importance of aligning the data collection with SRL model where a key characteristic of action research is the involve- [32, 38]. To this end, researchers have often utilised theory- ment of stakeholders, including teachers, students, and re- aligned coding schemes that define SRL processes at differ- searchers. Throughout the project, we adhere to this ap- ent levels of granularity by, for example, coding schemes proach in the process of development, testing, and imple- [39]. Based on these schemes, previous research has primar- mentation of the system. Figure 2 illustrates the overview ily relied on clickstream data from Learning Management of the research design based on the Action Research Model. Systems (LMS) to measure SRL behaviors related to time management, a crucial sub-construct of SRL [40]. These studies consistently demonstrate that clickstream-based measures of time management predict student performance in online learning environments. However, despite the promise of this microanalytic method and its availability due to recent technological ad- vances, its application remains limited in the field of SLA and language learning studies [41]. Furthermore, most research has focused exclusively on time management, leaving other critical SRL sub-constructs largely unexplored in the con- text of digital-trace data collection [40]. This gap highlights Figure 1: Action Research Model, adpated from [43] the need for future studies to broaden their focus to include other dimensions of SRL to fully leverage the potential of RQ1: To answer the first research question, we first con- interaction log data in understanding the complexities of ducted an intensive literature review about L2 reading com- student learning. prehension in order to decide on the characteristics to be implemented in an ICALL system for L2 reading. The aim was to understand the key factors that contribute to effective 3. Research questions L2 reading comprehension and how these can be supported Driven by the objective of advancing L2 learning research in a digital learning environment. As discussed in the Back- and refining educational tools to better support reading ground section (see section 2.1 and section 2.2), reading development in school contexts, this project centers the comprehension involves integrating text information, heav- attention on the following research questions: ily relying on vocabulary and grammar. Feedback plays a crucial role in improving comprehension by helping learn- ers bridge understanding gaps and enhance self-regulation. Figure 2: Overvivew of methodology based on Action Research Model The first prototype of our ICALL system, called ARES, in- cludes features to support these aspects (more discussion of the technical side of the system development can be found in Lee et al. (2024)). Following the Action Research Model, multiple consultations with English practitioners and teach- ers from German secondary schools (”Gymnasiums”) were conducted. This collaborative co-design approach ensured that the system’s features met not only the SLA theories, but also practical classroom needs and pedagogical insights. Figures 3–8 illustrate some features of the first prototype of the system, developed upon after the initial “Plan” phase of the first iteration cycle of the Action Research Model. Using the NLP tools, key features on the learner side include: • On-demand interactive lookup on language means: learners can access detailed explanations and exam- ples of language means directly within the reading text, adaptively helping them understand grammar rules in context according to their need (see Figure 3). • On-demand interactive vocabulary lookup: learners can access detailed explanations and examples of vocabulary in terms of its form, meaning, and use Figure 3: Lookup of language means directly within the reading text, adaptively helping them understand vocabulary in context according to their need (see Figure 4). • Elaborated feedback: learners receive detailed, per- sonalized feedback on their reading and comprehen- sion activities, highlighting areas of strength and providing targeted suggestions for improvement (see Figure 5). In addition to the features that support learners, educa- tional systems should also support teachers so that they can be used in real-life classroom contexts. At the same time, however, it should not replace the teachers. Rather, it should help teachers. Therefore, with the LLM (ChatGPT4o1 ), the system includes features and resources that empower teach- ers to effectively support L2 reading development in their classrooms, while at the same time it is designed in a way that teachers’ expertise is always involved in the process (more discussion of the technical side of the system develop- ment can be found in Lee et al. (2024)). They can post-edit suggestions by the LLM, confirm them, or add their own questions manually. In this way, teachers make the ultimate decision about what to show the students. Key features on the teacher side include: Figure 4: Vocabulary lookup • Customization of annotations on language means: teachers can customize which annotations on lan- guage means are shown to students to align with their instructional goals and the specific needs of 1 their students (see Figure 6). https://chatgpt.com/ Figure 5: Feedback for student’s response • Question generation: the system generates reading Figure 7: Question generation for a reading text comprehension questions (factual and inferential) based on the reading text, helping teachers provide questions tailored to the text (see Figure 7). • Feedback generation: the system creates personal- ized feedback for students based on their perfor- mance, helping teachers provide individualized sup- port (see Figure 8). • Evaluation: the system evaluates student responses to comprehension questions, providing immediate grading, which reduces the grading burden on teach- ers. • Minimalistic analytics: the system provides simple analytics on student performance and engagement, offering teachers quick insights without overwhelm- ing them with data. • Text bank and uploading: the system not only in- cludes a library of reading texts of a variety of topics but also lets the teachers upload texts, allowing them to tailor the reading materials to their curriculum Figure 8: Grading of individual submission and students’ interests. design. In order to enable Learning Analytics, all user ac- tivities such as button clicks, lookups of language means, reading comprehension question attempts, assignment sub- missions, viewing of specific feedback messages, and any other relevant user actions are logged through xAPI6 , an interoperability specification for learning technology, and stored in a Learning Record Store (LRS) in the database. Since the first version of the system is deployed, a study investigating teachers’ and students’ perceptions of the system is currently taking place in two intact English classes at secondary schools (students around age 13-14) in southwest Germany with the purpose of evaluating the system’s usability and overall task and system design. Figure 6: Selection of annotations of language means These mixed-gender classes are part of the academic track of the German education system. The curriculum at this grade level is equivalent to A2-B1 levels on the In terms of the technical aspect, ARES is built with Java Common European Framework of Reference for Languages at the backend, with a Jetty2 server and a Docker3 con- (CEFR) [46], representing the students’ fourth year of tainer. The database is PostgreSQL4 , and the frontend is EFL instruction in school. Over an eight-week period, based on a popular JavaScript framework, HTML, and Boot- students read two texts weekly using ARES as part of their strap5 that provides a highly extensible component-based homework assigned by teachers. A mixed-method approach 2 https://jetty.org/index.html with quantitative data from self-reports and qualitative 3 https://www.docker.com/ data from semi-structured interviews is employed. System 4 https://www.postgresql.org/ 5 6 https://getbootstrap.com/ https://xapi.com/ perceptions are assessed through a self-report questionnaire section (see section 2.4), it is critical to align the data col- of comprehensive evaluation of educational technology lection with SRL model [32, 38]. Consequently, the analysis adapted from [47], which contains closed-ended items in will be guided by the SRL processes proposed by [32] that de- eight evaluation categories such as Usability, Design, and fines the three macro level [54] of SRL processes: Planning, Learning Motivation with a 7-point Likert scale. Additional Engagement, and Evaluation and Reflection. Each process open-ended items asking what students and teachers liked phase is further divided into several micro-level SRL pro- or disliked, and what they wish for the system are included cesses in order to define fine-grained SRL processes. Details as well. For the analysis of the learning behavior from logs, about this theoretical framework and the SRL processes it students’ self-reported SRL skills in online learning (Online encompasses are provided in Table 1. Next, to extract the Self-Regulated Learning Questionnaire, OSLQ, adapted from SRL behavior implied by the actions, the actions will be ag- [48]) are also collected from students. After filling in the gregated into a common xAPI statement structure with the questionnaires, teachers and several students will be invited theoretical framework of SRL processes proposed by [32]. for a follow-up semi-structured interview to gain their Among seven micro-level SRL processes proposed by [32], perceptions of the system in-depth, which will follow the five processes are identified according to the functionality guideline suggested by [49]. For quantitative data analysis of the system where the actions are taken. Table 2 summa- of the self-reports, the mean and standard deviation of of rizes the actions in the systems in ARES mapped to each each close-ended item and category will be calculated. For proposed macro-level and micro-level SRL processes. quantitative analysis from the semi-structures interview, To explore how students with varying levels of self- a reflexive thematic analysis [50] will be conducted. The regulation approach their learning, we will compare their results will be discussed with the English teachers at the self-reported SRL skills with behavioral patterns recorded participating schools to refine the system’s usability and in the system. Behavioral variables will be tracked for each task design. student and assignment to provide a detailed profile of their learning behaviors. K-means cluster analysis will be em- RQ2: To answer the second research question, the study in- ployed to group students based on (1) their self-reported vestigating the effectiveness of the system is planned to take SRL skills and (2) their behavioral patterns as reflected in the place this school year in English classes (students around trace data in order to identify patterns that highlight how age 13-14) in secondary schools in southwest Germany. The well their self-perceptions align with their actual learning study will be administered via the ARES system, and use a behaviors. The resulting clusters will then be compared posttest/pretest design consisting of a battery of tests and to examine correlations between subjective and objective questionnaires. After providing parental consent, partici- measures of SRL, which will help reveal whether students pants will be introduced to the ARES system and complete with strong self-reported SRL skills also demonstrate strong the pre-tests and pre-questionnaires. The teachers will be behavioral evidence of self-regulation, or whether there are asked to assign at least two reading assignments per week discrepancies between the two, providing valuable insights over an eight-week period via the ARES interface. Based on into the alignment (or misalignment) between students’ per- the methodology of [23], rotational within-class randomiza- ceived and actual learning strategies. tion will be employed based on the condition. In the first four weeks, half of each class will serve as the intervention group, using the system with lookup and feedback on com- 5. Conclusion and contribution prehension questions features, while the other half will read Driven by the need to fill the gap between research on plain texts without such aids. In the second four weeks, language education, foreign language teaching insights, this will be reversed. After eight weeks, participants will be and real-life classroom usage, this article presented the instructed to complete the post-tests, post-questionnaires, development of the pedagogically grounded ICALL system and background questionnaire. that provides various learning supports for L2 reading Learning outcomes will be measured by pre- and comprehension and an overview of ongoing and planned post-tests that measure their English vocabulary knowledge studies, which focus on different aspects of the ICALL (Updated Vocabulary Levels Test, [51]), English reading system, examining learners’ behaviors through interaction comprehension (Reading section of TOEFL® Primary™ logs. The results of this project will provide AIED re- Step 2), general English proficiency (Elicited Imitation searchers and language educators with an interdisciplinary test, [52]), and L2 reading motivation (Reading Motivation perspective and further insights on the feasibility and Questionnaire, adapted from [53]). OSLQ [48] will be also capabilities of using the current NLP and AI (LLM) tools used to measure the students’ self-reported SRL skills. in language learning applications and inform system and During the eight-week period, participants’ interaction task design decisions for enhancing learning outcomes. with the system will be tracked. We plan to conduct a Apart from the research plans and studies outlined in this pretest-posttest and pre questionnaire-post questionnaire article, future directions include examining the feasibility comparison across groups, in which we expect improve- of leveraging the LLM to generate short answer questions ments in the measurements on which participants had and feedback, the classification accuracy of annotations access to the aids while learning. on language means, and the efficacy of different feedback types for students with different levels of SRL skills. RQ3: To answer the third research questions, the subset of log data that is stored as students interact with the system from the aforementioned studies addressing RQ1 and RQ2 will be used. Student’s behavioral data will be firstly col- lected as learning logs from the ARES system by extracting students’ interactions in the LRS in form of xAPI statements stored in the system database. As noted in the Background Macro-level Micro-level SRL Description SRL process process Task Analysis To get familiar with the learning context and the definition and requirements of a (learning) Planning task at hand Goal Setting To explicitly set, define, or update learning goals Making Personal To create plans and select strategies for achieving a set learning goal Plans Engage- Working on the Task To consistently engage with a learning task, using tactics and strategies ment Applying Strategy To revise learning strategies, or apply a change in tactics Changes Evaluation Evaluation Evaluating one’s learning process and comparing one’s work with the goal & Applying Strategy Reflecting on individual learning and sharing learning experiences Reflection Changes Table 1 Theoretical framework guiding trace-based measurement of SRL processes, articulated in [32] Macro-level Micro-level SRL Behavioral variables SRL process process Task Analysis Total number of visits to an assignment overview; Total number of visits to a tutorial video; Planning Sum of total time spent on a tutorial video Engage- Working on the Task Total number of visits to assignments before deadline; Sum of total time spent on assign- ment ments before deadline; Total number of visits to reading questions before deadline; Total number of assignments completed; Total number of correct responses; Total number of incorrect responses Applying Strategy Total number of access to vocab help; Total number of access to language means help Changes Evaluation Evaluation Total number of visits to assignments after deadline; Sum of total time spent on assignments & after deadline; Total number of visits to feedback to correct responses after deadline; Total Reflection number of visits to feedback to incorrect responses after deadline; Total number of visits to target answers after deadline Applying Strategy Total number of visits to class average score after deadline; Total number of visits to own Changes score after deadline Table 2 Matching map between the SRL processes and learning behavior data in the system Acknowledgments ronment, ReCALL 29 (2017) 313–334. doi:10.1017/s0 95834401700012x . This PhD project is funded by the German Ministry of Ed- [6] I.-C. Choi, Efficacy of an ICALL tutoring system and ucation and Science (BMBF) under the funding number process-oriented corrective feedback, Computer As- 01IS22076. sisted Language Learning 29 (2016) 334–364. doi:10.1 080/09588221.2014.960941 . [7] M. Heilman, K. Collins-Thompson, J. Callan, M. Eske- References nazi, A. Juffs, L. Wilson, Personalization of reading [1] L. Verhoeven, Second language reading acquisition, passages improves vocabulary acquisition, Interna- in: M. L. Kamil, P. D. Pearson, E. B. Moje, P. Af- tional Journal of Artificial Intelligence in Education flerbach (Eds.), Handbook of reading research, vol- 20 (2010) 73–98. doi:10.3233/JAI- 2010- 0003 . ume 4, Routledge, New York, NY, 2011, pp. 661–683. [8] A. Kate, Three reasons to be skeptical of artificial in- doi:10.4324/9780203840412.ch27 . telligence in schools, 2020. URL: https://www.edweek [2] L. A. Amaral, D. Meurers, On using intelligent .org/technology/three-reasons-to-be-skeptical-of-art computer-assisted language learning in real-life for- ificial-intelligence-in-schools/2020/02. eign language teaching and learning, ReCALL 23 [9] X. Chen, E. Bear, B. Hui, H. Santhi-Ponnusamy, (2011) 4–24. doi:10.1017/S0958344010000261 . D. Meurers, Education theories and ai affordances: De- [3] C.-C. Liu, P.-C. Wang, S.-J. D. Tai, An analysis of sign and implementation of an intelligent computer- student engagement patterns in language learning fa- assisted language learning system, in: M. Rodrigo, cilitated by Web 2.0 technologies, ReCALL 28 (2016) N. Matsuda, A. Cristea, V. Dimitrova (Eds.), Artificial 104–122. doi:10.1017/s095834401600001x . Intelligence in Education. Posters and Late Breaking [4] A. Oberg, P. Daniels, Analysis of the effect a student- Results, Workshops and Tutorials, Industry and In- centred mobile learning instructional method has on novation Tracks, Practitioners’ and Doctoral Consor- language acquisition, Computer Assisted Language tium. AIED 2022., Lecture Notes in Computer Science, Learning 26 (2013) 177–196. doi:10.1080/09588221 Springer, Cham, 2022, pp. 582–585. doi:10.1007/97 .2011.649484 . 8- 3- 031- 11647- 6_120 . [5] H. Ai, Providing graduated corrective feedback in an [10] J. Jung, Second language reading and the role of gram- intelligent computerassisted language learning envi- mar, Working Papers in TESOL and Applied Linguis- tics 9 (2009) 29–48. doi:10.7916/D88915FW . [11] W. Kintsch, The role of knowledge in discourse com- cational Psychology 11 (1986) 307–313. doi:10.1016/ prehension: A construction-integration model, Psy- 0361- 476X(86)90027- 5 . chological Review 95 (1988) 163–182. doi:10.1037/00 [26] H. Li, Z. Gan, Reading motivation, self-regulated 33- 295x.95.2.163 . reading strategies and english vocabulary knowl- [12] W. Kintsch, T. A. van Dijk, Toward a model of text edge: Which most predicted students’ english reading comprehension and production, Psychological Review comprehension?, Frontiers in Psychology 13 (2022) 85 (1978) 363–394. doi:10.1037/0033- 295X.85.5.3 1041870. doi:10.3389/fpsyg.2022.1041870 . 63 . [27] D. H. Schunk, B. J. Zimmerman, Motivation and self- [13] K. Koda, Reading and language learning: Crosslinguis- regulated learning: Theory, research, and applications, tic constraints on second language reading develop- Lawrence Erlbaum Associates Publishers, Mahwah, ment, Language Learning 57 (2007) 1–44. doi:10.111 2008. 1/j.1467- 9922.2007.00411.x . [28] R. S. Jansen, A. van Leeuwen, J. Janssen, R. Conijn, [14] Y. Choi, D. Zhang, The relative role of vocabulary and L. Kester, Supporting learners’ self-regulated learning grammatical knowledge in L2 reading comprehension: in massive open online courses, Computers Education A systematic review of literature, International Review 146 (2020) 103771. doi:10.1016/j.compedu.2019.1 of Applied Linguistics in Language Teaching 59 (2021) 03771 . 1–30. doi:10.1515/iral- 2017- 0033 . [29] S.-H. Jin, K. Im, M. Yoo, I. Roll, K. Seo, Supporting stu- [15] H. Chen, H. Mei, How vocabulary knowledge and dents’ self-regulated learning in online learning using grammar knowledge influence L2 reading comprehen- artificial intelligence applications, International Jour- sion: a finer-grained perspective, European Journal nal of Educational Technology in Higher Education 20 of Psychology of Education (2024) 1–23. doi:10.1007/ (2023) 37. doi:10.1186/s41239- 023- 00406- 5 . s10212- 024- 00793- x . [30] M. L. Bernacki, J. A. Greene, H. Crompton, Mobile [16] D. Meurers, R. Ziai, L. Amaral, A. Boyd, A. Dimitrov, technology, learning, and achievement: Advances in V. Metcalf, N. Ott, Enhancing authentic web pages understanding and measuring the role of mobile tech- for language learners, in: Proceedings of the NAACL nology in education, Contemporary Educational Psy- HLT 2010 Fifth Workshop on Innovative Use of NLP chology 60 (2019) 101827. doi:10.1016/j.cedpsych for Building Educational Applications, BEA 2010, As- .2019.101827 . sociation for Computational Linguistics, Los Angeles, [31] T. Adeyinka, S. Mutula, A proposed model for eval- California, 2010, pp. 10–18. uating the success of webct course content manage- [17] S. Walker, P. Schloss, C. R. Fletcher, C. A. Vogel, R. C. ment system, Computers in Human Behavior 26 (2010) Walker, Visual-syntactic text formatting: A new 1795–1805. method to enhance online reading, Reading Online 8 [32] M. Siadaty, D. Gašević, M. Hatala, Trace-based (2005). micro-analytic measurement of self-regulated learn- [18] J. Hattie, H. Timperley, The power of feedback, Review ing processes, Journal of Learning Analytics 3 (2016) of Educational Research 77 (2007) 81–112. doi:10.310 183––214. doi:10.18608/jla.2016.31.11 . 2/003465430298487 . [33] P. Robinson, A. Mackey, S. Gass, R. Schmidt, Attention [19] V. J. Shute, Focus on formative feedback, Review of and awareness in second language acquisition, in: Educational Research 78 (2008) 153–189. doi:10.310 S. Gass, A. Mackey (Eds.), The Routledge handbook 2/0034654307313795 . of second language acquisition, Routledge, New York, [20] C. J. Fong, E. A. Patall, A. C. Vasquez, S. Stautberg, A NY, 2012, pp. 247–267. meta-analysis of negative feedback on intrinsic mo- [34] D. Gašević, S. Dawson, G. Siemens, Let’s not forget: tivation, Educational Psychology Review 31 (2019) Learning analytics are about learning, TechTrends 59 121–162. doi:10.1007/s10648- 018- 9446- 6 . (2015) 64–71. [21] E. K. Swart, T. M. Nielen, M. T. S. de Jong, Does feed- [35] G. Siemens, R. Baker, Learning analytics and edu- back targeting text comprehension trigger the use of cational data mining: Towards communication and reading strategies or changes in readers’ attitudes? collaboration, in: Proceedings of the 2nd international a meta‐analysis, Journal of Research in Reading 45 conference on learning analytics and knowledge, As- (2022) 171–188. doi:10.1111/1467- 9817.12389 . sociation for Computing Machinery, New York, NY, [22] M. ter Beek, L. Brummer, A. S. Donker, M.-C. J. Opde- 2012, pp. 252–254. doi:10.1145/2330601.2330661 . nakker, Supporting secondary school students’ read- [36] G. Siemens, R. Baker, Student learning in higher educa- ing comprehension in computer environments: A sys- tion: A commentary, Educational Psychology Review tematic review, Journal of Computer Assisted Learning 29 (2017) 353–362. doi:1 0 . 1 0 0 7 / s 1 0 6 4 8 - 0 1 7 - 9 4 1 34 (2018) 557–566. doi:10.1111/jcal.12260 . 0- x . [23] D. Meurers, K. D. Kuthy, F. Nuxoll, B. Rudzewitz, [37] I. Roll, P. H. Winne, Understanding, evaluating, and R. Ziai, Scaling up intervention studies to investi- supporting self-regulated learning using learning an- gate real-life foreign language learning in school, An- alytics, Journal of Learning Analytics 2 (2015) 7–12. nual Review of Applied Linguistics 39 (2019) 161–188. doi:10.1007/s10648- 017- 9410- x . doi:10.1017/S0267190519000126 . [38] Q. Yu, Y. Zhao, The value and practice of learning ana- [24] P. R. Pintrich, The role of goal orientation in self- lytics in computer assisted language learning, Studies regulated learning, in: M. Boekaerts, P. R. Pintrich, in Literature and Language 10 (2015) 90. M. Zeidner (Eds.), Handbook of self-regulation, vol- [39] M. Raković, S. Iqbal, T. Li, Y. Fan, S. Singh, S. Suren- ume 4, Academic Press, San Diego, 2000, pp. 451–502. drannair, J. Kilgour, J. van der Graaf, L. Lim, I. Mole- doi:10.1016/B978- 012109890- 2/50043- 3 . naar, M. Bannert, J. Moore, D. Gašević, Harnessing [25] B. J. Zimmerman, Becoming a self-regulated learner: the potential of trace data and linguistic analysis to Which are the key subprocesses?, Contemporary Edu- predict learner performance in a multi-text writing task, Journal of Computer Assisted Learning 39 (2023) 703–718. [40] Q. Lia, R. Baker, M. Warschauer, Using click- stream data to measure, understand, and support self- regulated learning in online courses, The Internet and Higher Education 45 (2020) 100727. [41] Q. Yu, Learning analytics: The next frontier for com- puter assisted language learning in big data age, in: SHS Web of Conferences, volume 17, EDP Sciences, 2015, p. 02013. doi:10.1051/shsconf/20151702013 . [42] E. Farley-Ripple, H. May, A. Karpyn, K. Tilley, K. Mc- Donough, Rethinking connections between research and practice in education: A conceptual framework, Educational Researcher 47 (2018) 235–245. doi:10.310 2/0013189X18761042 . [43] S. Kemmis, R. McTaggart, The Action Research Plan- ner, 3rd ed., Deakin University Press, Geelong, 1988. [44] R. N. Rapoport, Three dilemmas of action research, Human Relations 23 (1970) 499–513. doi:10.1177/00 1872677002300601 . [45] M. Lee, B. Rudzewitz, X. Chen, Developing a pedagog- ically oriented interactive reading tool with teachers in the loop, in: Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2024), 2024. In press. [46] Council of Europe, Common European Framework of Reference for Languages: Learning, teaching, assess- ment – Companion Volume, Cambridge University Press, Cambridge, 2020. [47] J. W. M. Lai, J. D. Nobile, M. Bower, Y. Breyer, Compre- hensive evaluation of the use of technology in educa- tion – validation with a cohort of global open online learners, Education and Information Technologies 27 (2022) 9877––9911. doi:10.1007/s10639- 022- 10986 - w. [48] L. Barnard, Y. W. Lan, M. Y. To, V. O. Paton, S.-L. Lai, Measuring self-regulation in online and blended learn- ing environments, The Internet and Higher Education 12 (2009) 1–6. [49] E. Rosalind, J. Holland, What is qualitative interview- ing?, Bloomsbury Academic, London, 2013. [50] V. Braun, V. Clarke, Thematic analysis: A practical guide, Sage Publications, Los Angeles, 2022. [51] A. Godfroid, K. M. Kim, The updated vocabulary levels test: Developing and validating two new forms of the vlt, International Journal of Applied Linguistics 168 (2017) 34––70. doi:10.1075/itl.168.1.02web . [52] A. Godfroid, K. M. Kim, The contributions of implicit-statistical learning aptitude to implicit second- language knowledge, Studies in Second Language Ac- quisition 43 (2021) 606––634. doi:10.1017/S0272263 121000085 . [53] W. Wang, Z. Gan, Development and validation of the reading motivation questionnaire in an english as a foreign language context, Psychology in the Schools 58 (2021) 1151––1168. doi:10.1002/pits.22494 . [54] U. Kroehne, F. Goldhammer, How to conceptualize, represent, and analyze log data from technology-based assessments? a generic framework and an application to questionnaire items, Behaviormetrika 45 (2018) 527–563. doi:10.1007/s41237- 018- 0063- y .