This research extends the ACT-R cognitive architecture to tackle deceptive overgeneralization within Intelligent Tutoring Systems (ITS). Existing adaptive learning technologies, while efective, rely on learning data that may not fully capture the nuances of learner understanding, particularly in cases of deceptive overgeneralization. This phenomenon occurs when learners exhibit correct actions during monitored learning sessions, yet these actions are grounded in an incomplete understanding of the necessary conditions. Due to the reliance on observed correctness, ITS may falsely assess mastery, potentially ceasing to provide further necessary practice opportunities that could aid in the refinement of understanding. This study aims to identify ITS designs that may inadvertently foster such misconceptions and to develop methods for their detection, diagnosis, and correction. Utilizing experimental designs, think-aloud protocols, and educational data mining, the research seeks to refine the adaptivity of ITS and enable more accurate assessments of true skill mastery. This work contributes to Technology-Enhanced Learning (TEL) by enhancing the precision of automated assessments and supporting more reliable adaptive learning experiences.
CEUR ceur-ws.org
Adaptive learning technologies, powered by learning data
and dynamically adjusting to individual learner needs,
have proven efective across various educational settings
[
Deceptive overgeneralization describes an undesired learning state wherein a learner acquires a relevant but incomplete subset of the conditions necessary for a skill, yet manages to perform the correct actions. Such overgeneralization is “deceptive”, as it can lead to seemingly satisfactory performance during scrutinized learning sessions, as the learner’s observable actions align with those of individuals who have accurately mastered the skill.
Intelligent Tutoring Systems (ITS) with adaptive capabilities that dynamically select practice problems based on estimated skill mastery, might amplify the issue of deceptive overgeneralizations. Such environments may prematurely cease providing further necessary practice opporing these inaccuracies unaddressed. The consequences of failing to detect and address deceptive overgeneralizations can extend beyond academic performance, potentially afecting long-term educational pathways, career trajectories, and in some cases, leading to dire consequences.
My doctoral research aims to investigate the
mechanisms of deceptive overgeneralization by applying and
extending the well-established cognitive architecture,
Adaptive Control of Thought – Rational (ACT-R) [
solving step against the possible actions generated by the cognitive model, in order to provide individualized, just2.1. Adaptive Control of Thought – in-time learning support tailored to the learners’ specific Rational (ACT-R) approach to a problem [16, p.142].
In the task-loop, ITS employ knowledge tracing
algoACT-R, a cognitive architecture for understanding and rithms such as Bayesian Knowledge Tracing (BKT) [17]
modeling human cognitive processes, posits that cog- to dynamically adjust problem sequences based on
realnitive behaviors are orchestrated by productions [
However, the adaptivity of ITS is not without
limitaAdaptive learning, fundamental to ITS eficacy, is sup- tions. One key challenge lies in addressing deceptive
ported by various theoretical perspectives, such as Vy- overgeneralization—where learners perform correct
acgotsky’s zone of proximal development [
Adaptivity is not a binary property, but rather “a matter of degree” [1, p.523]. ITS distinguish themselves by adapting across all three major time scales defined by the 3. Deceptive Overgeneralization as a Adaptivity Grid: step, task, and design [1, p.525]. Possible Learning State
Within the step-loop, ITS provides timely and targeted
feedback at each problem-solving step. Indeed, timely Learning is typically characterized by a gradual and
confeedback is critical to enable the learners to continu- tinuous process rather than sudden transformative
inously monitor their learning and evaluate their problem- sights [18]. The Knowledge-Learning-Instruction (KLI)
solving strategies and their current understanding [
In ITS, specifically those developed using Cognitive Tutor Authoring Tools (CTAT) [24, 25], each production’s 3.2. Stickiness of Deceptive condition-action pair is structured as an IF-THEN state- Overgeneralization ment [26]: IF <condition> THEN <action>. Overgeneralization occurs when a learner acquires production The KLI framework delineates a relationship between obrules whose IF part is overly broad compared to the cor- servable and unobservable events: instructional events, rect IF part. In computational or logical terms, overgen- learning events, and assessment events [19]. Instruceralization can happen due to the omission of logical AND tional events cause learning events, which are unobservoperators in the IF part. Consider a target KC requiring able processes that result in changes in KCs, such as multiple conditions for its activation, represented as IF A acquisition of new KCs or refinement of existing KCs. AND B THEN <action>. Overgeneralization might arise The changes of KCs, in turn, cause learner performances when a learner acquires a KC that omits part of the con- that are observable during assessment events. Given that ditions, resulting in IF A THEN <action>. learning events are central yet unobservable, assessments
It is crucial to distinguish the phenomenon of deceptive are expected to be designed with the quality to accurately overgeneralization from the broader concept of “miscon- reflect the true nature of learning events. However, in ceptions.” Consider a simple algebra problem: Anderson cases of overgeneralization, certain designs may fail short. describes an observation that a student incorrectly solves Using set theory, overgeneralization can be visualized as the equation 2 = 6 by subtracting 2 from both sides, an inclusion relation and we can identify a specific type erroneously resulting in = 4 instead of = 3 [18]. Such of potential design flaw, as depicted in Figure 1. misconceptions lead to actions that are clearly incorrect, Many TEL environments, particularly those involvallowing for immediate observation, feedback provision, ing ITS, leverage automated evaluation and feedback and tailored subsequent training. In contrast, deceptive mechanisms to deliver learning at scale. The reliance model suggests, while competence develops in a more-orOvergeneralizedIF less linear fashion, consciousness initially increases and then decreases, as both novices (in Stage 1) and experts (in Stage 4) operate in states of relative unconsciousness, CorrectIF though for vastly diferent reasons [ 15, p.97]. I contend that deceptive overgeneralization may occur during any stage transition, including transitions towards Stage 4.
Experts, as they develop their proficiency and automaticFigure 1: Overgeneralization occurs when a learner acquires ity, may also be prone to forming inaccurate heuristics production rules whose IF part is a superset of the correct and cognitive shortcuts to enable fast task completion. rule’s IF part, covering an overly extended range. This rela- An example demonstrating that experts can form detionship can be expressed as OvergeneralizedIF ⊇ CorrectIF. ceptive overgeneralization, and that deceptive overgenCross marks within CorrectIF represent practice activities that eralization can lead to severe consequences, is the Croscannot test for overgeneralization. If all practice activities fall sair Flight 498 Crash. The oficial incident investigation within CorrectIF, focusing solely on correct actions, the in- report identified one human factor probable cause as folstructional design will fail to identify whether learners have lows: “when interpreting the attitude display instruments acquired the correct rule or an overgeneralization. under stress, the commander resorted to a reaction pattern (heuristics) which he had learned earlier” [29, p.10].
As demonstrated in Figure 3, a Soviet attitude dison these automated mechanisms can pose challenges for play indicates a left roll of the airplane with a counterall stakeholders regarding deceptive overgeneralization. clockwise rotation. The appropriate response, detailed TEL tools might mistakenly provide positive feedback in Algorithm 1, is to stabilize the airplane by rotating it to learners who perform correct actions based on an right. This rule acts as a cognitive shortcut that simpliinaccurate understanding of conditions, inadvertently ifes decision-making by minimizing the cognitive load reinforcing misconceptions. Instructors and researchers needed to interpret the display. However, errors can arise employing learning analytics or educational data mining if this shortcut is overgeneralized, omitting the condition are similarly at risk of being misled by seemingly satis- that it should only apply to Soviet displays, leading to factory learning data, potentially missing opportunities incorrect responses with other types of attitude displays. for intervention and correction that address learners’ incorrect understandings. Moreover, ITS, with its adaptive Algorithm 1 Correct Production Rule for Interpreting capabilities that dynamically select practice problems and (Soviet) Attitude Display to Stabilize an Airplane assess mastery, might amplify these issues. The reliance if the goal is stabilize an airplane and attitude display on observed correctness by knowledge tracing algorithms rotates counter-clockwise and it is a Soviet display can lead to premature conclusions about learner mastery, then halting further necessary practice that aids genuine skill rotate the airplane right development and refinement, leaving those misconcep- end if tions unaddressed. As what is captured and reported by TEL tools appears correct, encouraging, and satisfactory, deceptive overgeneralization may be particularly “sticky” and resistant to detection and change.
mander received training that was “in theory comprehensive,” exclusively at a flying school in the former Soviet Union [29, p.18]. However, upon transitioning 3.3. Both Novices and Experts Could be to aircraft equipped with Western systems, no special Prone to Deceptive diferential training was provided to highlight the diferOvergeneralization ences between Eastern and Western systems, nor did the commander undergo any unusual attitude training [29, If “practice makes perfect” were true to the extent that p.19]. Therefore, the commander “had no opportunity to well-developed expertise guarantee refined and accurate be trained in any other pattern of behavior” [29, p.96], skills, then deceptive overgeneralization could be efec- meaning no opportunities to ever detect and correct the tively addressed by providing ample practice opportuni- acquired deceptive overgeneralization. As the commanties in favorable learning conditions. However, I argue der resorted to the overgeneralization in the scenario as that even experts are not immune to deceptive overgen- illustrated in Figure 4, the commander kept rotating the eralization, despite their considerable mastery of skills. airplane right (further) when the airplane was already
Ambrose et al. [15, p.97] modeled mastery and its devel- rolling right, eventually resulting in a loss of control. opment into four stages, as illustrated in Figure 2. As this The acquisition of shortcuts can be modeled using the Do not know what they do not know 1
Recognize what they do not know and need to learn 2
Act deliberately with considerable competence 3
Act automatically and instinctively 4 Unconscious Incompetence
Conscious Incompetence
Conscious Competence
Unconscious Competence
However, it is possible that even experts who have masrate “macro-productions” during the process of building proficiency and automaticity if errors enter into the compilation process. Although composition increases overall eficiency by pruning redundant conditions and actions, these composed macroproductions tend to grow larger, particularly with an increase in the size of the condition (b) After counter-clockwise rotation The display reflects a “third-person view”, where the horizon stays fixed, and the airplane’s position is shown relative to the horizon. A counter-clockwise rotation (of the airplane relative to the horizon) indicates that the airplane is rolling left . could be compiled as shown in Algorithm 2. These pro- the possibility of condition omission [5, p.46]. process called knowledge compilation in the ACT-R the- sides [2, p.239]. With an increasingly more complex and ory, which serves to eliminate multiple production firings and the need for retrieval from declarative memory [4, some conditions will be overlooked, potentially leading composite condition side, it becomes more likely that p.169]. A primary compilation process, known as composition, is to takes sequences of productions that follow each other in solving a particular problem and collapses them into a single “macro-production” that has the efect of the sequence [2, p.235]. For example, Algorithm 1 duction rules are intentionally represented in pseudo code, mimicking the implementation style of cognitive tutors developed with CTAT [24]. This representation serves to highlight several benefits of composition: fewer conditions and actions, fewer variables to track, and the to overgeneralization. While human compilation is gradual (in contrast to computer compilation), which may provide some protection against errors of omitting conditional tests from entering compilation, this protection is not infallible and can only reduce, but not eliminate,
new productions generated through knowledge compilation do not replace, but rather coexist with old ones [2, p.237]. A process known as conflict resolution then determines which productions to apply [2, p.132]. This raises
Attitude Display to Stabilize an Airplane may be widespread, which highlights the importance of understanding their mechanisms through research.
Rule P1: The commander’s extensive experience, amounting to Condition: goal == stabilizeAirplane AND rollDirec- over 8,000 hours [29, p.15], categorizes him within Stage tion == unknown 4 of the mastery model illustrated in Figure 2, where inAction: subgoal = identifyRollDirection dividuals are capable of acting automatically and instincRule P2: tively. However, this incident starkly demonstrates that Condition: subgoal == identifyRollDirection AND dis- such automatic actions performed by experts, when based playRotation == counterClockwise AND displayType on deceptive overgeneralization, can lead to dire conse== Soviet quences. A similar case, that exemplifies the dangers of Action: rollDirection = left overgeneralization in aviation training, is the American Rule P3: Airlines Flight 587 crash, where poorly-designed trainCondition: goal == stabilizeAirplane AND rollDirec- ing led to deceptive overgeneralization, resulting in actions tion != unknown deemed correct during training but were inappropriate for Action: subgoal = recoverAttitude actual conditions, ultimately leading to catastrophic outRule P4: comes. Specifically, the American Airlines Advanced AirCondition: subgoal == recoverAttitude AND rollDirec- craft Maneuvering Program included an excessive bank tion == left angle simulator exercise intended to prepare pilots for Action: rotateAirplane(right) extreme wake turbulence. This equipped trainees with Composed Rule P1&P2&P3&P4: aggressive roll upset recovery techniques. Unfortunately, Condition: goal == stabilizeAirplane AND displayRota- the scenario used in training was overly extreme and tion == counterClockwise AND displayType == Soviet not representative of the actual aircraft type involved. Action: rotateAirplane(right) This inappropriate training “enabled” the first oficer to Eficiency Gain: mistakenly apply these excessive techniques during a 2 subgoals, 4 conditions, 3 intermediate cognitive ac- moderate wake turbulence encounter, leading to the intions, and 2 variables get reduced by composition lfight separation of the vertical stabilizer and culminating in a fatal plane nosedive [31]. It can be argued that had the pilot not been trained to perform such aggressive maneuvers, the disaster could have been entirely avoided.
In summary, acquiring a production rule that pairs correct actions with incorrect conditions is an undesirable learning outcome, which at best might later be rectified without severe repercussions, and at worst, could result in catastrophic outcomes. the question of why the commander chose the overgeneralized shortcut over the basic alternative productions.
The ACT-R strengthening mechanism might provide an explanation [2, p.250]. Production strength reflects the frequency of successful past applications [2, p.133]. Over the years, while flying Soviet aircraft, this shortcut—despite being overgeneralized—consistently led to correct actions within the context of Soviet attitude displays. 3.4. Summary This increased production strength may have made this This section presents the problem identification and exshortcut the preferred choice during conflict resolution. amination on the phenomenon of deceptive overgeneral
Another contributing factor to the commander’s selec- ization through literature review and case studies, yieldtion of the overgeneralization could be the medication ing several key characteristics of deceptive overgeneralefects, which potentially limited the commander’s cogni- ization that underscore the need for further investigation: tive capacity [29, p.107]. The improved eficiency of the composed shortcut may have prompted the commander 1. Deceptive overgeneralization is prevalent across to favor the overgeneralized macro-production over a various domains. sequence of basic productions, especially under stress 2. Deceptive overgeneralization can be “sticky”, difrequiring immediate action, and possibly while multitask- ifcult to detect and resistant to change. ing. Such demanding and stressful scenarios are common, 3. In certain cases, deceptive overgeneralization can particularly in fields where individuals are considered be worse learning outcomes than if the skill had experts and carry critical responsibilities. Moreover, sit- not been learned at all. uations involving limited cognitive capacity can occur 4. Both novices and experts could be prone to deto anyone. The ability to perform under conditions of ceptive overgeneralization. stress, sleep deprivation, or fatigue is crucial, as is the capability to efectively manage simultaneous secondary tasks [30]. This indicates that overgeneralized shortcuts
task-loop adaptivity. However, these systems are not specifically designed to prevent deceptive overgeneralMy doctoral research aims to investigate the mechanisms ization. My experimental design draws inspiration from of deceptive overgeneralization using the context of ITS studies on the Einstellung efect, which describes how and develop efective strategies for addressing deceptive practice with a fixed method can bias individuals toward overgeneralization. The proposed research questions applying this method even when better alternatives exare structured to methodically examine the formation, ist [33]. In my experiments, learners will practice using detection, remediation, and retrospective discovery of ITS until they have achieved mastery as deemed by ITS. deceptive overgeneralization: Subsequently, these learners will face tasks where the
RQ1: Formation of Deceptive Overgeneralization. actions they have learned are no longer suitable. As my What types of production rules are most susceptible to research contends that ITS may have limitations when deceptive overgeneralization? Under what conditions do it comes to accurately assessing true skill mastery, the ITS risk promoting deceptive overgeneralization? research plan will incorporate qualitative data collected
RQ2: Detection and Diagnosis of Deceptive Over- through think-aloud studies [34]. Specifically, “graduated generalization. What features can be integrated into novices”—learners who have completed training and are ITS to detect and diagnose deceptive overgeneralization? judged by the ITS to have mastered the content—will ver
RQ3: Remediation of Deceptive Overgeneraliza- balize their understanding of the conditions during these tion. What instructional strategies are efective at cor- sessions, in order to identify instances of deceptive overrecting deceptive overgeneralization? generalization. Next, to ascertain under what conditions
RQ4: Retrospective Discovery of Past Deceptive ITS may inadvertently promote deceptive overgeneralOvergeneralization. Can Educational Data Mining ization and to identify which features of instructional (EDM) techniques discover previously undetected decep- design are most susceptible to fostering these errors, my tive overgeneralization from existing education datasets? research plan includes conducting randomized controlled trials (RCTs) that compare diferent ITS interface designs 5. Methodology and problem sequencing.
RQ2: Detection and Diagnosis of Deceptive OverThis section has outlined the research methodologies generalization. To investigate features that can be intecorresponding to each of the research questions guid- grated into ITS for efectively detecting and diagnosing ing my doctoral study. To rigorously investigate the deceptive overgeneralization, RCTs will be conducted phenomenon of deceptive overgeneralization, a diverse to compare diferent ITS interface designs and problem methodological approach will be employed. The methods sequencing. range from experiments, think-aloud studies, and EDM Traditionally, ITS interfaces are designed to guide techniques, as summarized in Table 2. learners toward correct actions, potentially neglecting
RQ1: Formation of Deceptive Overgeneraliza- interface elements which represent potential incorrect tions. The initial step in my research is to evaluate actions that learners should avoid, as these elements do the hypothesized design flaw, as illustrated in Figure 1. not belong to the prescribed solution pathway. ConThis hypothesis suggests that when a series of practice sequently, learners might attempt to perform incorrect activities only evaluate whether learners have performed actions but find themselves unable to do so, making those the expected actions, such instructional designs may not mistakes undetected, uncorrected, and unlogged. One adequately determine whether learners have internalized hypothesized efective design is to provide practice opthe correct rule or an overgeneralization. portunities where “lack of action” is the correct response.
My research strategy includes conducting experiments Although detecting non-actions poses more challenges
with ITS that adhere to best practices in ITS design, such than evaluating actions, we may consider ITS design
as cognitive model development through Cognitive Task that incorporates interface elements that learners should
Analysis (CTA) [32], tailored hints and feedback, and avoid interacting with, in order to make “lack of action”
observable and test whether learners can appropriately
refrain from actions when the conditions do not warrant tribute to the TEL community if there is evidence that the
them. This approach is similar to including distractor op- research findings can also generate actionable insights
tions in multiple-choice questions (MCQs), where learn- using existing datasets. Therefore, the last research
quesers must correctly identify and decide against choosing tion focuses on retrospective analysis to discover past
such options. Of course, the expertise reversal efect [
RQ3: Remediation of Deceptive Overgeneraliza- stagnation as learners engage in repeated practice opportion. Similar to RQ2, RCTs that compare diferent ITS tunities [36]. ITS systems developed with CTAT, which interface designs and problem sequencing will be con- typically store learning logs in DataShop, which are ready ducted. One instructional design hypothesized to be candidates for retrospective analysis. efective involves providing side-by-side comparisons To efectively visualize and demonstrate learning between scenarios that do and do not warrant certain curves that may indicate overgeneralization, I will start actions. This approach requires learners to identify dif- with synthetic data. Synthetic data, artificially generated ferences in problem features, facilitating a deeper under- by computer algorithms and not derived from real-world standing of when specific actions are appropriate. events, mimics authentic datasets. The ethical gener
Incorporating both RQ2 and RQ3, the problem sequenc- ation and application of synthetic data is a widely acing design pattern illustrated in Algorithm 3 is hypoth- cepted practice in learning sciences, particularly within esized to aid both in initial induction and subsequent the realm of Educational Data Mining (EDM), as evirefinement, and can detect, diagnose, and remedy decep- denced by its use in numerous EDM research studies tive overgeneralization. The checkSAI() function, as in [37, 38, 39, 40, 41]. Synthetic data addresses the complexCTAT, represents the automated evaluation by ITS that ities of authentic learner data, aiding in the validation of compare learner actions with reference ones [24]. models for skill mastery assessment, and can faithfully reflect reality when properly modeled [ 41].
Algorithm 3 Problem Sequencing Design Hypothesized To examine how deceptive overgeneralization afects to Aid in Initial Induction and Subsequent Refinement learning trajectories, BKT was used to simulate perTarget Knowledge Component (KC): formance with problem sequencing illustrated in Algoif AND then rithm 3 with the following parameters: = 0.5, <action> = 0.2, = 0.1, and = 0.2. The learning end if process is modeled with a single KC with three possible Potential Overgeneralization: states: Unlearned, Overgeneralized, and Learned. This if then approach adheres to the BKT framework by treating the <action> learning progression as a transition between states. As end if learners in the Unlearned state receive repeated practice Problem Type 1: Designed for Induction opportunities, they may either remain in the Unlearned if AND then state, transition to an Overgeneralized state, or move checkSAI(<action>) directly to the Learned state. Learners in the Overgeneralend if ized state can only possibly progress to the Learned state Problem Type 2: Designed for Refinement through problems designed for refinement. Another core Problem Subtype 2.1: Unsuitable Context assumption made in the simulation is the probability of if AND NOT then correct responses based on knowledge state and problem checkSAI(NO <action>) phase, as illustrated in Table 3. Problems designed for inend if duction can be correctly answered (unless a slip occurs) Problem Subtype 2.2: Insuficient Information using either the correct generalization or an overgenif AND Missing Info about then eralization. For the problems designed for refinement, checkSAI("Not Enough Info") learners who either remain in the Unlearned state or who end if have adopted the overgeneralized rule are expected to answer incorrectly most of the time. However, rather than guessing like those in the Unlearned state, learners in
RQ4: Retrospective Discovery of Past Deceptive the Overgeneralized state will answer incorrectly unless Overgeneralization. In addition to designing and con- a slip occurs, which reflects how learners with decepducting experiments specifically for investigating decep- tive overgeneralization will “confidently” make mistakes tive overgeneralization, my research could better con- when the conditions do not actually warrant the actions.
nitive architecture to tackle deceptive overgeneralization.
My research seeks to refine the adaptivity of ITS and enable more accurate assessments of true skill mastery.
This work contributes to Technology-Enhanced Learning (TEL) by enhancing the precision of automated assessments and supporting more reliable adaptive learning experiences. [14] V. J. Shute, Focus on formative feedback, Review 371–416.
of educational research 78 (2008) 153–189. [29] A. A. I. Bureau, Final report of the aircraft accident [15] S. A. Ambrose, M. W. Bridges, M. DiPietro, M. C. investigation bureau on the accident to the saab Lovett, M. K. Norman, How learning works: Seven 340b aircraft, registration hb-akk of crossair flight research-based principles for smart teaching, John crx 498 on 10 january 2000 near nassenwil/zh, 2002.
Wiley & Sons, 2010. [30] R. A. Schmidt, R. A. Bjork, New conceptualizations [16] K. R. Koedinger, A. Corbett, et al., Cognitive tu- of practice: Common principles in three paradigms tors: Technology bringing learning sciences to the suggest new concepts for training, Psychological classroom, na, 2006. science 3 (1992) 207–218. [17] A. T. Corbett, J. R. Anderson, Knowledge tracing: [31] S. Board, In-flight separation of vertical stabilizer Modeling the acquisition of procedural knowledge, american airlines flight 587 airbus industrie a300User modeling and user-adapted interaction 4 (1994) 605r, n14053 belle harbor, new york november 12, 253–278. 2001, National Transportation Safety Board 490 [18] J. R. Anderson, C. D. Schunn, Implications of the act- (2001).
r learning theory: No magic bullets, in: Advances [32] J. M. Schraagen, S. F. Chipman, V. L. Shalin, Cogniin instructional Psychology, Volume 5, Routledge, tive task analysis, Psychology Press, 2000. 2013, pp. 1–33. [33] A. S. Luchins, Mechanization in problem solving: [19] K. R. Koedinger, A. T. Corbett, C. Perfetti, The The efect of einstellung., Psychological monoknowledge-learning-instruction framework: Bridg- graphs 54 (1942) i. ing the science-practice chasm to enhance robust [34] K. A. Ericsson, H. A. Simon, How to study thinking student learning, Cognitive science 36 (2012) in everyday life: Contrasting think-aloud proto757–798. cols with descriptions and explanations of thinking, [20] V. A. Aleven, K. R. Koedinger, An efective metacog- Mind, Culture, and Activity 5 (1998) 178–186. nitive strategy: Learning by doing and explaining [35] K. R. Koedinger, R. S. Baker, K. Cunningham, with a computer-based cognitive tutor, Cognitive A. Skogsholm, B. Leber, J. Stamper, A data reposscience 26 (2002) 147–179. itory for the edm community: The pslc datashop, [21] H. Zhao, K. Koedinger, J. Kowalski, Knowledge Handbook of educational data mining 43 (2010) tracing and cue contrast: Second language english 43–56. grammar instruction, in: Proceedings of the An- [36] K. R. Koedinger, R. S. Baker, K. Cunningham, nual Meeting of the Cognitive Science Society, vol- A. Skogsholm, B. Leber, J. Stamper, A data reposume 35, 2013. itory for the edm community: The pslc datashop, [22] N. M. Chang, Learning to discriminate and gener- Handbook of educational data mining 43 (2010) alize through problem comparisons, Ph.D. thesis, 43–56.
Carnegie Mellon University, 2006. [37] M. M. Rahman, Y. Watanobe, T. Matsumoto, R. U. Ki[23] LearnLab, Feature validity, 2011. URL: https:// ran, K. Nakamura, Educational data mining to suplearnlab.org/wiki/index.php?title=Feature_validity, port programming learning using problem-solving [Online; accessed 29-May-2024]. data, IEEE Access 10 (2022) 26186–26202. [24] V. Aleven, B. McLaren, J. Sewall, K. R. Koedinger, [38] N. Ndou, R. Ajoodha, A. Jadhav, Educational dataExample-tracing tutors: A new paradigm for intel- mining to determine student success at higher eduligent tutoring systems (2009). cation institutions, in: 2020 2nd International Multi[25] V. Aleven, B. M. McLaren, J. Sewall, K. R. Koedinger, disciplinary Information Technology and EngineerThe cognitive tutor authoring tools (CTAT): Prelim- ing Conference (IMITEC), IEEE, 2020, pp. 1–8. inary evaluation of eficiency gains, in: Intelligent [39] J. M. Patil, S. R. Gupta, Extracting knowledge in Tutoring Systems: 8th International Conference, large synthetic datasets using educational data minITS 2006, Jhongli, Taiwan, June 26-30, 2006. Pro- ing and machine learning models, in: Soft Computceedings 8, Springer, 2006, pp. 61–70. ing for Intelligent Systems: Proceedings of ICSCIS [26] K. R. Koedinger, J. R. Anderson, W. H. Hadley, M. A. 2020, Springer, 2021, pp. 167–175.
Mark, Intelligent tutoring goes to school in the big [40] C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, city, International Journal of Artificial Intelligence L. J. Guibas, J. Sohl-Dickstein, Deep knowledge in Education 8 (1997) 30–43. tracing, Advances in neural information processing [27] M. T. Chi, P. J. Feltovich, R. Glaser, Categorization systems 28 (2015).
and representation of physics problems by experts [41] M. C. Desmarais, I. Pelczer, On the faithfulness of and novices, Cognitive science 5 (1981) 121–152. simulated student performance data, in: Educa[28] B. H. Ross, Remindings and their efects in learning tional Data Mining 2010, 2010. a cognitive skill, Cognitive psychology 16 (1984)