Learning performance data (e.g., quiz scores and attempts) is significant for understanding learner engagement and knowledge mastery level. However, the learning performance data collected from Intelligent Tutoring Systems (ITSs) often sufers from sparsity, impacting the accuracy of learner modeling and knowledge assessments. To address this, we introduce the 3DG framework (3-Dimensional tensor for Densification and Generation), a novel approach combining tensor factorization with advanced generative models, including Generative Adversarial Network (GAN) and Generative Pre-trained Transformer (GPT), for enhanced data imputation and augmentation. The framework operates by first representing the data as a three-dimensional tensor, capturing dimensions of learners, questions, and attempts. It then densifies the data through tensor factorization and augments it using Generative AI models, tailored to individual learning patterns identified via clustering. Applied to data from an AutoTutor lesson by the Center for the Study of Adult Literacy (CSAL), the 3DG framework efectively generated scalable, personalized simulations of learning performance. Comparative analysis revealed GAN's superior reliability over GPT-4 in this context, underscoring its potential in addressing data sparsity challenges in ITSs and contributing to the advancement of personalized educational technology.
Intelligent Tutoring System (ITS) is a prototype of computer system designed to ofer
personalized and adaptive instructions through tracing and analyzing learning performance data such as
quiz scores and question attempts [
Tackling data sparsity for ITSs presents a practical yet challenging research area. Informed
by the machine learning literature [
The 3DG framework derived from its three core phases. In the first phase , a 3-dimensional
tensor is constructed to represent learning performance data, with dimensions corresponding
to learners, questions, and attempts. The second phase focuses on densifying the sparse tensor
by tensor factorization. The third phase entails the generation of learning performance data
based on generative models, tailored to the individual learning patterns of learners. The 3DG
framework integrates the multidimensional learner model with generative models to facilitate
scalable simulation sampling for individual learning patterns. The multidimensional learner
model in our framework is derived from the Tensor Factorization method, a widely-used
approach in predicting learner performance in many studies [
Our study investigated a dataset derived from the AutoTutor ITS, focusing on learning
performance in reading comprehension. This dataset originates from lessons developed for the
Center for the Study of Adult Literacy (CSAL) [
We propose a systematic simulation framework, 3DG, illustrated in Figure 1. This framework begins by structuring the initial learning performance data, sourcing from real-world learnerITS interactions, into a three-dimensional tensor by dimensions of learners, questions, and attempts. As depicted in the sparse cube space in Figure 1, filled cubes represent recorded values of learning performance, while transparent cubes indicate missing values. Tensor completion (based on tensor factorization) is then utilized, converting the sparse tensor to a densified one. The densified tensor provides invaluable information in identifying various learning patterns, which aids in dividing the tensor into sub-tensors by categorizing distinct learning patterns. Subsequently, generative models are harnessed to simulate additional data samples for enriching the original dataset based on each specific learning pattern. The entire operation is encapsulated for scalable simulation sampling and ultimately ofers a comprehensive dataset incorporating both imputed and augmented data. This framework was developed to address RQ 1. More detailed methods within this framework are described in the following subsections.
The three-dimensional tensor , representing the learning process, is defined as ∈ × × , where the = (1, 2, 3, · · · , ) is the maximum number of learners, = (1, 2, 3, · · · , ) the maximum number of questions, and = (1, 2, 3, · · · , ) the maximum number of attempts. Each element of indicates the performance variable of learner on question at the attempt . For instance, in the CSAL AutoTutor context, a binary variable = {0, 1} is used, where 1 signifies a correct answer and 0 denotes an incorrect one. We model the tenor as a factorization of two lower dimensional components: 1) a learner latent matrix of size × ( represents the set of latent features in tensor factorization), which captures learner-related latent features matrix/space (such as abilities and learning-related features); and 2) a latent tensor of size × × , representing the learner knowledge in terms of latent features during question attempts. The approximated tensor ˆ is obtained by the following formula: ˆ ≈ × (1) where can be interpreted as the latent feature space encapsulating learner-related efects, reflecting characteristics such as individual abilities/features and learning preferences. On the other hand, the tensor represents the interaction between attempts and question-related (knowledge acquisition) efects, adapting to various learner features.
To answer RQ 2, we used two generative models, GAN (Generative Adversarial Network) and GPT (Generative Pre-trained Transformer), to facilitate scalable simulations that are tailored to individual learning patterns. According to [32, 33], GAN model is uniquely structured with a dual-network architecture comprising a generator and a discriminator. This architecture enables GAN model to excel in generating high-quality synthetic data. In comparison, GPT model is distinguished by its use of transformer architecture, which empowers it to generate data that is not only contextually relevant but also maintains a high degree of coherence [34, 35].
Before initiating the simulations, we employ a clustering algorithm (i.e., K-means++) to categorize individual learning patterns based on similarities in learners’ performance. The learners-attempts matrix slice extracted from the ˆ , encapsulates the probability-based knowledge states associated with the performance on the th question , for all learners over attempts. In our analysis, we employ the “power law learning curve", a model widely recognized in educational and training research [36, 37, 38], to fit the learning performance with increasing attempts. In the power-law formula = , the represents the learning performance, quantified as the probability of producing correct answers, and is the number of opportunities to practice a skill or attempt. The parameter indicates the measurement of the learner’s initial ability or prior knowledge, and represents the learning rate at which the learner acquires knowledge through practice. We employ K-means++ [39, 40] to cluster the distribution of two model parameters ( and ), which assists in identifying distinct individual learning patterns.
As illustrated in Figure 2, the architecture of Generative Adversarial Network (GAN) consists of two distinct neural networks: the Generator and the Discriminator. The Generator, often a type of neural network like a convolutional neural network (CNN), is designed to create synthetic data samples. It is denoted as (· ). The Discriminator, typically another neural network which can also be a CNN (though its structure may vary based on the specific application), is tasked with evaluating whether the data samples are real (authentic data) or fabricated by the Generator. It is denoted as (· ). In the process (Figure 2), the Generator starts with a noise sample, usually drawn from a Gaussian distribution, which has dimensions compatible with the original data distribution. This noise sample serves as the initial input for the Generator, resulting in . Both this and the are then fed into the Discriminator (· ). The Discriminator’s role is to discern whether each sample is real or simulated. Concurrently, the Generator (· ) is trained to progressively reduce the diference between the distributions of the real and simulated data through iterative tuning.
Considering the limitations of using purely numerical values for interoperability and the enhanced semantic understanding that detailed descriptions provide, we have developed a mixedbased prompt approach for GPT-4 (illustrated in Figure 3). The prompt strategy integrates original matrix data with interpretive text, thereby enriching the context and interpretability of the data. Additionally, it incorporates the Chain-of-Thought (CoT) prompting technique [41], which involves appending guiding phrases such as ’Let’s think step by step’ at the end of the prompt to facilitate a more structured analytical process. Specifically, the constructed prompt includes comprehensive elements such as the reading material being analyzed, detailed information about the questions (including their answers), and the learners-attempts matrix data, complete with descriptive information about both its format and entries. Subsequently, a simulation request prompts GPT-4 to integrate the numerical and textual data in a coherent and insightful manner, ultimately driving the execution of a simulation. During the optimization process, these prompts are iteratively refined and adjusted to eficiently yield results that align with our specified objectives.
As illustrated in Table 1, the original dataset exhibits sparsity levels ranging from 80% to 85%
(as determined by calculating the proportion of missing values to the total number of entries).
By iteratively tuning the latent feature range [
These findings suggest that tensor completion (based on tensor factorization) can eficiently impute missing values in the original sparse performance data, notably for unexplored questions and attempts. This enhancement is crucial for facilitating more comprehensive analysis and modeling in Intelligent Tutoring Systems (ITSs). The latent features, closely associated with learner-specific characteristics during the learning process, are captured with nuanced detail, particularly in the context of reading comprehension. Further research is imperative to fully understand the underlying physical essence of these latent features.
The distributions of parameters and are illustrated in Figure 4 and Figure 5, respectively. These figures visualize the parameter distributions from a example cluster data set with an original size of 20, and exhibit simulations in increments of 1000, with total sizes ranging from 1000 to 20000.
Figure 4 demonstrates the distributions of parameters , which is used to represent the learner’s initial ability or prior knowledge. Figure 4a shows the distribution of the parameter obtained by GAN simulation. As the sample size increases, the range of parameter from the simulation sample mostly falls within the original range of parameter , although it exhibits a longer tail distribution extending beyond the original maximum value of parameter . The distribution of the parameter obtained by GPT-4 simulation is illustrated in Figure 4b. Unlike those obtained from GAN simulation, the range of parameter values here extends beyond the original range, which is particularly evident as the simulated sample size increases. This (a) Distribution of parameter by GAN simulation.
(b) Distribution of parameter by GPT-4 simulation. suggests that the initial learning ability in GPT-4 simulated samples exhibits more variability and divergence from the original data compared to those from GAN simulation.
Then, Figure 5 demonstrates the distributions of parameters as derived from both GAN and GPT-4 simulations. The parameter represents the learning rate, which reflects how quickly a learner acquires knowledge through practice. The GAN simulation produces a narrower range of parameter values, especially in terms of maximum and minimum values, when compared to the original dataset, as depicted in Figure 5a. With increasing sample size, this range generally maintains a consistent pattern. On the other hand, the GPT-4 simulation, as shown in Figure 5b, demonstrates a broader range for parameter , extending beyond the original scope. This contrast suggests that GPT-4 simulation may capture a wider variability in learning rates compared to GAN simulation.
(a) Distribution of parameter by GAN simulation.
(b) Distribution of parameter by GPT-4 simulation.
This paper proposed the 3DG systematic simulation framework based on generative models (particularly GAN and GPT) to address data sparsity challenges in learning performance data within intelligent tutoring systems (ITS). The framework involves representing learner data from problem-solving step attempts as a three-dimensional tensor with the axes of learners, questions, and attempts. Tensor completion, based on tensor factorization, is then utilized to impute missing performance data entries, generating a dense tensor. Such imputation computation leverages the similarities in learner performance across various questions and attempts, capturing the sequential and temporal dynamics of learning [42, 43]. We have demonstrated the integration of generative models, including GAN and GPT-4 for creating scalable, individualized learning simulations aimed at enhancing learner models for personalized instruction. Our comparative analysis reveals that GAN surpasses GPT-4 in terms of reliability for scalable simulations.
Overall, the GAN simulations demonstrate a narrower and more consistent range of values for parameters and , indicating higher reliability for scalable simulations compared to the broader value range exhibited by the GPT-4 simulations. The mechanism for the GPT4 simulations, refined through iterative optimization of GPT-4 prompts, involves selecting random values from a flat array of the original data. These values are then adjusted to match base probabilities, preserving the overall data distribution while facilitating the creation of an expanded dataset. Although valuable in computational simulations, this method generally underperforms in numerical computing compared to deep learning models, as demonstrated by the GAN’s performance in this study.
Our findings shed light on the potential use of GPT-4 in simulating learner performance represented through numerical values in future research. Firstly, employing mixed-based prompts improves interoperability with numerical data, thus enhancing the eficiency of subsequent modeling and simulation computations. Secondly, the Chain-of-Thought (CoT) prompting technique delineates the steps for the simulation task, efectively directing GPT-4 in its reasoning process. This includes a structured approach comprising: Understanding the Existing Matrix, Distribution Analysis, Clustering Information, and Simulation Process. Thirdly, the computational power of GPT-4 in modeling and simulation is attributed to its capabilities in self-search, self-programming, and self-computing, all of which are facilitated by prompt engineering. This significantly enhances its utility in data analysis and modeling for future research endeavors. However, integrating GPT-4 with numerical computation presents fundamental challenges, as we discuss in the following section.
The capability of GPT-4 in performing deep learning tasks involving numerical computations remains insuficient, primarily due to the intrinsic limitations of large language models and platform contraints. Future research could productively explore the integration of GAN with GPT models, aiming to improve their interoperability and computational capabilities. Furthermore, the degree of sparsity in the original performance data, particularly when formatted as a tensor, significantly impacts the performance of generative models. Therefore, investigating the sensitivity and robustness of tensor completion methods in response to diferent levels of data sparsity presents an important avenue for future studies. Such investigations are crucial for better integrating large language models within Intelligent Tutoring Systems (ITSs), potentially leading to more refined and efective educational tools.
We extend our sincere gratitude to Prof. Philip I. Pavlik Jr. from the University of Memphis and Prof. Shaghayegh Sahebi from the University at Albany - SUNY for their expert guidance on tensor factorization method. P. Pavlik, et al., Clustering the learning patterns of adults with low literacy skills interacting with an intelligent tutoring system., Grantee Submission (2018). [32] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, Advances in neural information processing systems 27 (2014). [33] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, Communications of the ACM 63 (2020) 139–144. [34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). [35] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., Improving language understanding by generative pre-training (2018). [36] A. Newell, P. S. Rosenbloom, Mechanisms of skill acquisition and the law of practice., Technical Report, CARNEGIE-MELLON UNIV PITTSBURGH PA DEPT OF COMPUTER SCIENCE, 1980. [37] H. Cen, K. Koedinger, B. Junker, Learning factors analysis–a general method for cognitive model evaluation and improvement, in: International conference on intelligent tutoring systems, Springer, 2006, pp. 164–175. [38] R. DeKeyser, Skill acquisition theory, in: Theories in second language acquisition,
Routledge, 2020, pp. 83–104. [39] D. Arthur, S. Vassilvitskii, K-means++ the advantages of careful seeding, in: Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, 2007, pp. 1027–1035. [40] B. Bahmani, B. Moseley, A. Vattani, R. Kumar, S. Vassilvitskii, Scalable k-means++, arXiv preprint arXiv:1203.6402 (2012). [41] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chainof-thought prompting elicits reasoning in large language models, Advances in Neural Information Processing Systems 35 (2022) 24824–24837. [42] C. M. Conway, M. H. Christiansen, Sequential learning in non-human primates, Trends in cognitive sciences 5 (2001) 539–546. [43] C. M. Conway, Sequential Learning, Springer US, Boston, MA, 2012, pp. 3047–3050.