The value-added applications of ChatGPT occur in many fields. Cooperation with ChatGPT has gradually become inevitable. This study aims to explore the potential of ChatGPT in the field of learning analytics, with a specific focus on predicting risk students while tackling prevalent challenges in learning analytics. Traditionally, learning analytics classification tasks have relied on machine learning models, leading to issues related to model interpretability and tailor learning suggestion generation. By utilizing the LBLS467 learning behavior dataset, experimental findings with ChatGPT-4 reveal its potential as a fundamental and accessible tool. While occasional performance variations are noted, ChatGPT holds promise as an alternative approach for basic at-risk student prediction within learning analytics. This study paves the way for further exploration of ChatGPT's potential in enhancing student support mechanisms and improving educational outcomes.
With the popularization of technology in recent decades, on-line learning platform such as Google
classroom had become increasingly popular. During the Covid-19 pandemic, the shift to remote
teaching had expanded the utilization of on-line learning environment in many aspects. An
advantage of online learning environments is their ability to comprehensively record students'
study habits, offering valuable data for learning analytics (LA). LA has recently become a necessity
in the educational environment, for example, research has demonstrated how LA works on two
Japanese universities to support education and learning [
Typically, LA for predicting at-risk students is carried out through machine learning (ML)
models or statistic methods. However, applying these approaches for risk student prediction may
require certain level of advanced knowledge in the field. Nonetheless, technology advancements
have expanded the range of options available to benefit needed users. Among various
technologies, Artificial intelligence (AI) had become one of the latest tools that can efficiently and
effectively help humans to deal with a variety of tasks. Among all the AI, ChatGPT is considered
as one of the most popular and powerful, one known for its successful application to a wide range
of domains, e.g., healthcare, translation, etc. [
To confirm the role and value of ChatGPT in LA is the core task of this study. In this study, experiments will be conducted using a set of educational datasets referred to as LBLS467 (Learning Behavior Learning Strategy 467). This dataset will be processed by ChatGPT-4 to make prediction on who are the risk students. Then, their performance will be evaluated and compared by calculate the accuracy. The study aims to evaluate ChatGPT's capability in risk student prediction, addressing the following research questions: • RQ1: How do ChatGPT make prediction? • RQ2: How accurate is the prediction result from ChatGPT?
In the realm of education, previous studies have mainly focused on the impact of ChatGPT on
student’s learning behavior, academic integrity concerns [
Nevertheless, few studies have explored ChatGPT's application in LA, possibly due to its textbased chatbot nature, which is better suited for tasks like text generation and summarization than numerical data analysis. Additionally, instructors and researchers may prioritize addressing AI misuse which is a more immediate concern over LA. Still, LA serves as a valuable long-term resource for course strategy that is worth investing in. It’s also crucial to acknowledge the potential of applying ChatGPT to various domains including LA.
Despite the shortage of studies on the implementation of ChatGPT in LA, some existing research
has involved ChatGPT in LA for various purposes. Research has pointed out the need of
interpreting the internals of predictive analytics and provide tailored advice according to the
analytics result to at-risk students [
Previous study has been conducted on how ChatGPT can become a student-driven education
technology and how it can possible be apply to LA [
LA and risk student prediction in the past were mainly conducted using machine learning
models or statistical analysis [
Machine learning models can assistant human with a variety of task, however, one common issue
of it is the lack of transparency and interpretability, hence, highlight the importance of
Explainable AI such as SHAP which aims to offer explanations for the predictive methods of
machine learning models [
In the study conducted in 2023, ChatGPT was employed to predict stock market movements
using news headlines, which is text-based data, and the finding revealed that ChatGPT actually
outperformed traditional sentiment analysis methods [
This dataset gathered the learning data from nine of programming classes with total 467 students
from 2020 to 2022. The participants were all university students from non-computer
sciencerelated departments [
The process of data preparation before getting into the experiment is depicted in Fig.1. Note that we define at risk students as students whose scores are lower than the Q1 score in their class. The Q1 score of each class is presented in Table1. Following data preparation, data frames were extracted base on the require number of features and data size and then split into training set and testing sets, this process is presented in Fig.2. Colum “Score” was not included in both the train test data frames because when making actual risk student prediction, the score of students is unknown.
The first stage feature extraction here was conducted by human researchers for the purpose to test the performance of ChatGPT handling data frames with different number of features. Features with more 0 values indicates fewer students are contributing data to this feature. Therefore, we assume features with lots of 0 value will have less effect on the prediction result of risk student. The feature extraction was conducted by setting up thresholds. Dropping features with more than x% of the value in the feature are 0.
The three feature numbers (9, 45, 80) were chosen for the below reasons. 9: Is the minimum number of features can be obtained with this threshold. Features that have more than 0.5% of their values as zeros were dropped. 80: Is the maximum number of features in this data frame, which included 26 features in the ‘br.csv’ file, 51 features in the ‘viscode.csv’ file, ‘TotalTime’ and ‘Risky’ column we appended and the ‘class’ column. The ’userid’ and ‘score’ columns was excluded from the training dataset as they are not relevant student’s risky status and ‘score’ is only used for labeling purpose. 45: Is approximately the number of features in between 9 and 80. If more than 81.30% of the values in a feature are zeros, that feature will be dropped. This middle point was chosen for the purpose to better demonstrate the change of accuracy among the numbers of feature.
ChatGPT-4 website was chosen for conduct the experiment because it accepts bigger amount of input data compared with previous ChatGPT versions and ChatGPT-4 had released a new function for data analysis by just upload the files. In addition, using website can keep the conversation with ChatGPT in a more organized way. Detail process of the experiment and prompt is showed in Fig.3. The performance of ChatGPT doing risk student prediction is evaluated by accuracy with the below equation. With True Positives (TP) are instances when ChatGPT correctly predict a risk student as “Yes”; True Negatives (TN) are instances when ChatGPT correctly predict a non-risk student as “No”; False Positives (FP) are instances when ChatGPT incorrectly predicted a nonrisky student as “Yes”; False Negatives (FN) are instances when ChatGPT incorrectly predicted a risky student as “No”.
("# % "&) Accuracy = ("# % (# % "& %(& ) (1)
Upon received the below prompt and the uploaded files, ChatGPT will go through the process depicted in Fig.4. In this process, depending on the prompt, predictions would be conducted with or without the use of ML model, as illustrated in Fig. 5 and Fig. 6 respectively.
ChatGPT really insist that the prediction should be conducted using ML model, if didn’t specified in the prompt, it will always apply machine learning model, with the Random Forest Classifier being the preferred choice in over 95% of cases. As ChatGPT explained, the choice of the Random Forest Classifier is based on its popularity and its robustness against overfitting, as well as its ability to effectively handle a diverse feature type. Occasionally, ChatGPT would choose other ML models such as Logistic Regression model for its binary classify characteristic or Gradient Boosting Classifier because it can handle a mix of continuous and categorical variables.
On the other hand, with the limitation of not to use ML model, ChatGPT would use heuristic approach, simple statistical methods or logical reasoning instead to complete the prediction. Two method it applies frequently are comparing the mean value of certain features between risky and non-risky student or analyze the correlation between each features and student’s risky status. In addition, ChatGPT will also take the distribution of risky and non-risky students in the training set into consideration. Mentioning there are lots of student being label as risky or non-risky.
The prediction results presented in Table 2 focused on how the change in number of features and data size would affect the prediction accuracy. Because of the different characteristic of data frames in the LBLS467 dataset, the input data frames in Table 2 only include learning behavior data. The inclusion of both learning behavior and learning strategy data frames are presented in Table 3. The prediction results in both Table 2and 3 were analyzed along with the column name description file which contain the description for each feature. Furthermore, the data presented in both Tables were obtained through a repetitive process of executing Figures 2 and 3, each repeated five times, with the average accuracy recorded.
A discernible pattern from both Table 2 and is that ML approaches in general outperform NonML approaches, with the accuracy of the ML approach surpassing the Non-ML approach in all instances. In addition, the t-test results reveal statistically significant differences in mean accuracy between the ML and non-ML approaches across the 9, 45, and 80 features data frames. This outcome suggests that ChatGPT might not be good at logical reasoning and use heuristic approach or simple statistical methods to accurately make prediction.
Furthermore, the mean accuracy with ML approach across the data frames with 9, 45, and 80 features in Table 2 is relatively similar, with each hovering around the 75% mark. In contrast, the mean accuracy with Non-ML approach gradually increases as the number of features increases. However, there’s no clear pattern regarding how the data size would affect the accuracy.
When the learning strategy data frame is included, all the average accuracy values in Table 3 surpasses the mean accuracy in Table 2. This indicates that adding the learning strategy information can help improve the prediction accuracy, especially with the non-ML approach.
However, it’s crucial to note that ChatGPT's performance in both ML and Non-ML approaches can be variability, occasionally resulting in either exceptionally high or low accuracy especially
with Non-ML approaches. This variability may be attributed to the selection of different ML models or heuristic approaches.
Despite its occasional performance fluctuations, ChatGPT proves capable of serving as a basic and convenient tool for LA and fundamental risk student prediction. It offers flexibility by enabling the application of both ML and non-ML methods for prediction, which opens up the possibility for further research to explore different kinds of input data for LA. With traditional ML approach, ChatGPT typically achieves accuracy levels of around 70-80%. It simplifies the process by handling data processing, model training, and code execution automatically. While manual ML model training may yield higher and more stable accuracy, it demands time and expertise. When course instructors find ChatGPT's predictive performance acceptable, it becomes a convenient option for implementing a learning risk classifier.
Nevertheless, it is important to acknowledge certain limitation in the research. Firstly, the LBLS467 dataset was collected form programming course, characterized by a substantial presence of numerical coding learning records, which can significant differ from other subjects. Moreover, the prompts being use for both ML and Non-ML approaches are almost the same. Tailoring prompts for different approaches might provide a more precise description of the tasks and potentially lead to improved performance. Further research endeavors could explore the applicability of ChatGPT to predict at-risk students using data from different subjects, enhance accuracy level and performance stability through prompt modification, and conduct experiments with text-based data. This study is supported in part by the National Science and Technology Council of Taiwan under contract numbers NSTC 112-2410-H-004 -063 – and NSTC 112 - 2636 - H - 008 - 005 -.