Computer programming courses have long been a significant challenge for students entering the field of information technology. This is because students must express their needs using computerunderstandable terms, logic, and thinking, often encountering obstacles in the process (Feurzeig et al., 2011). This challenge is considered a global issue, as both introductory and advanced programming language courses face high dropout rates, creating substantial pressure on students and teachers who may have high expectations for themselves (Luxton,2016). Many students give up when confronted with multiple syntax and logic errors, indicating a potential lack of problem-solving skills and perseverance (Cheah,2020).
In light of these challenges, this study primarily explored students' behavior records during the debugging process of coding, as errors represent obstacles and setbacks. Whether students can progress through these setbacks will be a key factor in their improvement and success. The study was designed to analyze and cluster students' debugging behavior data in programming courses using the Learning Management System. The research was expected to address three main research questions:
• RQ1. Can errors made by students in coding be differentiated into distinct clusters? • RQ2. Is there a difference in the number of successful program runs (Success_run) among students in different programming error clusters? • RQ3. Is there a difference in course learning scores (Score) among students in different programming error clusters?
Educational Data Mining (EDM) and Educational Process Mining (EPM) are data science approaches to analyze various Learning Management Systems (LMS) (Bogarín,Cerezo& Romero,2018). This data mining approach is an important part of learning analytics.
Particularly with the rapid expansion of Massive Open Online Courses (MOOCs), the interaction between online educational resources and learners is stored in extensive databases, creating educational big data for interpretation by educators (Ruipérez et al., 2022).
In numerous studies on learning analytics, there is often an exploration of dropout rates (failure rates) on online education platforms (Qian,2022). Additionally, researchers have analyzed differences in learning behaviors on a platform (Tong & Zhan, 2023), ultimately aiming to predict students' learning achievements (grades) (Li,Du & Yu,2023). These studies frequently involve classifying learners based on their learning preferences, allowing them to adapt their learning experiences according to the platform's diverse learning paths, recommendation systems, personalized learning strategies, and more.
The learning of programming languages involves various aspects of learning, such as problemsolving skills, computational thinking, and syntax comprehension (Nouri, Zhang, Mannila & Norén, 2020). Importantly, students need to sustain high motivation and genuine engagement in coding to make progress in learning how to program (Silva & Silveira, 2020). From the perspective of beginners, the difficulty lies in the inability to break down large problems into smaller ones. When hearing some specific terms (such as recursion, arrays, etc.), students may understand how they work, but struggle to translate that understanding into actual code (Lister et al., 2004). In this context, it is crucial to encourage students to engage in trial and error, as the experiences gained from mistakes help students engage in self-reflection and can even stimulate strong motivation to find satisfactory answers and iterate through the trial-and-error process (Sicora , 2019).
Since the sixteenth century, people have sought solutions to problems by encountering stimuli that lead to subsequent actions, which may involve continuous trial and error until success or, in some cases, giving up, which results in task failure (Boswell , 1947). Learning is inherently challenging, with difficulties progressing from shallow to deep. For example, students may encounter "Errors" while programming, which could contribute to exacerbating the failure rate in programming courses (Porter, Guzdial, McDowell & Simon, 2013) but can also serve as the driving force for problem solving (Noh & Lee, 2020).
Programming education is often considered to have a high entry threshold, possibly due to insufficient problem-solving skills among students or ineffective use of learning materials (Cheah, 2020). Therefore, in programming courses, continuous trial and error by students is viewed as a positive behavioral performance. This practice signifies students' continuous attempts, whether in syntax or logical reasoning, until they produce results matching their expectations (Ye et al., 2022). Efforts in trial and error also help students enhance their self-efficacy, as they gain confidence in how to deal with errors and develop problem-solving skills (Ahn , Mao , Sung & Black, 2017).
Hierarchical clustering is an unsupervised algorithm that organizes data points into a tree-like structure on a two-dimensional plane. It groups data points and produces a hierarchical structure based on the differences between data points (Alpaydin , 2020). Agglomerative hierarchical clustering is a bottom-up hierarchical clustering method that visualizes the hierarchical structure and underlying data clustering structure (Liu, Xu, Zeng & Ren, 2021). It is also a user-friendly and popular clustering algorithm.
The agglomerative hierarchical clustering process first assigns each object to its own cluster. It then uses distance or similarity measures (e.g. Euclidean distance for quantitative data, Manhattan distance for ordered but not necessarily quantitative data) or more complex methods (e.g. unweighted with arithmetic mean Pair group method (UPGMA) (Oyelade , 2019).
Based on N samples, there are initially N clusters, each cluster containing one sample. Iteratively merges the two closest clusters based on the chosen distance or similarity measure until the number of clusters is reduced to 1 or reaches a user-specified number (Cichosz , 2014).
In each successive iteration, the algorithm merges the closest pair of clusters based on the similarity criterion of features between data points until all data are in one cluster (Sasirekha & Baby, 2013).
Hierarchical clustering helps analyze educational big data, helping researchers identify different student learning styles, achievements, and behaviors, as well as assess individual engagement levels (Hung, Liu, Liang & Su, 2020 ; Trivedi & Patel ,2020 ; Yang, Chen, Flanagan & Ogata, 2022).
This research incorporates the "Learning Behavior and Learning Strategies" dataset collected by Lu et al.( 2022). This dataset predominantly consists of various actions recorded on a Learning Management System (LMS) as students engaged in learning programming. It encompasses a range of data points such as the number of errors generated, instances of code copying, frequency of code execution, and academic grades. The primary focus of the dataset is to capture the learning behaviors and strategies of students while they undertake programming tasks within the LMS environment (Lua et al., 2022).
For the clustering analysis, this study focused on the "viscode.csv" dataset, specifically using the "IndentationError," "NameError," "SyntaxError," and "TypeError" fields. These four types of errors were defined as indicators of programming trial-and-error, representing the problems and difficulties students encountered while running their code. The "Viscode-success_run" and "Score" fields were used as indicators to validate the effectiveness of clustering (Table 1), serving as the basis for addressing Research Questions 2 and 3 in the study.
This study compared actions taken by 452 students in a programming course using an integrated development environment, as recorded by the learning management system in the viscode.csv dataset. The data, preprocessed and de-identified, includes distinct class fields (a-i), fields for interactions with the integrated development environment, debugging attempts, execution counts, success running counts, and grades. After clustering, a new PseudoID field was introduced to represent a unique identifier for each student, also serving as an index after clustering (Ogata et al., 2017). Additionally, a Cluster field was added for conducting inter-group analysis of variances and comparing the correlation between program execution success and learning grades across clusters.
This research employs the agglomerative hierarchical clustering method from the Python sklearn.cluster module for systematic trial-and-error clustering. To avoid the skewing of results by any single feature, normalization is performed before clustering. This is critical as it prevents any one data column from exerting undue influence on the clustering outcome and maintains the robustness of the algorithm against outliers, which could be seen as noise.
Following this preparatory step, the clustering process commences. The distance metric adopted is the "Ward" method, designed to minimize the total within-cluster variance. Essentially, at each step, Ward's method selects two clusters to merge in a way that results in the least possible increase in total variance, thus preserving high similarity within the clusters.
Moreover, to ensure a balanced distribution of clusters, silhouette scores are utilized to assess the quality of clustering across different numbers of clusters, ranging from 2 to 9. Referencing Table 2, the study identifies 2 as the optimal number of clusters and proceeds with further data analysis using this configuration.
This study utilized the scipy.cluster.hierarchy library in Python to perform clustering analysis. Through this library, hierarchical clustering results were computed and visualized, as shown in Fig. 1. The chart reveals two distinct clusters with noticeable distances between them. Cluster 0 comprises 414 student records, Cluster 1 includes 38 student records. This study further applied Principal Component Analysis (PCA) to reduce the dimensions of data consisting of student IDs and their trial-and-error behaviors. By transforming the data into a two-dimensional chart, we made it straightforward to compare these behaviors against student performance.
In this study, the Python library seaborn was utilized to create a heatmap (Fig. 2), which presents the average number of trial-and-error attempts by students across different clusters. The heatmap clearly shows that students in Cluster 1 had a higher frequency of programming errors, such as IndentationError, NameError, SyntaxError, and TypeError, compared to those in Cluster 0. Notably, there are significant differences in the occurrences of NameError, SyntaxError, and TypeError between Clusters 1 and 0. Cluster 1 is characterized as the "Frequent Trial-and-Error Group," while Cluster 0 is referred to as the "Regular Trial-and-Error Group" in this study.
To address research question 2, the study analyzed the "Success_run" field, revealing through Figure 3 that students in the frequent trial-and-error cluster had higher average successful program runs compared to those in the infrequent trial-and-error cluster. This pattern suggests that students who frequently encountered system errors in the integrated development environment were more persistent, leading to more successful code executions.
The study further employed an independent sample T-test, using IBM SPSS, to investigate statistical differences between the clusters in terms of successful program runs. The findings showed a statistically significant difference (t = -5.49, p < .05), where the frequent trial-and-error cluster outperformed the regular trial-and-error cluster in successful program executions. This discrepancy likely arises from the frequent trial-and-error students' resilience and continuous engagement with problem-solving and coding adjustments, in contrast to students in the regular trial-and-error cluster who may have experienced reduced motivation and task completion rates after facing setbacks. Consequently, the frequent trial-and-error cluster exhibited a significantly higher number of successful program operations compared to the regular trial-and-error cluster, highlighting their effective learning and problem-solving approach.
To explore Research Question 3, the study performed a descriptive analysis of the "score" field across clusters differentiated by trial-and-error frequency. Figures 2 and 4 illustrate that clusters characterized by frequent trial-and-error tend to have higher scores than those with regular trial-anderror patterns. The box plot indicates that scores for the frequent trial-and-error group are predominantly ranged between 80 and 90 points. In contrast, the regular trial-and-error group not only exhibited wider score fluctuations (70-85 points) but also presented numerous outliers significantly below the average, with some scores approaching zero. This suggests potential dropout behaviors among certain students in the regular trial-and-error group.
The independent samples T-test statistics revealed a significant disparity in course performance between the two trial-and-error groups (t = -2.69, p < .05) as shown in Table 4. This finding resonates with the insights from RQ1 and Figure 1.
The study segmented program-based learning students by their trial-and-error occurrences, noting that a smaller contingent of students (38 in total) fell into the frequent trial-and-error group. Not with standing their group size, these students not only surpassed the regular trial-and-error group in the number of successful program runs but also outscored them. This underscores the significance of persistent trial-and-error efforts in learning; it is imperative for learners to persistently experiment and overcome challenges without capitulation to achieve success (Dong et al., 2019).
This study investigated the error patterns students demonstrated while debugging programs, categorizing them into two distinct groups: Cluster 0, the regular trial-and-error cluster, and Cluster 1, the frequent trial-and-error cluster. The analysis revealed that students in the frequent trial-and-error cluster not only had higher average course success rates and final scores compared to their counterparts in the regular trial-and-error cluster but also performed significantly better. These findings suggest that the debugging behaviors of different student groups can affect their learning outcomes. Educators should be cognizant of these differences and strategize appropriate responses to students' programming challenges. Moreover, when the frequency of trial-and-error attempts begins to wane, interventions such as encouragement or providing cues might be necessary to bolster students' motivation and align them with their peers (Xu, Yang, Liu & Jin, 2023).
Nonetheless, it is crucial to acknowledge the limitations of the dataset and context of this study. Given that all participants were novices in programming and enrolled in the same course, the breadth of their acquired knowledge may be limited. This research advocates for the use of more varied datasets and extended observational periods. There is also considerable variation in the amount of time different students dedicate to studying programming. While some engaged with the course material for over seven hours, others may invested mere minutes. As this study lacks precise data on students' study timings, future research could employ time series analysis to discern behavioral shifts among clusters and predict how debugging frequency might influence future learning. Such an approach could yield more precise learning recommendations and enable educators to tailor their focus on the effects of trialand-error activities on academic achievement.
This study is supported in part by the National Science and Technology Council in the Republic of China under contract numbers NSTC 112-2628-H-003-007-.