The last decade has seen an increase in the use of technology in educational environments with tools such as computers, electronic boards, virtual environments and learning management systems. Consequently, the data collected during the learning process have grown exponentially, along with their potential to generate knowledge about possible factors for academic success [1,2]. In this sense, the information generated can be used to guide teachers, institutions, and students in making decisions related to learning, teaching, and educational administration [3,4]. Data analysis in the educational area is known as learning analytics and is considered the future of education, especially in the context of higher education [3]. Learning analytics is based on the principles of traditional educational research, but it leverages innovations such as the collection of new forms of digital data and uses advanced computational analysis techniques from data science and artificial intelligence [5,6]. In the context of computer programming courses, learning analytics has been used to achieve different objectives. Several efforts seek to build models to automatically detect students at risk of failing a course [7,8]. Others propose to track course progress through visualizations of statistics and metrics representing student behaviors [9], and to generate personalized feedback to students in programming courses [10].
However, Ferguson [11] has identified that research in learning analytics lacks methods that use a wide range of datasets. This coincides with one of the challenges presented by Schmitz et al. [12], who find a lack of educational investigations that considers diversity of learners. This challenge is due to the fact that advances in education facilitates the inclusion of relatively new or different groups of students than the traditional ones [12]. Thus, in this work we propose to answer the following research question: What are the relationships between data generated from students' interactions with an educational platform and their academic performance in a computer programming course when considering different groups of students in the dataset? Thus, the objective of this work is to determine the existing relationships between measurements and metrics from the learning processes of computer programming students at the National University of Colombia and their academic performance. The methodology used is descriptive, correlational, quantitative and non-experimental.
One of the areas where learning analytics has gained importance is computer science [7]. In the context of computer programming courses, one of the most frequent objectives is the prediction of student behavior. For example, Azcona et al. [7] propose a model to automatically detect students at risk of failing a Python computer programming course and based on the predictions, provide personalized feedback. Other investigations seek to generate mechanisms for effective course monitoring with visualizations of statistics and metrics that represent the students' behavior. For example, Shen et al. [9], propose a heat map that visualizes the intensity of student access to educational resources and activities in an introductory Python MOOC. Additionally, through social network analysis, examines similarities and differences in access patterns. Another example is the set of visualizations proposed by Leony et al. [13] in a C programming course, aiming to inform teachers about students' emotions based on the interaction of students with an educational technology tool. Finally, other researches aim to generate personalized feedback to students. For example, Lu et al. [10] apply learning analytics to identify students who need immediate intervention in a Python MOOC, and based on this information, teachers build adaptive learning guides that are applied in an experimental group and compared with a control group.
The learning analytics' methodology of this work has a quantitative approach with a descriptive and correlational scope. The type of research design is non-experimental, since the data to be used have been collected without modifying the variables of the context. The methodology consists of 3 sequential phases: 1) data preparation, 2) data transformation, and 3) data analysis.
Data collection. First, the location and format of the available data is identified. Then, the data of the students' learning process is gathered through a data management and analysis tool.
A consolidation process is necessary, since the raw data is often disaggregated. Subsequently, the most appropriate data structures are identified for the organization and manipulation of the consolidated data.
Data cleaning. The variables present in the consolidated dataset which gives useful information about the learning process are identified. Also, the variables that are not related to the objective of the study or those that have data quality problems are discarded.
Identification of measurements. Firstly, a literature review of measurements and metrics used in educational research in computer programming courses should be performed. Subsequently, identify which of the measurements found in the literature are present in the dataset and which can be used to build suitable metrics.
From the results found on metrics in the literature, the equations needed for the estimation of metrics are established.
The equations proposed in the metrics design stage are applied in the data management tool and the values obtained are stored for later analysis.
Exploratory data analysis. Based on the previous measurements and metrics, an exploratory univariate analysis is performed to identify behaviors and trends. For this, descriptive statistics (arithmetic mean, dispersion measures, skewness, etc.) and visualizations (box plots, histograms, etc.) are used.
The relationships between metrics and measurements are identified through correlations or regression analysis. Additionally, supervised or unsupervised machine learning techniques can be applied if the objective of the work is to obtain classifications, regressions or groupings of data.
The results obtained are compared with similar findings of other studies. Then, the behaviors found are interpreted and the implications in the context of the research are specified.
Conclusions. The summary of the findings is presented along with its limitations and the possible threats to validity. Finally, future work is proposed based on the results of the study.
Results obtained by applying the proposed methodology to the 2-year (between 2019 and 2020) history of the Computer Programming course at the National University of Colombia are presented below.
Data collection. Data were collected from the interaction of students with the educational platform UNCode used for the automatic evaluation of programming exercises of class activities [14]. UNCode allows students to submit multiple program attempts (source code or Jupyter notebooks) built to solve programming tasks. For each solution attempt, the platform stores the program file, date and time of the submission. In addition, it offers automatic feedback through verdicts related to errors in the syntax, semantics and efficiency of the program. Also, it offers a numerical grading, depending on the test cases the designed program solved. Within UNCode there is a set of learners' support tools, such as: syntax highlighting, code auto-completion, linter (suggestions of good programming practices), visualization of code execution, custom tests and reports of grades [15]. The raw data was stored in a MongoDB database and it was exported to CVS (Comma Separated Values) files. The programming language Python was used, with its specialized libraries for data analysis and visualization (pandas, matplotlib, seaborn). Data consolidation. The exported files in CSV format for each course were stored in a shared folder in Google Drive. The data was organized in separate folders per course where there are files with the list of student usernames, the records of the submissions made per activity, and the final grades. In addition, each course contains a folder with the files submitted, organized by student and by activity.
The stored data includes all the courses of the university that use UNCode; for this reason, it contains data from courses other than computer programming. Therefore, a screening process is made by selecting only the programming courses (22 groups). Then, a filtering is performed within each course for activities and students with insufficient number of submissions. The post-filtering dataset contains 1352 activities and 735 students.
Identification of measurements. Table 1 specifies the 15 measurements that are considered of interest in the research, which are classified into four categories: 1) data related to the attempts of solution of programming assignments made by each student, 2) the verdicts obtained in each solution attempt, 3) the use of the platform tools, and 4) numerical grading. Additionally, the source code files sent as solutions to the programming tasks are used to obtain characteristics of the students' programs (software metrics).
Based on the identified measures, 24 metrics were designed, divided into three categories (verdicts, tool usage and software metrics). See Table 2. First, the verdicts' rates obtained for each type of verdicts in Table 1 are defined. These rates are the percentage of one type of verdict with respect to the total number of verdicts obtained by the student. The second category is the usage rates of each tool in Table 1. The usage rates are defined as the percentage of the number of accesses to a single tool in relation to the total number of accesses to all available tools (these two categories of rates are defined with the objective of quantifying the verdicts and tools of greater or lesser use by students).The third category are software metrics, which represent specific characteristics of the programs built by the students (see Table 3).
For each student the 7 verdict rates and the 5 tool usage rates are calculated by applying the equations in Table 2. Then, the software metrics are calculated using illustrates 21 measurements and metrics that have a statistically significant correlation (p-valor ≤ 0.05) with the academic performance.
The variable with the highest positive correlation is Accepted with coefficient of 0.43. This value is followed by Success_rate and Total_Submissions with coefficients of 0.25 and 0.24, respectively. These are expected results since a student with a high number of correct answers, success rate and attempts made is a student who is successful in solving the assignments. Subsequently, the variables with correlations less than 0.20 and greater than 0.15 correspond to Wrong_Answer, Time_Limit_Exceeded, Tokens_count, Custom_Input_rate, and NLOC. The num- ber of tokens, lines of code and Custom Input usage rate are metrics that were expected to be positively correlated, since as the student designs high content programs and is able to perform custom testing of the built code, his/her academic performance could be better. However, the number of incorrect answers and time limit exceeded errors were not expected to be in this group, because these verdicts indicate errors in the proposed solution.
In contrast, the variable with the highest negative correlation is Error_rate_Runtime_Error with −0.28. This metric was expected to be negatively correlated, given that a high runtime error rate represents that a large number of the non-executed attempts. The variables with correlations between −0.20 and −0.15 correspond to Linter_rate, MI, Custom_Input and Er-ror_rate_Wrong_Answer. In these results, it is rare that the rate of use of Linter and the accesses to Custom Input have negative correlations, since these tools are designed to facilitated students to build their solutions. The maintainability index and incorrect answer rate were expected to be negatively correlated, since a program with a high maintainability index indicates a low level of programming skills and a high incorrect answer rate means that most of the attempts made were unsuccessful.
Regarding the research question posed: what are the existing relationships between the data generated from students' interactions with an educational platform and their academic performance in a computer programming course when considering different groups of students?, some of the findings found in the results are discussed below. The total number of submissions made per student has a positive correlation with the final grade in the course. This may indicate that students with high performance use the platform as a source of feedback to improve the solutions constructed, making multiple attempts. This result agrees with the findings of Zacharis et al. [16] who found a weak correlation (from 0.20 to 0.39) between number of activities delivered and the final course grade. The positive correlations of the number of correct answers and the success rate indicate that the students have possibly acquired sufficient knowledge to successfully solve the course activities and this is reflected in their performances.
In contrast, the positive correlation of the number of incorrect answers, the number of memory limit errors exceeded, along with the number of verdicts and rates of time limit exceeded and compilation error may indicate that these verdicts provide sufficient feedback to guide the student to solve the possible errors present in the constructed solution. However, the rate of incorrect answer and execution error has negative correlations. In the case of high incorrect answers rate it may indicate a lack of understanding of the objective of the activity; with respect to the execution error rate, this may indicate the students' difficulty to obtain effective feedback for his learning process.
This work applies a non-experimental quantitative methodological design of learning analytics, consisting of three main phases: data preparation, transformation and analysis, applied to 2 years of historical data of a computer programming course. This allowed finding relationships between measurements and metrics of the learning process and the final students' performance of a computer programming course. The results show that the number of submissions, five verdicts(correct answer, incorrect answer, time limit exceeded, memory limit exceeded and compilation error), three verdict rates (correct answer, time limit exceeded and compilation error), the rate of use of Custom Input, and four software metrics (number of tokens, number of lines of code, cyclomatic complexity and difficulty) have a significant positive correlation with the academic performance of the students. This indicates that students with high performance may be able to use the error verdicts obtained as a source of formative feedback and that the construction of programs of high content, length and complexity is related to better academic performance. The variables with negative correlations are two verdict rates (incorrect answer and execution error), three tools' accesses (linter, multiple languages code and custom input), two tools' usage rates (linter and multiple languages code) and the maintainability index. These findings may indicate the need for reinforcement in some aspects of the course, such as, clarity in the objective of the activities, the applicability of the feedback offered by some tools (i.e., linter that focuses on the verification of good programming practices) and promote the learning of generalizable program construction skills.
The main limitation of this study is related to the quantitative approach, since the results are limited to evidencing relationships between measurements and metrics, but it is not possible to identify the causes of the found behaviors. As future work, the constructed dataset can be used to identify which metrics are most relevant and to track students and build a predictive model of final academic performance. Qualitative data will also be considered within the analysis in future works.
Measure of how easy to support and change the source code is (0-100).
a 𝑛 1 : The number of distinct operators. the Python libraries lizard 1 and radon 2 .
Exploratory data analysis. Figure 1 summarizes the most important results of the univariate analysis. In the case of total submissions made each student (top-left in Figure 1) on average makes (𝑥) 176.6 attempts with a standard deviation (𝑠) of 120.8. Regarding the average time between submissions (top-right in Figure 1) the 𝑥 is 7.1 ℎ with a 𝑠 of 19.8 ℎ. On the other hand, the plot of verdicts (bottom-left in Figure 1) evidences that the verdicts with the highest rates are Error_rate_Wrong_Answer and Success_rate with 48.9% and 31.7%, respectively. Regarding the tool usage rates, the total rates graph (bottom-right in Figure 1) shows that the most used tool by students is Custom_input_rate being 65.0% of the total accesses. This is followed by Python_Tutor_rate and Multiple_Languages_Codes_rate with percentages of 17.7% and 12.0%, respectively. Table 4 summarizes the statistical values describing the software metrics calculated based on the programs built by the students. Besides that, Figure 2 shows the distribution of the (uncode_grades) in each group 3 . The total 𝑥 is 4.1 with a 𝑠 of 0.9.
To identify which measurements or metrics are significantly related to the students' academic performance, a correlation analysis is performed. Considering that all variables are continuous, Pearson's correlation coefficient is used. Figure 2