Version Control Systems are commonly used for Information and communication technology professionals. They also allows to follow the activity of a single programmer working in a project. For these reasons, Version Control Systems are also used by educational institutions. The aim of this work is to demonstrate that the student performance may be evaluated, and even predicted, by monitoring their interaction with a Version Control System. In order to do so we have build a Machine Learnings model to predict student results in a speci c task of the Ampliacion de Sistemas Operativos subject from the second course of the degree in Computer Science of the University of Leon through their interaction with a Git repository.
The emergence of the Information and Communication Technologies have change the landscape of the teaching and learning processes. Teachers can employ a lot of tools in their classes with the aim to improve students learning. In addition students can use di erent application to learn in their education center and beyond it. However, Is it possible to say if a tool is improving student performance? If we can assert this, it would be possible to use the tool that better ts with speci c lessons or students. There are several studies regarding to this, and this issue is specially link to trends such as Learning Analytics and Educational Data Mining.
Copyright © 2018 for this paper by its authors. Copying permitted for private and academic purposes
The most accepted de nition of learning analytics considers that it comprises
\the measurement, collection, analysis and reporting of data about learners and
their contexts, for purposes of understanding and optimising learning and the
environments in which it occurs" [
In this case the idea is to explore how students' performance is a ected by
the use of Version Control Systems (VCSs). VCSs facilitate the management of
changes in the components of a software product or its con guration[
The aim of this work is to build a model that allows to predict student results at a practical assignment by monitoring their use of a VCS. We assume the premise that the students activity with this type of systems is an indicator of the evolution of their progress.
The rest of the paper is organized as follows: Section 2 describes the empirical evaluation of the classi cation algorithms presenting the experimental environment, materials, and methods used. Section 3 summarizes the results of the evaluation. The discussion of the results is developed in Section 4. Section 5 presents the conclusions and future lines of research. 2
Materials and Methods This section describes all the elements and the methodology used to build and evaluate the model for predicting student results. Among the elements used there are a speci c practical assignment to provide student results, and a VCS. Regarding the methodology, a set of classifying algorithms have been evaluated by analysing some well-known Key Performance Indicators (KPIs). 2.1
The Ampliacion de Sistemas Operativos (ASSOO) subject from the second course of the degree in Computer Science of the University of Leon broadens knowledge about operating systems. In particular, it addresses the internal functioning of storage management, both volatile (memory management) and nonvolatile ( le management). Issues related to security in operating systems are also addressed.
Main practical assignment consists on implementing an inode-based le
system called Ampliacion de Sistemas Operativos File System (ASSOOFS).
According to the proposed speci cation, this le system must work on computers
that run the Linux operating system. Therefore, students have to implement a
module for the Linux kernel [
This is an individual assignment and each student is encouraged to use a VCS during the completion of the task. 2.2
In software engineering, it is known as control of versions to the management
of the changes that are made on the elements of some product [
Version management can be done manually, although it is advisable to use
some tool to facilitate this task. These tools are known as VCSs [
A VCS must provide, at least, the following features: { Storage for the di erent elements to be managed (source code, images, documentation). { Edition the stored elements (creation, deletion, modi cation, renaming, etc.). { Registration and labelling of all actions carried out, of so that they allow an element to be returned to a state previous.
For the development of ASSOOFS, students are encouraged to use a Git
repository. Git follows a distributed scheme, and contrary to other systems that
follow the client-server models, each copy of the repository includes the story
complete of all the changes made [
In order to provide some organizing capabilities and private repositories for
students the GitHub Classroom platform was used [
Features Regarding the input data to predict results, usually called features in a Machine Learning (ML) context, we have considered the following information coming from students activity on their repositories: { Commits : total number of commit operations carried by the student. { #Days with commit operations : total number of days where there is at least one commit operation.
{ Commits/date: average number of commit operations per date. { Additions: number of lines of code added during the assignment completion. { Deletions: number of lines deleted during the assignment completion.
In addition to the above data, all obtained from the GitHub Classroom platform, we have also considered the students grade on a proof carried out to control the authorship of the code in student repositories. This authorship proof allows to verify that the students really worked in the content of their repository. The authorship proof has two possible results: \1", if the student passed the proof; \0" otherwise.
Input data explained above will be used by the model to predict a class: AP, for those students who will nish the practical assignment successfully; and SS, for those who not. 2.3
We want to generate a model whose inputs are quantitative, while its output is a discrete value: AP, and SS. Two types of ML algorithms may be used: classi ers and predictors, whereby considering the rst ones will be better. We have evaluated the following well-known methods that we think are the more promising ones: Adaptive Boosting (AB), Classi cation And Regression Tree (CART), K-Nearest Neighbors (KNN), Linear Discriminant Analysis (LDA), Logistic Regression (LR), Multi-Layer Perceptron (MLP), Naive Bayes (NB), and Random Forest (RF).
AB Ensemble methods are techniques that combine di erent basic classi ers turning a weak learner into a more accurate method. Boosting is one of the most successful types of ensemble methods, and AB one of the most popular boosting algorithms.
CART A decision tree is a method which predicts the label associated with an
instance by travelling from a root node of a tree to a leaf [
KNN Although nearest neighbours is the foundation of many other learning
methods, notably unsupervised, supervised neighbour-based learning is also
available to classify data with discrete labels. It is a non-parametric technique
which classi es new observations based on the distance to observation in the
training set. A good presentation of the analysis is given in [
LR Linear methods are intended for regressions in which the target value is
expected to be a linear combination of the input variables. LR, despite its name,
is a linear model for classi cation rather than regression. In this model, the
probabilities describing the possible outcomes of a single trial are modeled
using a logistic function.
MLP An arti cial neural network is a model inspired by the structure of the
brain. Neural networks are used when the type of relationship between inputs
and outputs is not known. It is supposed that the network is organized
in layers (input layer, output layer and hidden layers). An MLP consists
of multiple layers of nodes in a directed graph so that each layer is fully
connected to the next one. An MLP is a modi cation of the standard linear
perceptron and, the best characteristic is that it is able to distinguish data
which is not linearly separable. An MLP uses back-propagation for training
the network, see [
NB This method is based on applying Bayes' theorem with the \naive"
assumption of independence between every pair of features, see [
To evaluate the previous methods, the implementation of the Scikit-learn
library has been used [
In order to train the models we have use the data obtained by the ASSOO
students from de 2016{2017 course presented at [
To evaluate the above algorithms with there input data, we have followed
the method proposed at [
accuracy =
P Tp + P Tn
P total data
The three models with the highest accuracy classi cation score have been pre-selected for in-depth evaluation by considering the following KPIs: Precision (P ), Recall (R), and F1-score; all of which were obtained through the confusion matrix.
The Precision (P ) is computed as shown at equation 2, where P Fp is the number of false positives.
P =
P Tp
P Tp + P Fp
The Recall (R) is computed at equation 3, where P Fn is the number of false negatives. (1) (2)
R =
P Tp
P Tp + P Fn According to the above results, as shown at Table 1{left, RF classi er works better (accuracy score = 0.8) than any other for selected features, in this case: Commits, #days with commit operations, commits/date, additions, and deletions. CART classi er works slightly worse (accuracy score = 0.7) than RF, while all the other classi ers o er very poor results.
Once the best models are pre-selected, a deeper analysis with the confusion matrix of each one is given. Another important item that should be analysed is the sensitivity of the model for detecting a passed assignment (AP): i.e., the rate of APs that the model classi es incorrectly. Table 1{right and Fig 1, show that the RF classi er gets better average values for Precision (P ), Recall (R) and F1-score than CART and LR. This work aim to build a model to predict students results by monitoring their activity at VCSs. We start from the premise that analysing the students activity at VCSs allows to predict their results.
To build the model several classi ers have been evaluated. In addition to select the best classi er, we have demonstrated that our premise is true due to the fact that we can predict the students results with a success high percentage. However, the models were evaluated using a small dataset. It would be desirable to get a larger volume of data to perform the analysis.
Regarding the chosen features, we observe that in addition to consider the repository activity, adding an authoring proof helps to increase the accuracy.
Future work will be related to the tuning the hyper-parameters of models in order to obtain better results. In addition, we need to increase de training dataset.