Recent advances in information technology has led to an increasing number of applications to be developed and maintained daily by product teams. Ensuring that a software application works as expected and that it is absent of bugs requires a lot of time and resources. Thanks to the recent adoption of DevOps methodologies, it is often the case where code commits and application builds are centralized and standardized. Thanks to this new approach, it is now possible to retrieve log and build data to ease the development and management operations of product teams. However, even if such approaches include code control to detect unit or integration errors, they do not check for the presence of logical bugs that can raise after code builds. For such reasons in this work we propose a framework for continuous defect prediction based on machine learning algorithms trained on a publicly available dataset. The framework is composed of a machine learning model for detecting the presence of logical bugs in code on the basis of the available data generated by DevOps tools and a dashboard to monitor the software projects status. We also describe the serverless architecture we designed for hosting the aforementioned framework.
daily. Given the high number of activities that must be taken into account, such
as managing branches (development, stage, production, features), architecting
software applications, coordinating the developers and interacting with project
managers, it’s useful to have some automatic tools that alert the developers in
case bugs are detected in code. For such a reason, we propose a framework that
aims at easing the management and development activities and that integrates
with DevOps methodologies, with a focus on Continuous Integration (CI) and
Continuous Delivery (CD) operations. With CI/CD operations, thanks to
analysis tools such as Jenkins [
The rest of this paper is organized as follows: Chapter 2 describes the state of the art for continuous defect prediction whereas Chapter 3 describes the dataset, the preprocessing operations, the models used and the developed dashboard. In Chapter 4 some infrastructural considerations are described. Finally, Chapter 5 draws some conclusions and some future work. 2
DevOps is a software development methodology used in computer science that
aims at enhancing communication, collaboration and integration between
developers and information technology operations [
When it comes to software bugs, they usually appear during the software
development process and are dicult to detect or identify, thus developers spend
a large amount of time locating and fixing them. In order to detect them, many
machine learning algorithms have been developed and tested [
From all these studies emerge that, apart from the choice of the algorithm
that varies according to the used dataset, software metrics are extremely
important for fault prediction in quality assurance, hence, identification of proper
metrics is essential in all software projects [
In the context of DevOps CI builds, software bug detection plays an extremely
important role, particularly at change-level. Change-level defect prediction, also
known as just-in-time defect prediction, aims at predicting defective changes
(i.e. commits to a version control system) and is more practical because it can
not only ensure software quality in the development process, but also make
the developers check and fix the defects just at the time they are introduced.
There are a lot of studies about this. For example, in [
Scientific community also proposed several datasets related to continuous
defect prediction. In [
The objective of this research is to develop a framework capable of identifying bugs from commited code in order to provide from the one hand a synoptic point of view of the status of software projects and from the other hand alerting if some inconsistencies and logical bugs are detected.
The proposed framework consists of three main components: a data processing pipeline, a machine learning model for classification and a monitoring dashboard. In the following sections we will described all these components in details, by focusing on the publicly available dataset used for training the model, the preprocessing operations executed to make the dataset compliant with data from the production environment, the machine learning models tested that follows the trend of the state of the art and the implemented dashboard with the way of using and the kinds of analysis performed over it. 3.1
The dataset used for model training is the Technical Debt Dataset [
Jira issue tracker. – Sonar measures: contains the SonarQube measures such as number of code lines in the commit, the code complexity and the number of functions. – Commits: contains the information retrieved from the git log including the commit hash, the message, the author, the date and timezone and the list of branches. – Commit changes: contains the changes contained in each commit, including the old path of the file, the new path, the type of change (added, deleted, modified or renamed), the di↵, the number of lines added. – Jira issues: contains Jira issues for each project with information such that the key, the creation and resolution dates and the priority. – Fault inducing commits: reports the results from the execution of the
SZZ [
Since the real case production environment slightly di↵ers from the aforementioned dataset due to some constraints imposed by CI/CD tools, that impose us to have data aggregated at push granularity instead of commit granularity, the Technical Debt Dataset has first been processed and synthetically modified to match the push granularity by aggregating subsequent commits with windows of varying lengths between 1 and 5. Aggregation of numerical features has been executed in some cases by averaging the numerical values while in other cases selecting only the min/max values, depending on the meaning of the feature in the context of software development.
In addition, dataset features have undergone a selection process that involves
studying the correlation between each feature with respect to the dataset label,
as shown in Figure 1. The process consisted of a Recursive Feature Elimination
(RFE) [
Feature name Description token count Token count of functions complexity Cyclomatic complexity nloc Lines of code of the file lines added Number of lines added lines removed Number of lines removed modification type Type of changed applied
(modify, delete, add, rename) files changed Number of modified files count e↵ort Time needed to solve the issues sum class complexity Complexity of classes in commits count severity Severity level of the issues max
Table 1. Detail of selected features
Aggregation Source sum, mean PyDriller sum PyDriller sum, mean PyDriller sum, mean PyDriller sum, mean PyDriller count PyDriller
PyDriller SonarQube
SonarQube SonarQube
The final set of features selected for training is described in Table 1. The FRE process allowed us to select only 18 features. in Table 1 we can see the feature name, the description, the aggregations and the source. Each feature is considered once per each aggregation, thus, as an example, the ”token count” feature is considered twice, both as sum of token count of each commit and as mean of token count of each commit. Moreover, the feature ”modification type”, given its categorical nature, is considered as a unique feature for each category it can assume (modify, delete, unknown, add, rename). 3.2
The models chosen for continuous defect prediction are the ones mostly used in
scientific research, such as in [
As far as RF is concerned, its hyperparameters have been identified by means of grid search optimization over a 5-fold cross validation, that resulted in the following selection: – Bootstrap: yes – Criterion: gini – Number of Estimators: 250 – Minimum number of samples for splitting: 3 The overall performance on the test set are shown in Table 2.
Where the True Negatives (TN) are 57 and the True Positives (TP) are 69 whereas the False Positives (FP) are 7 and the False Negatives (FN) are 6.
The other model tested is a Dense Neural Network (DNN), a particular case of the ANN, trained for binary classification. Its hyperparameters have been identified by means of grid search optimization over a 5-fold cross validation and early stopping criterion to avoid overfitting and select the number of epochs, that resulted in the following selection: – Optimizer: Adam – Learning rate: 0.01 – Number of epochs: 5 – Loss function: Categorical Cross Entropy – Accuracy metrics: Accuracy The network architecture is composed of a dense layer with an output dimension equal to 64 and 1344 parameters, a dropout layer and a final dense layer with an output dimension equal to 2 and 130 parameters. Performance of the DNN model on the test set is shown in Table 3.
Where the True Negatives (TN) are 53 and the True Positives (TP) are 58 whereas the False Positives (FP) are 22 and the False Negatives (FN) are 6.
As shown above, performance of the RF is better than the one of the DNN, especially when comparing the precision and the recall metrics of the positive class. Indeed, the RF model outperforms the DNN when considering all the metrics. 3.3
According to the results of the trained classifiers, the RF model has been deployed and used in a production environment. In order to visualize the predictions of the model and keep track of all the operations performed in the di↵erent software projects, each project has been connected to the model and results have been recorded in a database to be visualized. Thus, a monitoring dashboard have been designed to easily read the results of the analysis. Given the high number of data collected daily, we include in the dashboard distributions and time series plots, in order to give a synoptic point of view of the software projects’ status.
In details, the dashboard firstly includes some global metrics that resume the operations performed in the di↵erent code repositories over which the extraction and prediction algorithms have been connected. In particular the global metrics included are: – Number of commits: total number of commits analyzed by the system. – Lines added : number of code lines added in the analyzed commits. – Lines removed: number of code lines removed in the analyzed commits. – Files changed: number of files modified in the analyzed commits. – Total bugs: number of bugs identified by the machine learning algorithm in the analyzed commits.
The dashboard also includes a detailed tables that contains some references useful to reconstruct the history of each commit, such as the username, the branch name, the Jenkins job name, the commit hash, the critical violations and the bugs detected by SonarQube. All these informations are correlated with the model prediction, so that it is easy when the model detect a bug to identify the project, the branch and the authors of the interested commit.
Then, with a pie chart it is possible to easily understand the percentage of commits with a possible bug and with a time series chart it is possible to see the trend of detected bug together with some other important features, such as number of lines added, number of files changed and number of lines removed.
Finally, the dashboard gives the user the possibility to filter the content in order to reduce the number of visualized data and change the dimensions of the analysis. The available filters are based on date range, name of the repository, branch name, Jenkins job name, username and model prediction. In this way, all the charts and metrics previously described can be recomputed with specific filters, thus deeply exploring each project according to di↵erent dimensions.
The developed dashboard has not given only the possibility to detect bugs in real time, but also the chance to evaluate the quality of the software projects. Indeed, thanks to the di↵erent charts and filters, it has been possible to identify projects or branches with more bugs, which generally means that are more complex projects/branches or projects/branches for which the requirements are not well defined. Moreover, it has been possible to monitor also the resources that contribute to software projects. As an example, it has been possible to identify resources that usually do not commit frequently, but only in case of large changes or improvements in team’s code quality. 4
To conclude, we present some considerations related to the infrastructure that we set-up to serve the proposed framework.
We deployed the services using Google Cloud Platform [
The goal of this research was that of defining a framework to detect bugs in software projects’ commits in a change-level defect prediction scenario. In order to define such framework, we firstly analyzed the state of the art for machine learning algorithms applied to support CI/CD operations, with a focus on continuous defect prediction. The analysis of the state of the art allowed us to define the main machine learning approach to defect prediction, the main publicly available datasets and some related works. Thus, we selected an approach and a public dataset that fit our needs to create the aforementioned framework. However, di↵erently from the related works that focus on processing the logs of CI tools, given the constraints set by CI/CD tools used in our production environments, we needed to preprocess the dataset to consider an agglomerations of record based on subsequent commits. Then, we designed and tested two models: a RF and a DNN both with grid search over 5-fold cross validation for hyperparameters optimization and early stopping for DNN. Experimental results showed that, on the preprocessed dataset, RF outperformed DNN, with an average F1-score over the positive class of 0.91 against 0.81. Thus, we define a Google Cloud based architecture to host our framework for real time monitoring, that allowed to link di↵erent software projects to the RF model and register all the logs produced in order to visualize the results. We also design a monitoring dashboard, that allowed us to derive important insights and evaluate software quality.
The developed framework is extremely useful, especially thanks to the synoptic point of view that provides, however some enhancements can be performed. As an example, some future developments could involve the augmentation of the features with user-specific information to make the model learning userpatterns. Other future developments are related to a posterior analysis of the machine learning models. Indeed, we couldn’t train a model on a real dataset and a manual analysis of the results showed that this is the reason why the model produces some false positive. Thus, validating the results of the models and retrain the model with a real dataset could enhance the model’s performance and allow for a better usage of the framework.