Background: Continuous Integration (CI) is an agile software development practice that involves producing several clean builds of the software per day. The creation of these builds involve running excessive executions of automated tests, which is hampered by high hardware cost and reduced development velocity. Goal: The goal of our research is to develop a method that reduces the number of executed test cases at each CI cycle. Method: We adopt a design research approach with an infrastructure provider company to develop a method that exploits Machine Learning (ML) to predict test case verdicts for committed source code. We train ve di erent ML models on two data sets and evaluate their performance using two simple retrieval measures: precision and recall Results: While the results from training the ML models on the rst data-set of test executions revealed low performance, the curated dataset for training showed an improvement on performance with respect to precision and recall. Conclusion: Our results indicate that the method is applicable when training the ML model on churns of small sizes.
Meding2
CI is a modern software development practice, which is based on frequent
integration of codes from developers and teams into a product's main branch [
In order to address this challenge, a CI system should be able to pinpoint exactly which test cases should be executed in order to maximize the probability Copyright © 2019 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). of nding defects (i.e. to reduce the \empty" test executions). To achieve this, the CI system needs to be able to predict whether a given test case has a chance of nding a defect or not, or at least whether it will fail or pass { predict the verdict of a test case execution.
We set o to address the problem of predicting test case verdicts by training ve ML models on a large data set of historical test cases that were executed against changes made to a software developed at company A. The term "code churn" is de ned as a measure that quanti es these changes. Throughout the remaining sections of this paper, we use this term to refer to committed code made during di erent CI cycles.
Our research is inspired from a previous study conducted by Knauss et el.
[
How to reduce the number of executed test cases by selecting the most e ective minimal test suite when integrating new code churns into the product's main branch?
Our study was conducted in collaboration with a large Swedish-based infrastructure providing company. We study a software product that has evolved over a span of a decade by di erent cross functional teams. As a result of our study, we present a method that uses ML to predict test case verdicts (MeBoTS).
To address this research question, we conduct a design research study, where
we develop a new method and evaluate it on the company's data set. Our method
is based on the research by Ochodek et al. [
The remainder of this paper is organized as follows: Section II provides background information about two categories of ML. The sections that follow provide: an overview of the most related studies in this area, a description of the method that we developed in our study as well as the results, validity analysis, recommendations, and nally, conclusion. 2 2.1
Machine learning is a class of Arti cial Intelligence that provides systems the
ability to automatically make inferences, given examples relevant to a task [
In supervised learning the training data-set fed into the ML model contains
the desired solution, called labels. The model tries to nd a statistical structure
between these examples and their desired solutions [
In Machine learning, a decision tree is an algorithm that belongs to the family
of supervised learning algorithms. The algorithm has an inherent tree-like
structure and is commonly used for solving classi cation and regression problems
[
Deep Learning is a branch of ML that was founded on the premise of using
successive learning "layers" to achieve more useful representation of the data
[
The amount of changes made to software over time is referred to as code churn
[
In the following we discuss related work on the speci c use of machine learning for test case selection or prioritization. 3.1
Around 2015/16, we nd the rst machine learning based approaches for
testcase selection. With only 4 studies included in the systematic mapping study by
Durelli et al. [
Busjaeger and Xie [
Chen et al. [
Spieker et al. [
Most recently, Azizi and Do [
Palma et al. [
However, to the best of our knowledge, no other learning-based method
works for code-churns. The only exception is one of our previous collaboration
with Knauss et al. [
The following section is a description of the MeBoTS method used in this
research, which comprises of three sequential steps, as shown in Figure 1. The
method utilizes two Python programs and an open source textual analyzer
program, called CCFLEX [
The program rst loops through the extracted list of tests and looks at the change history log maintained by Git and performs a le comparison utility (di ) on a pair of consecutive baselines in the tests list. Note that each baseline value is a hash representation of a revision (build), pointing to a speci c location in Git's history log. The result is a ne-grained string that comprises the committed code churns, where each LOC in the churn is compiled with its: 1) lename, 2) physical le path, 3) test case verdict, 4) baseline hash code.
The resulting string is then arranged in a table-like format and written in a csv le, named as 'Lines of Code' in Figure 1. 4.2
The result of the extract from Git is saved as an array (code churn, lename, physical le path, test case verdict and baseline). This le is the input to our textual analysis and feature extraction. The textual analysis and feature extraction use each line from the code churn and: { creates a vocabulary for all lines (using the bag of words technique, with a speci c cut-o parameter), { for the words that are used seldom (i.e. fall outside of the frequency de ned by the cut-o parameter of the bag of words), a token is created, { nds a set of prede ned keywords in each line, { check each word in the line whether it is part of the vocabulary, it should be tokenized or if it is a prede ned feature.
An example input is presented in Table 2. The input contains an example code in C.
For the textual analyses, we can pre-de ne (arbitrarily for this example) two features: \if" and \int". The bag of words analysis also found the word \printf" as frequent. It has also de ned the following tokens: { \a" { to denote the words (of any length) that contain only lowercase letters (e.g. \condition"), { \Aa" { to denote the words that start with capital letters and continue with lowercase letters (e.g. \Hello"), { \0" { to denote the numbers, i.e. sequence of numbers of any length (e.g. \10")
The manual features and the bag of words results are then used as features in the feature extraction. Table 3, which corresponds to the input from Table 2.
Table 3 is a large array with the numbers, each representing the number of
times a speci c feature presents in the line. This way of extracting information
about the source code is new in our approach, compared to the most common
approaches of analyzing code churns. Compared to the other approaches, MeBoTS
recognizes what is written in the code, without understanding of the syntax or
semantics of the code. This means that we can analyze each line of code
separately, without the need to compile the code or without the need to parse it.
This means, that we can take code churns from di erent les in the same
baseline and analyze them together. MeBoTS also goes beyond such approaches like
Nagappan et al. [
We exploit the set of extracted features provided by the textual analyzer in step 2 as the independent variables and the verdict of the executed test cases as the dependant variable, which is a binary representation of the execution result (passed or failed). The MeBoTS method uses a second Python program that utilizes and trains an ML model to classify test case verdicts. The program reads the BOW vector space le in a sequence of chunks, merging the extracted feature vectors and the verdicts vector into a single data frame that gets split into a training and testing set before it is fed into the models for training. Table 4 shows an excerpt of the generated data frame.
The study has been conducted at an organization, belonging to a large infrastructure provider company. The organization develops a mature software-intensive telecommunication network product. The organization consists of several hundred software developers, organized in several empowered agile teams, spread over a number of continents. Given that they have been early adopters of lean and agile software development methodologies, they have become mature in these areas of work. They have also implemented CI and continuous deliveries.
The organization is also mature with regard to measuring. For instance every agile team, as well as leading functions/roles, uses one or more monitors to display status and progress in various development and devops areas. A wellestablished and e cient measurement infrastructure, automatically collects and processes data, and then distributes the information needed by the organization. 5.2
The data-set provided by company A contained historical test case execution results for a mature software product that has evolved for almost a decade. The analyzed product consisted of over 10k test cases and several million lines of code written in the C language. We decided to test the MeBoTS on a set of randomly selected tests that, presumably, reacted to changes in the source code during di erent CI cycles. Our selection of test cases was based on the granularity of test executions whose verdicts changed from one state to another(see Table 1).
The extracted data-set belonged to twelve test cases that were executed 82 times during di erent CI cycles. The size of the extracted churns was 1.4 million lines of code, among which 618k lines were labeled as passed and 776k as failed. To better understand the shape of the data-set, we visually inspected the size of code churns covered by each test execution. The scatter plot in Figure 2 shows the distribution of the 82 extracted test executions, belonging to the twelve test cases. Each mark on the plot represents one test case execution. The x-axis represents the number of lines in the code churn the test case was executed on. The y-axis represents the overall number of test executions for the executed test case. Test executions of the same test case are marked with the same symbol. The visual inspection of the scatter plot suggests that our data-set comprised of churns of varying sizes (large and small). We interpreted this distribution with uncertainty of whether large churns contain additional noise that would adversely a ect the training of the ML models. As a result, we decided to curate the original data-set by ltering out tests that were executed on churns whose total size exceeded 110k lines. The visual inspection of the curated data-set is represented in Figure 3, comprised of 290k lines of code, containing a fairly balanced representation of the binary classes (passed and failed), with 110k lines belonging to the passed class and 180k lines belonging to the failed class.
The two data-sets described above were used for training the ML models. The rst data-set was used in the rst phase, whereas the second curated dataset was used in the second phase of ML training, and ultimately became our focus due to data size homogeneity. 5.3
To select the most suitable model for classifying test verdicts, we selected ve di erent ML models and trained them sequentially. The ve models are: 1) Decision Tree, 2) Random Forest, 3) AdaBoost, 4) Multilayer NN, 5) CNN
The choice of selecting the three tree-based models was due to their low computational cost and white box nature, whereas the selection of ANN models were based on their ability to abstract large and complex number of features.
Each of the ve classi cation models for test verdicts uses i) the historical test case verdicts ii) the feature vectors in the bag of words table as the baseline of prediction. The evaluation was done in two iterations. In the rst iteration, the ve models were trained on the original data-set, which contained a mix of large and small churns, comprising a total of 1.4m lines of code for 500 feature vectors. The second iteration involved training the models on the curated data-set, which almost contained 290k lines for the same 500 features, as in the rst iteration. Both data sets, curated and original, were split as follow: 70% for the training set and 30% for the test set.
To save the long run time required by automated hyper-parameter tuning
tools such as grid and random search, the con guration of the models was done
manually. Table 5 summarizes the hyper-parameters used for training the
models. We used the implementation of Decision Tree, Random Forest, and
AdaBoost algorithms available in the Python scikit-learn library [
The hyper-parameters of the three tree-based models were kept in their default state as found in the scikit-learn library. The only alterations made were in the 'random state' value in Decision Trees and the n estimator (number of trees) in both Adaboost and Random Forest. With respect to the ANN models, the architecture of the multilayer ANN was represented with three sequential dense layers that consisted of: one input layer, one hidden layer, and one output layer. For the CNN, the stack of layers comprised of: a Reshape layer, a Convolution layer, a Maxpooling layer, and four Dense layers. The learning in both models was induced over 100 epochs (iterations), as can be seen in table 5
The performance of the classi ers were evaluated using simple retrieval measure: recall and precision.
These measures are based on the following four categories of errors: { True positives: correct prediction of test executions that pass { True negatives: correct prediction of test executions that fail { False positives: incorrect prediction of test executions that pass { False negatives: incorrect prediction of test executions that fail Precision is the number of correctly predicted tests divided by the total number of predicted tests, calculated as follows: precision =
jT rueP ositivej jT rueP ositivej + jF alseP ositivej Recall is the number of correctly predicted tests divided by the total number of tests that should have been positive.
recall =
jT rueP ositivej jT rueP ositivej + jF alseN egativej While recall and precision measures relate to one another, precision is a measure of exactness, whereas recall is a measure of completeness indicating the percentage of all predicted failed tests in our case. Since the goal of this study is to reduce the amount of test executions without altering the e ectiveness of testing, we needed to minimize the risk of missing tests that will actually fail and, therefore, accept some false alarms in the prediction of failed tests. Thus, we believe that the measure of the mode's precision is more important than its recall. 6
To answer the research question, we present the results of using MeBoTS for predicting test case verdicts. We interpret the results of the analysis in light of the reported rates of precision and recall and the values of the four categories of errors: true negatives, false positives, false negatives, and true negatives, as shown in Tables 6 and 7 list the results of training the ve models using the original and curated data sets. 6.1
The evaluation of the ve models in the rst iteration reports a mean precision of 47% and a mean recall of 17%, suggesting that all models achieved low performance. The best result was obtained by the Multilayer NN model with a precision rate of 50.3% and a recall rate of 31%. The precision and recall rates for the ve models can be seen in Figure 4. The interpretation of these values suggest that out of the 406,948 lines of code that actually triggered a test case failure, the model correctly predicted test failure for 226,994 lines, whereas it could correctly predict a passing test verdict for 5,758 out of 11,428 lines. Similarly, the results of the four categories of errors for the other models can be seen in Table 6 and interpreted in the same fashion.
Model Decision Trees Random Forest
AdaBoost Multilayer NN
CNN
The second iteration of analysis involved training the same set of models on the curated data-set, which excluded tests covering churns with quantities above 110k lines of code. The results, shown in Figure 5, indicate an improvement in precision and recall when compared to the results in the rst iteration for the same types of models. Table 7 reports the results from the second round of training, showing a mean precision of 70% and a mean recall of 44.5%. The Multilayer NN model performed best in prediction, such that, it correctly predicted 48,755 lines that actually triggered a test case failure, out of 67,363 lines in the test set.
Model Decision Trees Random Forest
AdaBoost Multilayer NN
CNN
The results show that we can predict the verdict of a test case with a precision
of 73.3%. This means that we can use the results to reduce the test suite by
excluding tests that are predicted to pass, but we need to know that there is
26.7% probability that we miss a test case failure. This means that the reduction
of the test suite comes with a cost. This cost can be reduced, for example, if we
collect the test cases which were not executed and execute them with lower
frequency (e.g. during a nightly test suite instead of hourly builds).
When analyzing the validity of our study, we used the framework recommended
by Wohlin et al. [
The major threat to external validity for this study comes from the number of extracted tests that were used for training the classi ers. We only studied one company and one product and a limited number of test cases. This was a design choice as we wanted to understand the dynamics of test execution and be able to use statistical methods alongside the machine learning algorithms. However, we are aware that the generalization of the results for di erent types of systems require further investigations using tests and churns from di erent systems. 8
In this section, we provide our recommendations for practitioners who would like to utilize MeBoTS for early prediction of test case verdicts.
{ The choice of using deep learning or tree-based models for solving this supervised ML problem does not lead to better prediction performance. For this reason, we suggest the use of Decision Trees, since they require less computational time and provide knowledge as to how the results of classi cation were derived. { We suggest to only utilize code churns that are homogeneous and small in size prior to applying features extraction with BOW. Small code churns introduce less noise and therefore the quality of the predictions is higher.
This can also save practitioners ample time for data curation. { We recommend that practitioners only extract historical test executions that have failed due to reasons related to functional defects for training the ML model. This knowledge can be obtained from testers/developers who are familiar with the recurrent issues in the source code. 9
This paper has presented a method (MeBoTS) for achieving early prediction of test case verdicts by training a machine learning model on historical test executions and code churns. We have evaluated the method using two data sets, one containing a variation of large and small churns, and a second containing only small churns. The results from training the models on small churns revealed a precision rate of 73% and a recall of 43.6%, suggesting that the application of the method is promising, yet more investigation is required to validate the ndings. Moreover, contrary to other existing methods that use statistical correlations for predicting test verdicts, the main advantage of MeBoTS is the ability to predict verdicts of new code changes as they emerge during development and before they get integrated into the main branch.
We believe that the results of this study open new directions for studies to investigate the e ectiveness of MeBoTS on di erent types of systems using larger set of small churns with more test case executions. Finally, studies that investigate the impact of using di erent feature extraction techniques, such as word embedding are encouraged to identify any changes in the overall performance of MeBoTS.
This research has been partially carried out in the Software Centre, University of Gothenburg, and Ericsson AB.