Regression testing is an important testing activity in continuous integration (CI) since it provides con dence that modi ed parts of the system have not adversely a ected its expected behavior. As test suites grow in size over the evolutionary cycle of software, executing large test suites becomes costly and precluding. Since CI provides a large volume of data, machine learning approaches can be used to allow test orchestrators make inferences about which subset of test cases to run at each CI cycle. MeBoTS is a machine-learning-based method that utilizes CI data to improve test case selection (TCS). In order to decide which extraction algorithm is more suitable, we designed and performed an experiment to investigate the e ect of using one of two widely used feature extraction algorithms|Bag of Words (BoW) and Word Embeddings (WE)|on the predictive performance of the MeBoTS classi er. We used strati ed cross-validation and precision and recall measures to evaluate the performance of two machine-learning models trained on the input generated by the feature-extraction algorithms. The results from this experiment show a signi cant di erence between the models' performance scores with a higher mean precision and recall scores for the BoW based classi er. We conclude that the use of BoW allows training a more accurate MeBoTS classi er than WE.
Continuous integration is used increasingly often in software engineering projects. Both large and small software companies use this technique to increase the quality of their products, as continuous integration advocates small increments in software code and frequent testing. However, one of the challenges in continuous integration is the need for resources for testing, which needs to be done on 3 every single code commit. In our work, we addressed this problem by de ning a method for predicting whether a given source code commit should be tested by a speci c test case. Based on the observation that faults may occur in similar patterns of source code, our method abstracts these patterns by tokenizing lines of code and weighing them using their associated frequency. This means that the numbers of syntax tokens in source code are regarded as predictors to test case failures. Using these feature vectors (source code tokens) as predictors to test failures provides a basis for test orchestrators to prioritize and select test cases.
The prediction algorithms used in our method (MeBoTS, [
Henceforth, in this paper, we pose the following research question: RQ: Is there a statistically signi cant di erence between the performance of the test predictor based on the usage of BoW vs. WE?
In order to address this question, we design an experiment, where we use 15 di erent sets of code commits as experiment factors. We use the statistical performance measures of recall and precision as the dependent variables. The results of the experiment show that the prediction from the WE-based classier have a statistically signi cant lower precision and recall scores than that produced by a BoW-based classi er. This means that the simpler method for making predictions is better in this context.
The paper is organized as follows: in Section 2 we introduce related work on code feature extraction techniques in text classi cations; in Section 3 we introduce background information; in Section 4 we describe all the steps in the design of our experiment; in Section 5 we provide the results of our experiment; in Section 6 we introduce some threats to validity and limitations; in Section 7 we describe our recommendations for future implementation of MeBoTS; nally, in Section 8 we make our conclusions. 2
Since the ultimate goal of this research is to improve TCS, we start by presenting an overview of some proposed TCS approaches and explore their drawbacks. 3 Copyright 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Then we review the literature on studies that have examined the e ectiveness of BoW and WE in text classi cation. 2.1
Rothermel and Harrold [
A TCS technique proposed by Volkolos and Frankl [
Dynamic slicing based approaches for TCS use slice executions to determine
which subset of test cases should be exercised. Agrawal et al [
In a study conducted by Chao et al. [
Similarly, Enriquez et al. [
MeBoTS is a machine learning based model that aims at predicting test case verdict using historical test execution results and code churns. The term code churns is used here to refer to source code changes made between two checkins. The method is comprised of 3 steps, as shown in Fig 1. This section brie y describes these steps.
Code Churns Extractor (Step 1) The method uses a code churn extractor
program that collects and compiles churns of source code from one or more
repositories. The program expects one input parameter: a time ordered list of historical
test case execution results queried from a database, where each element in the
list is a metadata state representation of a previously run test case. Each state
contains a hash reference that points to a speci c location in Git's history for
the tested check in. The program performs a le comparison utility (di ) across
pairs of consecutive commit hashes in the list using the GitPython library [
Textual Analysis and Features Extraction (Step 2) The second step in the method is to extract features from the collected code churns (output of step 1) and transform the source code into a numerical form. For thWe used an open source tool (cc ex) that utilizes BoW for modelling textual data. The input to cc ex is the output of the churn extractor in step 1. cc ex uses each line from the code churn and: { creates a vocabulary for all lines (using the bag of words technique, with a speci c cut-o parameter) { creates a token for the words that are seldom used(i.e. fall outside of the frequency de ned by the cut-o parameter of the bag of words) { nds a set of prede ned keywords in each line { checks each word in the line to decide if it should be tokenized or if it is a prede ned feature This way of extracting information about the source code is new in our approach, compared to the most common approaches of analyzing code churns. In contrast with other approaches, MeBoTS recognizes what is written in the code, without understanding the syntax or semantics of the code. This means that we can analyze each line of code separately, without the need to compile the code and without the need to parse it.
Training and Applying the Classi er Algorithm (Step 3) We exploit the set of extracted features provided by the textual analyzer in step 2 as the independent variables and the verdict of the executed test cases as the dependant variable, which is a binary representation of the execution result (passed or failed). The MeBoTS method uses a second Python program that utilizes and trains an ML model to classify test case verdicts. The program reads the BoW vector space le in a sequence of chunks, merging the extracted feature vectors and the verdicts vector into a single data frame that gets split into a training and testing set before it is fed into the models for training. Word embeddings is one of the approaches used to represent words in a machinefriendly way. It has been proposed as an alternative to the bag-of-words model and quickly become the state-of-the-art method used while training neural networks. In word embeddings, each word is represented as a dense, low-dimensional oating-point vector. Such vectors are learned from data. Another important bene t of using word embeddings is that the geometric relationships between word vectors should re ect the semantic relationships between these words. Using the word embeddings algorithm for source code classi cation could be benecial for two main reasons. Firstly, it allows us to represent each of the tokens in a line of code and feed them to a neural network capable of processing sequences (e.g., a convolutional neural network). Secondly, it captures similarities between the roles of tokens in the code without the need for parsing the code.
4.1
The study has been conducted at an organization, belonging to a large infrastructure provider company. The organization develops a mature software-intensive telecommunication network product. The organization consists of several hundred software developers, organized in several agile teams. The organization is mature with regard to measuring. For instance every agile team, as well as leading functions/roles, uses one or more monitors to display status and progress in various development and devops areas. A well-established and e cient measurement infrastructure, automatically collects and processes data, and then distributes the information needed by the organization. 4.2
Our data-set comprised of historical test execution results and code churns for software that has lived and evolved for over a decade at the collaborating company. The analyzed software was written in the C language and contained a few million lines of code and a test pool size of over 10k test cases. In this experiment, our sample data-set comprised of 150k LOC belonging to 12 test cases, with 46% of the lines belonging to the 'passed' class and 54% to the 'failed' one. 4.3
The subjects of our study are samples of the original data-set. The strati ed cross-validation technique was used to partition the data-set into 15 di erent subsets (k=15), such that the representation of the binary strata have approximately an equal representation across the 15 samples. Each subset consisted of 9200k LOC for validation and approximately 140k LOC for training. The representation of the binary classes in each fold followed the same distribution of classes in the original base set with 46% of lines belonging to the 'failed' class and 54% to the 'passed' class. 4.4
In this study, we used an open source measurement tool [
In our study, we used the Continuous Bag-Of-Words (CBOW) variant of the
Word2Vec word embedding algorithm proposed by Mikolov et al. [
All the vectors representing the 15 subsets of code churns were categorized into
two groups, one group containing the BoW vector les and another for the
Word2Vec generated outputs. To evaluate the e ect of WE, we ran 15 trials of
training a random forest model on the BoW vector representations in the BoW
group and another 15 trials for training a CNN model on the les in the WE
group. Our choice of training a random forest model on the BoW vectors is based
on the fact that RF is known for performing well with high dimensional data,
such as those generated by BoW transformations [
To decide whether to use a parametric or non-parametric statistical test, we checked if the data sample was normally distributed. We plotted frequency histograms for the precision and recall scores of both models (RF and CNN) for int \space a \space = \space 10
;
Encoding tokens with their identifiers from the vocabulary 1 2 3 2 4 … 0
… the 15 folds and examined the distribution of the points. We decided to run a Shapiro-Wilk test to check if the distribution of points follow a Gaussian curve. The test results were statistically signi cant for the RF models when using BoW (Precision with BoW: Test statistic = 0.484, p-value = 0.000, Recall with BoW: Test statistic = 0.538, p-value = 0.000), which means that the assumption of normality in both samples can be rejected. Conversely, the results of precision and recall with WE suggest that the distribution of both samples were normal (Test statistic = 0.929, p-value = 0.262, Recall with WE: Test statistic = 0.893, p-value = 0.075). Since the statistical results of the Shapiro-Wilk test suggest that we have issues with normality in the precision and recall results, we decided to run a non-parametric test for comparing the di erence between the precision and recall scores obtained by both models. The Mann whitney rank-based test was selected as an appropriate method since it can handle skewed data where the data does not follow a normal distribution.
We found a signi cant di erence between the precision and recall scores for both the RF and CNN models. The results of the comparison for the precision scores showed a test statistics of 12.5 and a p-value below 0.001. Similarly, the comparison between the recall scores for the same models reported a test statistics of 32.5 and a p-value of less than 0.001, suggesting a signi cant di erence in the recall metrics. Table 1 summarizes the mean scores of the 4 performance met
Input Embedding layer Convolution 1D Indices of tokens Embedding vectors layer padded with zeros of tokens
Max Pooling 1D
Convolution 1D layer
Max Pooling 1D
Flatten Dropout Dlaeynesre rics (precision with BoW, precision with WE, recall with BoW, and recall with WE). The results show a higher mean precision and recall for the RF model than those obtained by the CNN model. This brings us to believe that using BoW for features extraction in the MeBoTS is more e ective than WE in this context. In this paper we have only used a single industrial data-set that belongs to software, while other industrial software written in di erent languages and domains might reveal considerably di erent results. This was a design choice as we wanted to understand the dynamics of test execution and be able to use statistical methods alongside the machine learning algorithms. However, we are aware that the generalization of the results for di erent types of systems require further investigations using tests and churns from di erent systems. Another limitation comes from the randomness in selecting code churns and test cases without being ascertained about the nature of their failures. For example, there is a chance of encountering one or more tests that had failed due to non-functional related issues, for instance, a machinery failure at execution time. Likewise, the possibility of having tests that failed due to defects in the test script code and not the base source code exists. To minimize this threat, we collected data for multiple tests, thus minimizing the probability of identifying tests which are not representative. 7
This section provides our recommendations to practitioners who would like to use the MeBoTS method: { We recommend practitioners to try a variation of classi cation models with di erent hyper-parameter tuning to assess models' e ectiveness when using the MeBoTS method. { Using a BoW based classi er in the MeBoTS method surpasses that used with WE, and therefore, we recommend the use of the simple BoW modelling technique for extracting features from code churns for training a classi er. 8
In this study, we experimented with a set of industrial code churns and test execution results the e ectiveness of using WE as an alternative feature extraction approach to BoW in the MeBoTS method. By conducting a total of 30 trials of training and validating two prediction models on the BoW and WE vector representations, we empirically compared the di erence between the models' precision and recall scores. The results con rm with statistical signi cance (p-value less than 0.001) that modelling code churns with BoW results in higher prediction performance as compared with a WE-based model. In terms of future work, more empirical studies with larger industrial data are needed to validate the e ectiveness of both techniques in the context of MeBoTS. Moreover, training a series of word embeddings using a di erent variation of parameters such as the vector and window sizes is needed to draw more conclusive results about the e ectiveness of WE in this context.