Software defects can lead to severe issues in software systems, such as software errors, security vulnerabilities, and decreased software performance. Early prediction of software defects can prevent these problems, reduce development costs, and enhance system reliability. However, existing methods often focus on manually crafted code features and overlook the rich semantic and contextual information in program code. In this paper, we propose a novel approach that integrates JavaBERT-based embeddings with a CNN-BiLSTM model for software defect prediction. Our model considers code context and captures code patterns and dependencies throughout the code, thereby improving prediction performance. We incorporate Optuna to find optimal hyperparameters. We conducted experiments on the PROMISE dataset, which demonstrated that our approach outperforms baseline models, particularly in leveraging code semantics to enhance defect prediction performance.
JavaBERT and CNN-BiLSTM multiple times, and outputs
Researchers have explored various models for feature ex- the best hyperparameter values through these executions.
traction in software defect prediction, from traditional ma- Then we retrain the model in another version of the code
chine learning to deep learning. Initially, Support Vector based on the obtained hyperparameters and test the model
Machines (SVM), as employed by Elish et al.[
In short, SVM has difficulty discovering the deep semantics of the source code, DBN has limited depth so it is difficult to understand the complex relationships in the JavaBERT = EncoderJavaBERT(1, 2, . . . , ) source code, CNN has difficulty capturing long-distance correlations, and RNN and LSTM can only capture a sin- Models typically cannot process code text sequences gle temporal pattern. BiLSTM may have challenges in directly. Through JavaBERT, we embed code text into a capturing local patterns. continuous vector space, using these vectors as inputs to
To solve these problems, we combine the advantages the model, making it easier for the model to compute and of CNN in detecting local patterns with the advantages understand the code. of BiLSTM in processing sequences, allowing for comprehensive code inspection. We further incorporate Jav- 3.2. Feature Extraction using aBERT to dynamically adjust token embeddings based CNN-BiLSTM on the entire input sequence, thereby deepening the representation and capturing interdependencies among code tokens.
Bidirectional Long Short-Term Memory networks (BiLSTM) to extract features. This is the key part of our approach, where after extracting features with CNN, it is refined with the sequential capabilities of BiLSTM.
key steps, all aimed at improving prediction performance. 3.2.1. Feature Extraction with CNN
As shown in Figure 1, we first use JavaBERT to convert Utilizing Convolutional Neural Networks (CNN)[
)︃ where [, ] is the input at position (, ), [, ] represents the kernel at position (, ), is the bias, and signifies the activation function. 3.2.2. Refinement of Features with BiLSTM The
Bidirectional
Long
Short-Term
(BiLSTM)[
ℎ = BiLSTM(, ℎ−1 , ℎ+1)
STM) model. It is computed based on the input at the current time step, the previous hidden state ℎ−1 , and terns and connections over time, amplifying the feature representation. In summary, we refine the feature maps obtained from CNN using BiLSTM to achieve a comprehensive code representation. This fusion of capturing local patterns and accounting for temporal dependencies improves software defect prediction performance.
Our methodology integrates the refined BiLSTM outputs with statistical features (such as shown in Table 2) extracted from dataset. This step concatenates the vectors obtained from the BiLSTM and the vectors of statistical features obtained from the dataset into longer vectors, making full use of the description information of the code.
work developed by Akiba et al.[
In this section, we will discuss a crucial step in our methodology: determining optimal hyperparameters by leveraging shared features among different versions of the same project. Usually, code with similar version numbers exhibits a high degree of similarity. By harnessing these inherent similarities, we attempt to find hyperparameters that can generalize across various versions, ultimately enhancing model performance.
Using the Ant project as an example, our aim is to demonstrate the transferability of hyperparameters obtained from training on one version (e.g., 1.5) to another (e.g., 1.6). This transferability is valid as both versions originate from the same project, sharing similar code structures and functionalities. This enables the hyperparameters obtained from one version to serve as a foundation for other versions within the same project, thereby solidifying our model configuration.
We start by selecting version pairs, using the Ant project as an illustration. Here, we designate version 1.5 for training and version 1.6 for testing. Next, we deifne the performance metric to optimize, such as the F1 score. Subsequently, Optuna conducts multiple experiments, traversing various hyperparameter combinations and evaluating their performance on the designated testing dataset. Through these iterative experimentation and evaluation stages, Optuna determines the hyperparameter set that maximizes the chosen performance metric.
This process can be represented as:
comprised of Java projects. This dataset spans various
domains and project scales, providing project details like
name, description, version, and bug rate. Table 1 shows an
overview of the projects we use that are in the PROMISE
Java Dataset. Since Optuna’s process of finding
hyperparameters takes a lot of time, we only selected a part of the
projects in the PROMISE data set. Statistical features also
= Optuna (Ant 1.5, Ant 1.6) play a vital role in code analysis, offering insights into
Here, (Ant 1.5, Ant 1.6) embodies the objective func- code composition and behavior. To enhance our study, we
tion maximized during the hyperparameter optimization carefully selected a subset of these features, as shown in
process, with Ant 1.5 as the training dataset and Ant 1.6 Table 2.
as the testing dataset. After obtaining optimal hyperpa- To prepare the data for analysis, we conducted
thorrameters through the Optuna process, we seamlessly ough data preprocessing. Using the "javalang"[
This operation optimizes hyperparameters across ver- dom oversampling exclusively on the "Bug" class files. sion pairs, contributing to enhanced model adaptability This deliberate strategy generated synthetic data instances, and performance in varying project iterations. improving class distribution and mitigating potential bias towards the majority class.
For each project listed in Table 1, we selected the smallest two version numbers to serve as versions Y and Y+1 for Optuna’s hyperparameter optimization process. The search space for the hyperparameters was specified as shown in Table 3. The number of trials for each project was set to 30. After completing these experiments, each project will produce a different set of hyperparameters that allow the model to output the highest F1 score, and a model trained on these parameters using version Y. These hyperparameters were then applied to train new models on version Y+1 for each project. Then the model trained on version Y and the model trained on version Y+1 were evaluated against the code of version Y+2. We conducted each evaluation test three times and calculated the mean to obtain the experimental result.
model in comparison to baseline models. Table 4 presents a detailed performance comparison between our CNNBiLSTM model and the baseline models concerning precision, recall, and F1-score. For instance, "ant_1.5_1.6" represents the experimental results obtained by using version 1.5 of Ant as the training dataset and version 1.6 as the test dataset. The results demonstrate a consistent outperformance of our model across all metrics. Figure 2 complements the table by providing a visual representation of the F1 scores, where the x-axis represents pairs of software versions used for training and testing (e.g., ant_1.5_1.6), and the y-axis represents the corresponding F1 values obtained during testing. This figure shows that the F1 of our model is higher than the base model most of the time.
baseline models: To address RQ2, Figure 3 presents the F1 scores of our model across different projects and their respective versions in the PROMISE dataset. In this figure, the x-axis • Support Vector Machine (SVM): SVM, a classic represents pairs of software versions used for training and and widely adopted machine learning algorithm, testing (e.g., ant_1.5_1.6), while the y-axis represents the excels in both linear and non-linear classification corresponding F1 values obtained during testing. When tasks and is known for its effectiveness in handling we examined the model’s performance across different high-dimensional data. projects and its various versions, we observed certain noteworthy patterns. Specifically, within the same project, suggests that the predictive performance of our model tends to vary when applied to certain projects. Although we cannot pinpoint the exact reasons behind these changes at this time, we speculate that they may have been influenced by a variety of factors, including project-specific characteristics, code complexity, and domain-related differences. such as Lucene, POI, and Xalan, our models show a high degree of performance consistency across different versions. This shows that our model is able to predict results consistently when dealing with different versions of certain projects. This consistency can be partially attributed to the higher code similarity found between ver- Figure 3: F1 Score Across PROMISE Projects sions within the same project, making it easier for models to capture shared features and patterns.
There are some differences between versions of Ant and Synapse, these differences are relatively minor. In contrast, projects such as Camel and JEdit show more performance fluctuations, even within the same project. This
To address RQ3, in this section, we study the impact of hyperparameters on the performance of the CNN-BiLSTM model for code defect prediction. Initially, we set the hyperparameters to the following values: the number of epochs is 10, the batch size is 64, the learning rate is 1e-4, the number of CNN filters is 128, the number of BiLSTM hidden units is 256, and the CNN filter size is 5 . After that, we fixed other hyperparameters, and then gradually manually adjust one of the other parameters, the CNN iflter or the number of BiLSTM hidden units, to observe changes in model performance.
Figure 4 and Figure 5 show our experimental results, the x-axis is the change in the number of CNN filters Figure 4: Effect of CNN Filter Length on F1 Score and BiLSTM hidden units, and the y-axis shows the F1 score. We can see that the model performance fluctuates greatly when a single parameter changes. For example, the smaller the number of CNN filters, the better the performance of the model. In Figure 5, the F1 score drops after BiLSTM hidden unit is 16, but performs better and tends to be stable after 256. Exploring the impact of each hyperparameter individually would be a time-consuming task, and it is difficult to predict how the model will behave when these hyperparameters are combined. So we used Optuna, which will constantly try to search for hyperparameters that can make the model perform better based on the search algorithm.
Figures 6 and 7 show the F1 score (y-axis) for a certain number of trials (x-axis). Specifically, Figure 6 is a scatter plot, representing the F1 score that was obtained in each Figure 5: Effect of BiLSTM Hidden Units on F1 Score trial, e.g., when the trial number is 5, the F1 score is the value for the fifth trial. Figure 7 represents the best model performance that can be achieved based on the search until the current trial model is executed. So, in figure 5.4. Threats to Validity 7, when the trial number is 5, the F1 score is the best In our research, we have identified and addressed several F1 score from the first to the fifth trial. We can observe potential threats to the validity of our findings. that through continuous repetition and search, Optuna can The implementation of our Python experimental code gradually search for better results. The entire process is for processing source code text and building models poses automated, which greatly simplifies our hyperparameter a potential threat due to the possibility of bugs. To mitigate tuning process. this, we took measures by leveraging mature third-party libraries (such as javalang and PyTorch) and conducting thorough code inspections. Additionally, we applied random oversampling during data preprocessing, which could introduce bias. Future work will explore alternative methods to handle class imbalance and assess their impact on results. Moreover, the use of Optuna for hyperparameter optimization introduces potential variability in results due to different search spaces and numbers of trials. To reduce these threats, we plan to conduct more extensive searches and explore larger search spaces.
Our choice of a subset of projects from the PROMISE dataset due to time constraints may impact the generalizability of our findings, as the results may not generalize well to other projects. To address this, we intend to include a broader range of projects in future research.
We evaluated our models using a limited set of performance metrics, specifically precision, recall, and F1 measure. To reduce these threats, we will consider incorporating additional metrics such as AUC-ROC and MCC, among others, to provide a more comprehensive assessment of model performance.
that leverages JavaBERT-based embeddings with a CNNBiLSTM model for software defect prediction. Our approach harnesses semantic and contextual information in program code to enhance prediction accuracy. Through comprehensive experiments on the PROMISE dataset, we have demonstrated the superiority of our model over baseline models based on precision, recall, and F1-score metrics.
Although our study improves the performance of software defect prediction compared to baseline models, we still have many future works to do. In addition to what we discussed in the "threats to validity" session, we can also train the BERT model in different languages to adapt our methods to different programming languages.