Well-written bug reports should encompass bug descriptions, reproduction steps, environment details, and solutions. Bug report summaries also need to include such information to be highly informative for developers. However, traditional bug summarization techniques only apply summarization techniques to bug reports, and the generated summaries do not contain such information in a balanced way. In this paper, we propose summarizing duplicate bug reports by including bug descriptions, reproduction steps, environment details, and solutions. For that, our approach combines a supervised classification approach with the pre-trained summarization model BART. Additionally, we performed comparative experiments to demonstrate the efectiveness of this new approach in comparison to existing Summary and Authorship datasets. The experiments reveal that our approach outperforms the state-of-the-art method, achieving a 5% and 7% higher F-score for the Summary and Authorship datasets.
CEUR ceur-ws.org
Efective bug reporting is crucial in software development, requiring comprehensive bug reports that encompass detailed bug descriptions, reproduction steps, environmental specifics, and proposed solutions. Bug report summaries, often the first point of reference for developers, should similarly provide this essential information.
Previous work considered observed bug behaviors, steps to bug reproduction, and possible
solutions as the key information[
This paper presents an innovative approach to summarizing bug reports and generating
structured summaries, ensuring that specific attributes are extracted and presented comprehensively.
In that case, well-organized bug report summaries containing essential information may help
to navigate lengthy bug reports quickly[
In our work, we describe four attributes to be included in final bug report summaries: Description and the possible Solutions of the bug, its Environment, and Reproduction steps. Thus, we provide a new approach to generate summaries where specific details are extracted and presented in a structured way. The approach would be convenient for developers to process clear, organized information.
Initially, we employ a supervised classification method with domain-specific features to
extract environment and reproduction steps from the original bug report. Subsequently, we
calculate scores for each sentence in the remaining text after classification. We use these scores
to rank and select candidates for inclusion in the final summary, generated using the pre-trained
summarization model BART[
To assess our approach’s efectiveness, we evaluated accuracy and readability across two
datasets, including the widely-used bug report Summary (SDS) and Authorship (ADS) datasets[
The rest of this paper is organized as follows. Section 2 describes the Related Works. Sections 3 and 4 describe the Proposed Approach and the Experimental Setup. Section 5 discusses the results. We conclude in Section 6.
Rastkar et al.[
Rastkar et al. consider classifiers as an approach and uses them directly for summarization, while in our work, we utilize classifiers as a pre-step for summarization. Notably, we evaluate our data on a bug corpus provided by their study.
Mani et al.[7] challenge the eficiency of supervised models for summary generation,
emphasizing their reliance on large data and training limitations. They introduce an innovative
unsupervised approach comprising noise removal and summarization stages. Before
summarizing issue reports, the original text is categorized into groups like Question, Code, Investigation,
and Others. Their classifier module employs four methods: Centroid, Maximum Marginal
Relevance (MMR), Grasshopper, and Diferent Rank. Their evaluation demonstrates that
unsupervised techniques outperform the supervised method[
In this research, we also depend on a noise removal stage. However, the classification methods we employ difer in terms of techniques and methodology. While we classify sentences in bug reports based on their attribute consideration, Mani et al. focus on the inherent features of the sentences themselves.
In the context of mining software repositories (MSRs), the efectiveness of controlled
approaches strongly relies on attributes, which diferentiate data. However, no systematic analysis
has examined how to create new attributes for bug report summarization. Jiang et al. [8]
introduce a unique method involving crowdsourcing for supervised summarization, identifying
11 new text features. Unlike prior work[
In their study, they employ various ranking techniques to identify textual similarities among multiple sources using feature-based approaches. In contrast, we prioritize ranking sentences based on their topic relevance, thus selecting candidates for subsequent summarization.
Li et al. [9] underscore the significance of bug report characteristics, which they believe
can afect summarization technique efectiveness. Bug reports involve conversations and
include diferent types of sentences in both natural and software languages. They represent
the predefined fields with important information for identifying key sentences. To solve
these challenges, they introduce the DeepSum approach, an unsupervised auto-encoder model.
DeepSum leverages deep neural network hidden layers to compress input feature vectors from
bug report sentences and selects summary sentences based on the weights assigned by sentence
selection algorithms. Compared with the current study, these are based on sentence types of bug
reports, whether they are written in natural language or software sentence language. Extensive
experiments demonstrate that DeepSum significantly outperforms comparative algorithms
[
Huai et al.[10] introduced an intention-based bug report summarization approach (IBRS), defining seven key intention categories within bug reports. They expanded the Ratskar BRC corpora into Intention-BRC for their evaluation, examining the connection between bug report summaries and the underlying sentence intentions. This strategy led to an improved bug report summarization method incorporating sentence intentions. The authors established an intention taxonomy for bug reports, categorizing intentions into seven groups: Bug Description, Fix Solution, Opinion Expressed, Information Seeking, Information Giving, Meta/Code, and Emotion Expressed. They proposed a mixed model combining linguistic pattern matching and machine learning to classify sentences based on their intentions. IBRS leverages implicit information within sentence intentions, potentially enhancing bug report summarization in software development and debugging. However, the authors must consider factors such as extensive data and validation, potential computational complexity, and the risk of readability and information retention issues in summary rearrangement.
In contrast to Huai et al.’s intention-based approach, our work adopts a classifier-based strategy, prioritizing the extraction and summarization of bug report attributes rather than sentence intentions. Additionally, while Huai et al. emphasize the connection between bug report summaries and sentence intentions, our research centers on the relevance of sentences to the issue report topic, resulting in distinct approaches to bug report summarization.
Nawaz Tarar et al.[11] introduce a technology akin to the DeepSum approach[9], employing
an auto-encoder. Their method calculates sentence similarity through sentence embedding,
followed by K-means clustering to organize sentences into clusters. Sentence ranking is utilized
to select high-ranking sentences based on their information content for summarization. The
critical diference between their work and ours is the utilization of diferent baselines in sentence
selection. They are incorporating additional clustering techniques resulting in significantly
better F-score and Rouge metric performance compared to previous approaches [
Liu et al.[
Although, in the current study, we utilize the same score presented in the work of Liu et al., we use it in diferent contexts of reducing the number of noisy sentences with a combination of topic relevance scores. Thus, we can significantly reduce the number of unpleasant sentences.
Gupta et al.[12] emphasize the significance of timestamps, sentences related to the topic, authorship, and noise filtering. They dissect textual features to separately select comments and sentences based on hypotheses, ultimately categorizing the extracted features into commentspecific and sentence-specific features. They prioritize finding critical comments first and then apply selective sentence-specific features to identify crucial sentences within those comments. Thus, the authors paid attention to selecting sentences regarding some sentence specifications. Combining description and comment summaries enriches the overall summaries. In contrast, we emphasize the relevance of sentences to the issue report topic and use weight assignment techniques to filter noise. However, the approach’s dependency on the dictionary can be a weakness, as bug reports often contain references, jargon, and domain-specific language that may need to be adequately covered by the chosen dictionary, potentially leading to subpar summaries.
Koh et al.[13] address the limitations of existing bug report summarization methods, which
rely on training data or word frequencies and often need to improve in producing high-quality
summaries. They aim to provide a more versatile and practical approach by introducing sentence
significance factors. These factors serve as criteria for identifying meaningful sentences in
bug reports. The proposed deep-learning-based method utilizes sentence-to-sentence cohesion,
topic association, and believability. In contrast, our approach combines the topic association
factor with additional score assessment to identify sentences that align with bug description
and behavior. Experimental results demonstrate that their approach outperforms the
stateof-the-art BugSum[
In their work, Demeter et al.[14] presented an insightful analysis of customer-facing summarization systems, emphasizing the superior performance of BART over PEGASUS and T5 in both automatic and human evaluations. They discovered that summarizers exhibit significantly diminished performance when applied across diferent domains. However, a noteworthy finding was that a system fine-tuned on heterogeneous domains displayed robust performance across all domains, rendering it an ideal choice for a broad-domain summarizer. This work underscores the necessity for heterogeneous domain summarization benchmarks and unveils substantial variability in system outputs, a phenomenon best discerned through human evaluation, which often eludes standard leaderboards reliant solely on automatic metrics. Furthermore, their human evaluation revealed a strong preference for BART summaries, particularly those trained on mixed data, even surpassing summaries generated by BART fine-tuned on in-domain data, a preference that defies automatic metric capture. These insights solidify BART’s position as a leading model for customer-facing summarization systems.
Mercan et al.[15] conducted a study on text summarization, which involves condensing large amounts of text into concise summaries. The research involved training an LSTM model and ifne-tuning BART, T5, and PEGASUS using a resume dataset, with BART-Large emerging as the top performer based on the highest ROUGE score. Additionally, they used open-source datasets for training and fine-tuning. These findings highlight BART and BART-Large’s significance in text summarization, particularly in resume classification tasks.
Compared to our research, studies of Demeter et al. and Mercan et al. advocate fine-tuning as the primary procedure for harnessing summarization models. Our study emphasizes preprocessing steps preceding summarization to make bug reports more suitable for a generic model, as fine-tuning could require significant resource allocation.
Overall, the novelty of our work lies in the approach to bug report summarization. While previous research has focused on sentence intentions or linguistic patterns, our contribution is summarizing bug reports by explicitly considering their attributes. Although we employ similar scoring and weight assignment techniques that have been utilized in the past, the key innovation is in using a classification process as the primary domain-specific stage. This classification process helps select and filter sentences, ensuring the resulting summaries are highly relevant. Furthermore, by incorporating pre-trained models for the summarization task, we enhance the eficiency and accuracy of the bug report summarization process, contributing to a more efective and targeted approach to software development and debugging.
Our proposed summarization approach consists of four key steps. To commence, we initiate data preprocessing through three primary stages: removing stop words, word stemming, and vectorization. Following this, we employ a pre-trained random forest classifier to identify sentences within bug reports relevant to the Environment and Reproduction Steps domains.
Subsequently, we proceed with the scoring process for the remaining sentences in the text,
implementing an initial noise reduction step. This scoring process relies on two core features,
as previously outlined by Koh et al. [13] and Liu et al.[
In the final step, we merge the generated summaries with the sentences extracted during the classification process. Figure 1 illustrates the approach scheme.
Our data preparation process initiates by extracting textual information from bug reports. After this, we perform essential preprocessing steps. Tokenization breaks the text into discrete units, facilitating subsequent analysis. Next, we remove common stop words to streamline the data.
After refining the preprocessed text, we move on to a crucial domain feature extraction phase. Here, we employ BERT-as-a-Service, transforming our text into numerical vectors. These vectors capture essential attributes and intricate patterns, driving our subsequent analysis.
Our meticulous data preparation encompasses extraction, tokenization, stop-word removal, and domain feature extraction. Each step aims to ensure data integrity and utility for our research and analysis.
In the second stage of our approach, we leverage a pre-trained Random Forest classifier, as detailed in Section 3 of our methodology. We subject each sentence within the bug reports to meticulous analysis during the classification process, using the preprocessed sentences from the preceding text preprocessing step as inputs.
This step categorizes the sentences into two sets, each associated with the Environment or Reproduction Steps groups. These classified sentences play a pivotal role in our subsequent summary generation process, ensuring we efectively convey essential information related to Environment and Reproduction Steps in the final summaries. Thus, as the output of the second step of the approach, we get three sets of sentences. The first two are sets of classified sentences for Environment and Reproduction Steps, respectively, and the third set is the remaining text sentences. The last one will pass to the following sentence ranking Step, while the first two remain until summaries are generated.
The Random Forest (RF) classifier, a widely adopted machine learning algorithm, functions by constructing an ensemble of decision trees during training[16]. Each decision tree is crafted using a random subset of the training data and a random subset of the features, thereby introducing variability into the individual trees. The ensemble strategy consolidates the predictions from these diverse decision trees, typically through a majority voting mechanism in classification tasks. We employed the RF classifier as our predictive model in our specific classification task for several compelling reasons. First and foremost, its simplicity and ease of integration make it an ideal choice for the task at hand. Random Forest operates efectively without requiring extensive hyperparameter tuning, making it straightforward to implement in the classification framework[17].
Moreover, its versatility and robustness make it a commonly employed algorithm in classification tasks across various domains. The model’s ability to handle both categorical and numerical features and its capability to mitigate overfitting further underscores its suitability for binary classification in this context. The primary aim of this model was to acquire insights and identify patterns within a labeled dataset comprising sentences along with corresponding ’Environment’ or ’Reproduction Steps’ labels.
To enhance the learning process, we introduced a feature extraction method that generated sets of eight and five informative features for each sentence’s textual content, corresponding to ’Environment’ and ’Reproduction Steps,’ respectively. The features were derived from analyzing various bug reports, considering common patterns and tendencies observed in the language used. We chose these features to capture critical attributes such as sentence length, numerical content, and other pertinent characteristics, as detailed in Table 1. The rationale for deriving each feature is grounded in the observation that bug reports often exhibit consistent patterns, such as short lengths in environmental sentences, the inclusion of numerical data, the presence of fixed common phrases and keywords, the absence of questions, and other identifiable traits. While
To count of words in the sentence.
Presence of a ’:’ colon in the sentence.
To capture whether the sentence starts with key phrases like ”OS version” or ”OS” common in bug reports.
To identify documents that match specific keyword criteria while considering document length. (often shorter for environment descriptions).
To identify of key phrases in the sentence, considering the presence of numbers and digits (common in bug reports).
To detect special characters (’|’) To determine whether a given text contains a version number or a product name.
Filtering out sentences that contain question marks, indicating queries or uncertainties.
To capture whether a given sentence in a text document has a verb that is directly followed by a direct object.
To identify whether a sentence in a text document is written in the imperative mood by checking if it contains a verb in the base form that serves as the root of the sentence’s syntactic structure.
To identify whether a sentence in a text document contains a verb that matches a specific verb form pattern and is the root of a clause.
To identify whether a sentence starts with a pattern that resembles a numerical enumeration or list item. we acknowledge that individual projects may exhibit unique features, we focused on specifying the most common and evident ones based on our analysis. Subsequently, we partitioned the dataset into training and testing subsets, with 80% of the data allocated for model training and the remaining 20% reserved for model evaluation.
The Random Forest classifier underwent training on the designated training data, enabling it to establish connections between the extracted features and the associated ’Environment’ and ’Reproduction Steps’ labels. This training process empowered the model to make predictions regarding labels for new sentences based on their feature representations. After training, we serialized the model to a file for future use, and we also saved the feature extraction method to ensure consistent data preprocessing for inference tasks.
In the subsequent step of our process, we delve into the crucial ranking phase, where we employ
two key features sourced from the works of authors [13] and [
In the final step of our process, Summarization, we take the ranked sentences from the previous phase and utilize a pre-trained BART model to generate a concise summary. We aim to create summaries encompassing essential information, including issue descriptions and workarounds with possible solutions. Following the generation of the summary, we seamlessly integrate it with the two sets of classified sentences obtained from the Classification step. This grouping ensures that our summaries capture the core elements of the bug report and provide comprehensive insights into issue descriptions and potential solutions.
As a summarization model, we leverage the capabilities of the Bidirectional and AutoRegressive Transformers model (BART). BART is a pre-trained transformer-based model known for its proficiency in various text generation tasks, particularly in abstractive text summarization. While we can perform fine-tuning processes on BART for tasks specific to particular domains, like bug report summarization, we opt not to engage in this pre-training and fine-tuning procedure. We decided because the first two steps efectively addressed specific aspects of the bug reports, making the input data readily understandable for the model after this initial cleaning process.
• RQ1. What is the performance of our approach in classifying bug report attributes? Our initial research inquiry is centered on assessing the performance of our proposed approach in classification tasks. We aim to evaluate its accuracy and efectiveness in classifying data and compare its performance against established benchmarks. This evaluation is essential for gauging its suitability in real-world classification scenarios. • RQ2. How efectively does our approach perform in terms of bug report summarization? We seek to assess the clarity and comprehensibility of the summaries generated by our approach and compare them to existing summarization baselines. This evaluation is essential to gauge the practical usefulness of our approach in conveying information clearly and efectively.
To show the efectiveness of the proposed approach, we conduct an evaluation procedure with two Summary (SDS) and Authorship (ADS) Datasets. Thus, we aim to show comparative results and the reliability of the approach. Datasets comprise 36 and 96 bug reports from four open-source projects, namely Eclipse, Mozilla, KDE, and Gnome, each for SDS and ADS, respectively.
To make the evaluation possible, we modified the ground truths of each Dataset. Earlier works in bug report summarization did not consider attributes of bug reports separately as we did in our current work. Thus, there is no ground truth for them. To mitigate this, we add sentences that may contribute to “Environment” or “Reproduction Steps” attributes in both SDS and ADS datasets, modifying the golden summaries to contain original ground truth with an extended set of sentences.
We primarily rely on BugSum [
In response to research question RQ1, our primary goal is to demonstrate the efectiveness of our chosen approach in classification. We gather classified sentences alongside the previously identified golden sentences in the same file for evaluation. This approach allows us to compute the accuracy for each bug report, and we subsequently determine the average accuracy across all 21 bug reports. We assess performance using three key metrics: Precision, Recall, and the F-Score.
For research question RQ2, our focus is on assessing the accuracy of the generated summaries. After generating the summaries, we combine them with the previously classified sentences from the initial phase, resulting in final summaries that consist of three essential segments. We then utilize the ROUGE toolkit to obtain numerical evaluation results for each bug report and calculate the Accuracy scores. In the final step, we calculate the entire dataset’s average readability and accuracy scores.
Recall measures the efectiveness of a generated summary in capturing and encompassing vital details from the reference summary. It quantifies the ratio of pertinent content in the generated summary to the total pertinent content within the reference summary. In the context of bug report summaries, a high recall reflects the capacity to accurately capture and integrate essential information.
Recall = ℎ
⋂ ℎ
Precision assesses the accuracy and relevance of information in a summary. It determines the ratio of pertinent content in the summary to the total content. In the context of bug report summaries, high precision signifies that the summary efectively concentrates on crucial details of the bug report.
Precision = ℎ
⋂
The F1 score ofers a well-rounded evaluation by considering both precision and recall. It represents the harmonic mean of these two metrics and contributes to an all-encompassing assessment of the quality and efectiveness of summaries, bug report summaries included. This balance is evaluated comprehensively, considering the interplay between precision and recall.
F − score = 2 ∗ ∗ (3)
+
These three metrics above are calculated from the selected sentence set Chosen and the golden standard sentence set GSS to measure the accuracy of the summaries.
We apply the ROUGE toolkit to evaluate our experimental results from the position of readability. The tool measures the quality of generated summaries by counting continuously overlapping units between the summaries and the ground truth. As n-gram ceiling units in the ground truth, we use 1 and 2.
Rouge − n = ∑
∑− ∑− ( − ) ℎ ( − )
Here, refers to the generated sentence, refers to the reference summary, and − is a contiguous sequence of n items as words and characters within a sentence. The numerator counts the number of n-grams between the generated and golden summaries. The denominator counts the number of the n-grams in the golden summary.
Table 2 displays the experimental results for classification, showing varying performance across diferent attributes and datasets. When classifying the Environment attribute in the SDS dataset, the approach achieved a precision of 0.75 and a recall of 0.93, resulting in an F-score of 0.73. Similar precision (0.84) and recall (0.83) values in the ADS dataset led to the same F-score of 0.73. However, for the Reproduction Steps attribute, the SDS dataset yielded a precision of 0.81 but a lower recall of 0.55, resulting in an F-score of 0.5. In contrast, in the ADS dataset, the precision (0.84) was higher, and the recall (0.91) was significantly better, resulting in a notably higher F-score of 0.77. The inherent characteristics of each dataset contribute to the discrepancy. (1) (2) (4)
The SDS dataset comprises 36 bug reports, primarily characterized by a higher noise level. These reports often contain sentences from patches within the bug report, making them prone to misclassification as ’Environment’ or ’Reproduction Steps’ sentences. As a result, the classification model on the SDS dataset demonstrated a lower recall (0.55), indicating a higher likelihood of misclassifying sentences and counting non-sentences in the ’Reproduction Steps’ category.
In contrast, the ADS dataset comprises 96 bug reports that, in comparison to the SDS dataset, are generally shorter, predominantly written in natural language, and frequently lack explicit information about the ’Environment’ and ’Reproduction Steps.’ This characteristic of the ADS dataset prevents misclassification and contributes to a higher recall (0.91), leading to a significantly better F-score of 0.77. These results underline the influence of dataset quality on classification performance, and the presence of noisy bug reports may contribute to lower recall rates.
However, when considering the Rouge-n results, the Proposed Approach registers relatively lower scores when compared to the original BugSum. The reduced Rouge-1 and Rouge-2 scores imply that the generated summaries may not faithfully replicate the exact phrases and sentences found in the original bug reports. It is essential to highlight that this contrast arises because the Proposed Approach utilizes an abstractive methodology, while BugSum employs an extractive one.
The evaluation results demonstrate that the Proposed Approach excels in bug report summarization, exhibiting superior precision, recall, and F-score. These results underscore the efectiveness of the Proposed Approach in generating informative and relevant summaries and suggest its suitability for the task at hand.
Overall, this paper has introduced an innovative approach to bug report summarization designed to extract and present specific attributes comprehensively. Our approach initiates with the application of a supervised classification method, leveraging domain-specific features to extract environment and reproduction steps from the original bug report. Subsequently, sentence scores are computed for the remaining content post-classification, and these scores are employed for ranking and selecting candidates to be included in the final summary, generated with the aid of the pre-trained summarization model, BART.
To validate the efectiveness of our approach, we conducted a comprehensive evaluation across two datasets, specifically the widely-used bug report SDS and ADS datasets. Our results have proven promising, with remarkable improvements demonstrated, showing an impressive increase of 5% and 7% in F-scores for SDS and ADS, respectively. This robust performance underscores the superior accuracy of our approach when compared to baseline models. Overall, this work represents a significant step in enhancing bug report summarization eficiency and precision in the software development landscape. [6] J. He, J. Zhang, H. Ma, N. Nazar, Z. Ren, Mining authorship characteristics in bug repositories, Sciece China. Information Sciences 60 (2017) 1–16. doi:10.1007/ s11432-014-0372-y. [7] S. K. Kumarasamy Mani, R. Catherine, V. Sinha, A. Dubey, Ausum: Approach for unsupervised bug report summarization, 2012, p. 11. doi:10.1145/2393596.2393607. [8] J. He, N. Nazar, J. Zhang, T. Zhang, Z. Ren, Prst: A pagerank-based summarization technique for summarizing bug reports with duplicates, International Journal of Software Engineering and Knowledge Engineering 27 (2017) 869–896. doi: 10.1142/ S0218194017500322. [9] X. Li, H. Jiang, D. Liu, Z. Ren, G. Li, Unsupervised deep bug report summarization (2018) 144–14411. [10] B. Huai, W. Li, Q. Wu, M. Wang, Mining intentions to improve bug report summarization, 2018, pp. 320–363. doi:10.18293/SEKE2018-096. [11] M. I. Nawaz Tarar, F. Ahmed, W. H. Butt, Automated summarization of bug reports to speedup software development/maintenance process by using natural language processing (nlp) (2020) 483–488. doi:10.1109/ICCSE49874.2020.9201846. [12] S. Gupta, S. Gupta, An approach to generate the bug report summaries using two-level feature extraction, Expert Systems with Applications 176 (2021) 114816. doi:10.1016/j. eswa.2021.114816. [13] Y. Koh, S. Kang, S. Lee, Deep learning-based bug report summarization using sentence significance factors, Applied Sciences 12 (2022). URL: https://www.mdpi.com/2076-3417/ 12/12/5854. doi:10.3390/app12125854. [14] D. Demeter, O. Agarwal, S. B. Igeri, M. Sterbentz, N. P. Molino, J. M. Conroy, A. Nenkova, Summarization from leaderboards to practice: Choosing A representation backbone and ensuring robustness, CoRR abs/2306.10555 (2023). URL: https://doi.org/10.48550/arXiv. 2306.10555. doi:10.48550/arXiv.2306.10555. arXiv:2306.10555. [15] Mercan, S. Cavsak, A. Deliahmetoglu, S. Tanberk, Abstractive text summarization for resumes with cutting edge nlp transformers and lstm (2023). [16] V. Kulkarni, P. Sinha, Random forest classifiers: A survey and future research directions,
International Journal of Advanced Computing 36 (2013) 1144–1153. [17] N. Kanvinde, A. Gupta, R. Joshi, Binary classification for high dimensional data using supervised non-parametric ensemble method (2022).