As bug reports are often duplicated, researchers have proposed detecting and classifying duplicate bug reports. Researchers also utilized the duplicated relationships of bug reports to summarize a bug report. However, to the best of our knowledge, the previous bug report summarization is limited to single document summarization techniques. In this paper, we propose applying a multi-document summarization technique to duplicate bug reports so as to obtain more informative summaries. In our work, we demonstrate the results of summarizing duplicate bug reports with our multi-document summarization technique and discuss our future research direction.
If we generate good summaries of any text, we can easily understand and analyze the text. Up to now, various techniques and methods have been developed for text summarization. However, there are no universal methods that show high performance, because, according to a document type, text features to be used for text summarization will be different. Therefore, we need to focus on a specific document type to improve the quality of final summary results.
Bug report summarization is one of areas that
require text summarization methods. Many
researchers have conducted studies in this area.
However, few have investigated the impact of
duplicate bug reports on bug report
summarization. On the one hand, duplicates are
much desirable, so developers and researchers
sought to reduce the number of duplicate bug
reports. Meanwhile, we believe that, to some
extent, duplicates may contain additional
information that can be useful for developers to
perform tasks such as localization and bug
detection [
Therefore, we sought to apply an existing
multi-document summarization method [
The rest of this paper is organized as follows. Section 2 describes the background of used algorithms and methods. Section 3 describes applied approaches. Section 4 discusses results. We conclude in Section 5.
In this section of the work, we would like to briefly explain methods and algorithms used in the approaches.
The Term frequency–inverse document frequency (TF-IDF) is a numerical statistic that reflects how important a word is to a document in a collection or corpus. The TF-IDF value increases proportionally to the number of times a word appears in the document. It is offset by the number of documents in the dataset that contain the word, which helps to adjust for the fact that some words appear more frequently in general. 2.2.
The centroid-based method is often used in
text summarization to determine salient sentences
in a document set. A sentence vector is
represented based on the TF-IDF of containing
words. A word can be a centroid of the sentence
if its TF-IDF value is greater than a given
threshold. To summarize, creation used sentences
containing multiple centroid words. In text
summarization, the centroid-based method
eliminates the information overlap, in summary,
using the cosine measure [
The original Maximal Marginal Relevance
(MMR) solves the information retrieval problem
to measure the relevance between the user query
and sentences in the document [
3.1.
The first approach was proposed by Hai et al.
[
The first module includes stop words removal, word stemming, and converting input sentences into a vector. A bag of words is used as the most straightforward way for vector representation. However, the method of using a bag of words does not contain the semantic meaning of the sentence, so Word Embedding is integrated into a system to help compute the semantic relationships among sentences. As a model, they used Google's pretrained Word2vec. Regardless of the architecture of the embedding model, the primary function is to take the data as input and try to predict words. Total embedding vectors are the input for the next module.
The second module starts with the k-mean algorithm, which groups input embedding vectors of sentences from input documents into clusters. As a result, we have candidate Sentences that can include in the summary. However, not all the cluster sentences are appropriate and may be too poor, so the centroid-based method is applied to get the most critical sentences among the output sentences of k-means. Then, the MMR is used to remove redundancy sentences from the output of the centroid-based module. Finally, information about sentence positions is used to put selected sentences in the correct order in summary. 3.2.
The second approach is more simplified,
which consists of the TF-IDF word frequency and
the Relative sentence location in documents [
For the preprocessing, the approach uses the standard NLTK library. This step contains word tokenization and stemming.
The second step of the approach assigns scores to each sentence in the input document based on the sum of text feature values. Features are relevance to the topic, length of the sentence, sentence position, and TF-IDF frequency value. Following the approach, the most important sentences are from the input document's beginning and end. Also, short sentences are not as valuable as long ones. Sentences with the highest sum of feature scores are selected to be in the final summary.
To evaluate the quality of generated summaries, we take duplicates of 17 bug reports. Among them, only two bug reports contain more than two duplicates. Overall, we applied 35 summaries. During the analysis of generated summaries, we noticed no difference between those with two duplicates and those with three or two. Thus, the number of duplicates doesn’t make a difference in bug report summarization. 4.2.
We apply the ROUGE toolkit to evaluate our experimental results from the position of readability. The tool measures the quality of generated summaries by counting continuously overlapping units between the summaries and the ground truth. As n-gram ceiling units in the ground truth, we use 1 and 2.
In Formula 1, the denominator is the number of the n-gram in the ground truth (GT), and the numerator is the number of n-gram ceiling units between generated summaries (s) and GT. − = ∑ ∈ ∑ ∑ ∈ ∑ ∈ ∈ ( ( ) ) (1)
As for the quantitative one, Fig. 3 illustrates average Rouge-1 and Rouge-2 scores for the data on two approaches. Regarding the figure, Approach 1 got 0.52 and 0.33 for R-1 and R-2, respectively. Approach 2 shows higher results for 2% and 10% for R-1 and R-2, respectively. However, this may be since the information generated by the second model are slightly larger in volume. At the same time, the results of the first one are somewhat shorter compared to the golden summary.
We provide qualitative results to understand how length can affect the quality and overall content of generated summaries. Fig. 2 illustrates the contents of three documents: the summary generated with the first method, the second method, and the human-annotated golden summary. The figure clearly shows a slight difference in the length of the reports compared to the golden summary. However, this does not significantly affect the quality of the summary since there are no repetitions of sentences that are identical in meaning, which is the primary concern when working with duplicates.
Since we aim to show the difference between
bug reports with duplicates and single bug reports,
we experiment with the T5 [
When summarizing the text, it is essential to
include important information such as the bug
behavior, steps to reproduction, and possible
solution [
Summarizing the results of the qualitative and quantitative analysis, we believe that the duplicates performed relatively well for summarization with models not explicitly trained for bug reports. Moreover, using duplicates for bug report summarization guarantees more informative summaries. It gives positive expectations for the future in case of using models adapted to bug reports with all text preprocessing steps, including text features.
We implemented the task of bug report summarization on two multi-document approaches. One is an unsupervised machine learning algorithm, and the second method includes a k-means clustering algorithm, combining with the centroid-based method, maximal marginal relevance, and sentence positions. To evaluate the performance of summarizing duplicate bug reports, we took 17 examples of bug reports with duplicated ones. Thus, we provided satisfying quantitative and qualitative results even with models not designed for bug reports. Our experiments have shown that duplicates can be excellent used to make summaries more informative, while multidocument summarization techniques can reduce the redundant contents. With such results, we will continue to improve the performance by developing multi-document summarization techniques for duplicate bug reports, where we will take into account all the features of documents.