The bug- xing process requires software maintenance resources. Usually, defects are submitted, xed and closed, but sometimes they have to be reopened because of a change of resolution. It happens when a defect was evaluated incorrectly at the beginning. This problem can increase maintenance costs and software quality in general. In this paper, we investigate the characteristics of such defects and their bug reports and call them \underestimated". Our research is based on general statistical indicators and text descriptions of defect reports. We propose using di erent methods of feature selection and ranking in order to reveal the signi cant terms of such defects. The top of signi cant terms of the underestimated bug reports can help to nd the root causes of such life cycles.
Defect management is an essential part of improving the technical stability of software. Usually, software companies use bug-tracking systems (BTS) in order to manage defects. Structured information about defects is a big advantage of a BTS, where a bug is represented as a set of attributes. Gathering data from bug reports allows us to accumulate statistics, we also use these data for predictions and analysis.
The knowledge of defect statistics and the defect management strategy helps
to create an e ective approach to defect prevention because it can help to
reduce bug migration into the later stages of development [
We would also like to notice that our work is devoted to revealing the reasons of underestimated defect reports more than their future classi cation. Our goal is to understand the nature of such bug reports via the feature selection and ranking methods. Thats why we mainly investigated the top list of terms.
We claim the following contribution in this work: { We reveal special characteristics of underestimated bug reports. { We propose using di erent methods of feature selection and ranking for determining the most signi cant terms of underestimated bug reports. { We conduct an analytical study to investigate the potential causes of the initial resolution of such bug reports via the most signi cant terms.
The remainder of this paper is organized as follows: in Section 2, we present an overview of the related work; in Section 3, we describe the structure of a defect report; in Section 4, we outline the process of clustering. Further, in Section 5, we present the results of the experimental evaluations of this technique; and Section 6 lists our conclusions. 2
There are many researchers who deal with analysis of bug reports which have certain speci cs. They investigate the nature of these speci cs, the root causes of their occurrence and ways to predict them.
Zaineb and Manarvi analysed the reasons of bug rejection in order to
decrease the possibility of submission of invalid bug reports [
Gegick, Rotella and Xie proposed to classify bug reports into security and
non-security bug reports [
The problem of classifying issues into defects and non-defects is very popular
too. Antoniol et al. [
This review of the related work demonstrates the importance of understanding the nature of di erent types of defects. The aforementioned researchers investigate various speci c types of bugs and their potential life cycle stages. In our work, we propose to consider the problem of underestimated defects as a special case of the reopened bugs problem. In this paper, we investigate the submitted bug reports which were reopened because, at some point, they were understood and evaluated incorrectly. We aim to reveal the causes of such situations in order to prevent them in the future and create possible recommendations. 3
Each bug report has its own life cycle. A life cycle depends on the characteristics of the software development cycle, the proprietary and the domain aspects of the project. During the life cycle, a defect report s priority, status, etc. can change.
For this research, we only considered closed and resolved defects because only such bugs have the values of Resolution, Time to resolve, Count of attachments, Count of comments, etc., known for a fact. For the defects that have not been closed or resolved, the values of these attributes are inde nite. We selected the defects where Resolution has such values as \Done" or \Fixed" because they look as ordinary. But then, we split them into two categories. The rst category | called \type 1" | includes bug reports that have never been reopened because of a change of resolution. It means that they have never been rejected or considered a non-defect or non- xable. The second category | called \type 2" | includes the defect reports that had an alternative resolution before they got a \Done" or a \Fixed" resolution. These alternative resolutions could have such values as \Rejected", \Won t Fix", \Not defect". In this paper, we called the bug reports of type 2 \underestimated".
We propose to compare the two types according to the following metrics:
{ time to resolve as an indicator of how expensive the defect report is [
Thanks to this comparison, we can reveal the distinguishing characteristics of underestimated defect reports. 4 4.1
We extracted 45,341 bug reports from three di erent project communities on
JIRA, a popular bug-tracking system. They include open-source projects of
JBOSS [
As Table 1 shows, defect reports of type 2 have distinctive characteristics. The bugs of type 2 require more time for xing than the bugs of type 1. They also have a larger number of collateral comments and attachments. These characteristics may be suitable for reopened defects or may not be suitable, such as in the case of trivial reasons of reopening like addition of labels. So we prove that defects of type 2 are \expensive-to-resolve" since they require a lot of human input and time. Therefore it is important to investigate and prevent them.
According to Table 1, defect reports of type 2 have \Blocker" to \Critical" priorities, just like type 1. It means that underestimated defects are important because they can lead to a situation when a Critical or a Blocker bug can persist in the system for a long time and undermine software quality.
The comparison of description lengths shows that the description of bugs of type 2 is more complicated than \scarce text". It means that they can have a detailed description, but, for some reasons, it was evaluated incorrectly.
We propose to analyse the description of bug reports as a source of answers as to why these defects became underestimated. 4.2
The description of defect reports is in text format, so it needs to be transformed via natural language processing methodologies. We made the following steps: { tokenization that chops the text into words, { removal of stop-words list that was expanded with names of months, week days, the submitter and assignee s names, parts of logs, stack traces, etc., { stemming that maps related words to their basic form and helps to reduce the in ectional forms.
Then, we built a \Bag of words" model. This vector model takes into account
the number of occurrences of each term, rather than the exact order of the terms.
Every bug report is presented as a vector of n terms. A set of bugs is presented
as corpora or matrix n m where n is the number of all terms, and m is the
number of all documents. If a term occurs in the bug-report, its value in the
vector is non-zero. We used TF-IDF weighting for computing these values [
T F IDF = T F (t; d) IDF (t; D) T F (t; d) =
f req(t; d) maxw2df req(w; d) IDF (t; D) = log2( d 2 DjD: jt 2 D ) (1) (2) (3)
Where freq(t,d) is term frequency, i.e. the number of times that term t
occurs in documentd ; maxw2df req(w; d) is the maximal frequency of any term in
document d ; d 2 D : t 2 D is the number of documents containing t ; D is the
corpus - the total document set [
We used the following methods of feature selection and ranking:
1) Chi-Squared is the common statistical test that measures the divergence
from the expected distribution, if one assumes that feature occurrence is actually
independent of the class value [
2) Recursive feature elimination (RFE) is a recursive process that ranks
features according to some measure of their importance [
3) Random forest is a method that builds an ensemble model of decision trees
from random subsets of features and bagged samples of the training data [
Where xi are the characteristics describing the model, 2 [0; 1] is a value
on the logistic regression curve [
In the original bug report dataset, we marked bugs as 0 if they belong to type 1 and 1 if they belong to type 2. We matched each indexed defect report with its class f0, 1g. This column was used by the feature selection and ranking techniques.
We built the top of the most signi cant terms with the aforementioned feature selection and ranking techniques that include chi-square, recursive feature elimination, features importance of random forest and coe cients of logistic regression. The top 15 of the most signi cant terms of JBOSS, Jenkins and Sakai projects is presented in Tables 2, 3, and 4. Chi2 RFE Random Forest Logistic regression cast , busi , import , cast , materi , capabl , color , request , osgi , connector , normal , event , busi , consequ , classexternallink , jar , constructor , day , lineheight , bundl , bundl , determin , condit , error , vdb , download , fonttyl , busi , network , end , error , osgi , comment , facet , le , open , request , includ , event , materi , jar , later , like , comment , spec , long , comment , constructor , lookup , network , fonteight , connect , redirect , osgi , properti , server , lot perform consol , vdb
Having analysed the received results, we propose to split them into the following groups of terms: Chi2 RFE Random Forest Logistic regression stdout , avoid , job , git , ssl , capac , build , server , dynam , correspond , error , password , testsuit , detect , con gur , stdout , password , ensur , password , error , wrapper , general , run , setup , certif , head , use , copi , stderr , increment , document , document , git , introduc , need , perforc , larg , jdk , testsuit , dynam , emailtext , listen , jenkin , upstream , perforc , previous , log , way , upstream , provis , poll , avail , setup , servic , follow , ndbug , gitssh strang long log Chi2 RFE Random Forest Logistic regression recommend , administr , user , recommend , desir , app , tool , appear , idea , applic , appear , addit , addit , breadcrumb , error , edit , retract , exit , recommend , resourc , exit , explicit , question , call , pool , role , use , desir , random , process , classexternallink , pool , uniqu , recommend , make , mean , person , retract , chang , entri , app , situat , info , inform , font , stay , click , idea , portfolio , trunk , screen , gradebook , audio , write , follow , creat , edit uniqu list requir 1) \Prejudiced term" are the terms that are connected with biased wording in the software bug description. The text seems to be full of subjective assessment or personal impression of the submitter. So the defect report may be considered a \fancy" or just a wild guess. Examples of such terms are: idea, recommend, desir, larg, strange, long, comment, etc. Below are some samples of use of these terms extracted from the defect reports under analysis: I think the idea is to . . .
I recommend . . . it would be more desirable . . . unprofessional comments . . . goes through after long time . . . large log le . . . there is something strange with . . .
It is very important to isolate such terms and avoid them in the future defects because they obscure the facts with subjectivity and increase the likelihood of preserving a bug with a high or even critical and/or blocker priority in the system.
Some terms may seem as standard forms of politeness. But sometimes a bug report is overloaded with such \terms of politeness", which can divert the developers focus from the software problem itself.
2) \False friend" terms are the terms that might seem useful and look organic in a technical text, but, surprisingly, in the context of defect reports, they can decrease the transparency and make the meaning ambiguous. The examples of such terms are: error, perform, appear, document, con gur, etc. Some samples of use of these terms extracted from the defect reports under analysis are presented in Table 5. Example of use of these terms Clari cation
Ambiguous interpretation of \documentation" The documentation claims that . . . leads to a misunderstanding between the developer and the submitter.
Add the name again and continue The lack of details about \non-appearance" but the name does not appear . . . makes the bug non-reproducible.
When trying to perform operation, The lack of details about what is being the exception is thrown . . . performed can confuse the developer. Some plugins fail to startup with The absence of conditions and details leads to a the following error . . . biased assessment of the root cause.
This group of terms is especially dangerous. The submitter describing a
software defect uses these terms and overlooks the details because he or she thinks
that the description is comprehensive.
3) Domain-related terms. Examples of such terms are: osgi, gitssh, retract,
gradebook, bundl, etc. Knowing them is very useful because it gives an
opportunity to reveal potential areas of testing [
It is important to mention that these groups of terms can overlap. It means that some areas of testing can be rather complicated, and there are more possible cases for misunderstanding and underestimating a potential problem. For example, the submitter describes user actions which are connected to one of the areas of testing. He or she notices errors or exceptions. However, due to the complexity of the area of testing and the fact that some necessary details are missing, it is di cult for a developer to understand the cause of these results: it is not obvious whether it is a possible defect or just an erroneous chain of submitter s actions.
We have compared the methods of feature selection and ranking and noticed that chi-square is prone to place the \prejudiced" terms at the top, and the feature ranking by random forests is prone to place the \false friend" terms at the top. So they can be useful for revealing the words that can create a situation where some facts or details are omitted.
We also compared the methods of feature selection and ranking in order to check their accuracy. We used the cross-validation technique for this task. The received results are presented in Table 6. We discovered that the cases without feature selection have lower accuracy values than others. According to Table 6, Random Forest has the highest accuracy. This paper is devoted to the problem of underestimated bug reports. These are reports where the resolution of the described defect was changed from a potentially incorrect one, such as \Reject", \Not a defect", etc., to an ordinary one of \Done" or \Fixed".
We have revealed the speci cs of such bug reports. They are long to resolve and have a large number of collateral comments and attachments. They can be considered as potentially problematic bugs, i.e. the defects that require human and time resources.
We have proposed several methods of feature selection and ranking in order to build the top of the most signi cant terms. We used chi-square, recursive feature elimination, features importance of random forest and coe cients of logistic regression.
We compared di erent methods of feature selection. We found out that the cases without feature selection have lower accuracy values than others and Random Forest has the highest accuracy among the methods.
We have analysed the received tops of terms and proposed to split them into three groups. The rst group - \prejudiced" terms - includes terms with subjective assessment. The second group - \false friend" terms - consists of terms with a potentially dangerous lack of details. The third group - domain-related terms - includes terms associated with an area of testing. The analysis of these groups can help understand the key problems of underestimated defects as well as prevent them from occurring.
In the nearest future, we plan to analyse each potentially problematic resolution separately. We also plan to compare underestimated bug reports with defects that have the nal resolution of \Rejected", \Won t Fix", etc. It can provide a better picture of the speci cs of such defect reports. Consequently, it can help to generate recommendations in order to reduce the occurrence of underestimated bugs.