Diverse research groups have identi ed that analyzing the bugs-to-code x links may generate many interesting theories. However, this kind of datasets are usually not easily available from the software development projects. Usually, the bug tracking and the related code x activities are divided into separate processes and use separated repositories thus complicating the linking of the data. Numerous techniques have been developed but we still lack the empirical data collected with standard data collection procedures. In this paper we examine the e ectiveness of the bug linking techniques. In particular we evaluate the proposed technique based on the regular expressions and compare its e ectiveness with other known bug linking techniques. Our case study shows that the proposed technique is equally e ective as some other already proposed techniques in eliminating the linking bias we identi ed some di erences that may have in uence on the linking precision. However, the technique is not addressing data quality issues that may be present in software repositories.
7:48 tool that would based on the numerous use open source available repositories build as much as possible bug-to-code fix datasets. This paper deals with definition of such a tool that we will refer to as BuCo (Bug-Code) tool.
In the last few years, research on bug–fix datasets has progressed well, since the number of open source projects and using of open source project repositories grow and reported many important results. The popularization of open source software enabled individuals to introduce different ways of software development and code analysis. However, datasets bias still remains as a huge issue for wider generalization of the results and development of sound theories. The bug–code fix linking process is extremely hard task and there is still no standard protocol that would be applicable for wider community and numerous software development repositories. Different linking techniques have been proposed in order to minimize the linking bias and some even try to minimize the impact of data quality issues. However, we believe that different linking strategies may not be equally effective for all kind of repositories and software projects. Obviously, proposing the general dataset collection procedures and linking techniques may not be relevant. Instead of proposing the best performing one for open source repositories we review several linking techniques and evaluated their effectiveness on different datasets. We discuss its applicability for data collection from complex open source software projects.
Many researchers mine open source software development repositories trying to collect bug–fix datasets. One of the most commonly applied technique is using regular expressions [Bachmann et al. 2010; Fischer et al. 2003; Schrter et al. 2006; S´liwerski et al. 2005; Cˇ ubranic´ and Murphy 2003]. It is a sequence of characters that forms a search pattern, most commonly used for string matching used for information retrieval [Baeza-Yates and Ribeiro-Neto 1999]. Drawing on the idea introduced by a neuroscientist and a logician in the early 1940s, it was formally introduced by a mathematician 15 years later, relating the concept to finite automata [Kleene 1956]. First implementation of a regular expression compiler was developed in 1968 by the Unix pioneer, Ken Thompson [Thompson 1968], and has since been used in various problem domains. A regular expression consists of both regular and meta–characters. Both types are combined to form an expression for identifying the pattern sequence (sequence of interest) in a sequence of characters. In other words, the pattern sequence is an expression representing prescribed targets in the most concise and flexible way to direct the automation of character sequence processing.
A regular expression processor translates into a nondeterministic finite automaton, which is then run on the sequence of characters to recognize subsequences that match the regular expression [BaezaYates and Ribeiro-Neto 1999]. The most commonly used linking technique is based on regular expression and is searching in source code commit messages for a specific keyword such as fixed or bug and bugID (such as 42233) usually using form of regular expression [Bachmann et al. 2010].
Many researchers aiming to develop the most effective linking technique and to understand dataset bias and data quality issues have invested huge efforts to develop a reliable dataset for the benchmarking purposes. As part of the study published with [Bachmann et al. 2010] the authors work with an experienced Apache developer who helped them to manually develop an ground truth dataset. The main focus of the study was to understand the quality of the data provided in the open source software development repositories. They found out that many bug fixes are not identified within the commit messages or even not reported within the bug tracking repository and that this finding might be a serious threat to studies based on the such datasets. However, the sample size they had was not large enough to make any statistical conclusions but provide some reasonable evidence to threats to external validity of studies performed on linked datasets from open source repositories. One solution to the data quality problem they see in improving the linking process with information available from the whole social eco–system available such as revision control system, bug tracking database, mailing systems, email discussions, etc.
That finding motivated a number of studies aiming to improve the linking techniques. Improvements may be classified into two directions; direction that involves additional information available from the open repositories and direction that introduces fancy algorithms for the link prediction purposes. One very good example is ReLink, a tool proposed in [Wu et al. 2011] that offered a 2–cycle linking process. In the first round the traditional linking technique based on the regular expression as mentioned above has been extended with new features as follows: bug fixing time that is close to the change–commit time, the change logs and the bug report share textual similarity, and that the developers responsible for a bug are typically the committers of the bug–fixing change.
In the second round, based on the linking dataset in the first round as training sample and some learning rules, predicts additional linking data items. They compared their results with the ground truth file prepared by [Sureka et al. 2011]. As reported in [Wu et al. 2011] they obtained the consistent results with [Sureka et al. 2011] for the Apache project. Also, they conclude that many open source projects hosted by Google are similar and follow the features they implemented in the ReLink. On the other hand, they identified that linking improvements were not equally effective for all datasets. For example linking improvement for Apache was marginal compared to other two Android projects in the study. The reason for this result lies in relatively good data quality of Apache compared to Android projects. We assume that for bigger and complex open source projects the data quality probably tends to be maximized for several reasons. Firstly, the project takes longer and community has time to learn ways of working and less human error happens and another reason is that the samples are getting bigger and thus marginalize errors.
Recently published study [Bissyande et al. 2013] has validated their results and assessed the effectiveness of ReLink quantitatively and qualitatively. Again, they confirmed that in the different project setting the effectiveness of ReLink improvements may be marginal but also they show that these improvements may lead to new errors thus significantly influencing the linking bias. They also provided a benchmark dataset collected from Apache Software Foundation with input and output files provided for 10 projects.
Our goal is to identify and explore the most effective linking technique across number of different environments. In this study we define an experiment for that purpose. The ReLink tool, as a good example of combining the simple search with advanced prediction techniques is chosen for this experiment. However, it was only compared with other similarly complex approaches as well as the ground truth dataset. Moreover, because of the input data it provides to open community we decided to investigate how well does it perform comparing to the simpler search approaches. That is why we plan to conduct our experiment in several stages: (1) Analysis 1: Compare the ReLink tool with the simple search on Apache HTTPD dataset given by ReLink as presented in figure 1 (2) Evaluate the results, manually investigate the incorrect links and construct a regular expression search criterion (3) Analysis 2: Compare the ReLink tool with the regular expression search using the same
Apache HTTPD dataset as presented in figure 1 (4) Analysis 3: Compare the ReLink tool and the regular expression search with the benchmark dataset for the Apache Opennlp project as presented in figure 2
The analysis 1 consists of comparing the ReLink tool with the basic idea of linking bugs and commits searching for the Bug ID number in the commit messages. We believe this approach to be the most reliable one. The only question is how to search for the Bug ID effectively. That is why our first approach to the linking was as simple as that. The only improvement we added was to search only for the latest commit that contains the bug ID in order to discard all the duplicated entries, bug fixing attempts and workarounds. The input bugs are taken from the ReLink webpage1 while the source code is taken from the Github repository of the Apache HTTPD project2. These inputs are given to the BuCo Analyzer tool and the linking is done using simple search mechanism.
The results of analysis 1 will indicate the weaknesses of simple search. The manual investigation of the incorrect links will look for regularities and consistent errors in order to yield a criterion for more sofisticated search. We will construct the regular expression based on this criterion and repeat the analysis. For the analysis 2 we use the same Apache HTTPD project with the same inputs as in the analysis 1 (figure 1).
SVN
by: [Wu et al. 2011]
Date, Comments] by: [Wu et al. 2011] Bugs[.BtxutgID, Type, Status, Owner, Reporter, ReportDate, ModifiedDate, LastDate, Summary] by: [Wu et al. 2011]
Bug-Commit
Links Links.txt [BugID, revisionNo] by: [Wu et al. 2011] Bug-Commit
Links
The analysis 3 compares the ReLink tool and the regular expression in a controlled environment of a benchmark dataset. The Opennlp project is analyzed and given by [Bissyande et al. 2013]. The goal of the Opennlp data is to provide a benchmark dataset for comparison of studies such as this one. It offers the subset of bugs, the source code history and the correct links between the bugs and commits made by developers. Furthermore, it offers the comments on all of the given bugs extracted from bug tracking repository and the bug-commit links made by the ReLink tool. The bugs for the Opennlp project are taken from the benchmark dataset3. The source code is taken from the Github repository of the Apache Opennlp project4. These inputs are given to the BuCo Analyzer tool and the linking is done using regular expression search mechanism. The details are presented in figure 2. 1https://code.google.com/p/bugcenter/wiki/ReLink 2https://github.com/apache/httpd.git 3http://momentum.labri.fr/bugLinking/ 4https://github.com/apache/opennlp SVN
by: [Bissyande et al. 2013]
Date, Comments] by: [Bissyande et al. 2013] Bugs[.BtxutgID, Type, Status, Owner, Reporter, ReportDate, ModifiedDate, LastDate, Summary] by: [Bissyande et al. 2013]
Bug-Commit
Links
by: [Bissyande et al. 2013] Bug-Commit
Links
by: [Bissyande et al. 2013] Bug-Commit Links
One of the first challenge encountered in our research was to find the appropriate tools for such a demanding task of data collection. We managed to find no tool that could provide us with the source code analysis of software product metrics and the number of bugs contained within every software module. That is why we developed a tool of our own and named it the Bug Code (BuCo) Analyzer. Its goal is to collect software fault’s data and find its dependencies in the most efficient way possible. The tool is based upon the following technologies: Apache HTTP server, MySQL relational databases, Git and Subversion source control tools, Python and data mining. They were chosen after their reliability and experienced gained from using them in past projects.
The local database within the tool is constructed so that it can contain the bug information details, the links between the bug and the source code changes (commits) and the list of source code modules and all of their metrics for projects of three major open source communities: Eclipse, Mozilla and Apache. The bug details can be downloaded from the Bugzilla repository into our database and the complete source code history can be downloaded through the tool interface and stored locally as a GIT repository. The BuCo Analyzer also offers interface to other tools. One such tool is the ReLink. Its task is to establish connections between the bugs and the commits. However, that is just one of the techniques the tool offers for this linking process. The complete architecture of the BuCo Analyzer tool is presented in figure 3
As mentioned earlier, our intention is to analyze large and complex software products with long lasting evolution, i.e. a great number of releases. All the analyses then become demanding and slow. To optimize and gain every little boost in terms of speed some smart thinking and known techniques are used in the creation. MySQL databases which would store the data needs to be passed through normalizations steps and has to be modeled after the tool’s needs. To further speed up the process repositories are downloaded and analyzed locally instead of analyzing them remotely over the Internet. Other optimizations are done by building efficient search patterns and compiling the patterns when they are necessary.
The tool was being improved iteratively. The first and the most notable aspects is the introduction of pattern matching using regular expressions. This version is able to collect all commits for every bug ID found and their respective messages and implements the collection of bug IDs and downloads
User
BuCo Analyzer Tool
User Interface
Core Product
External
Interfaces Internal
Interfaces Database
File system
Bugzilla
GIT
ReLink of repositories for different foundations. The latest tool version offers a high credibility level for the bug-commit links and the software product metrics associated to the software modules.
The results of analysis 1 are presented in table I. The ReLink tool performed the linking with the same bugs but with different source code repository, the SVN. The difference between the two repositories is obvious in the number of commits they contain and it reflects on the output results as well.
Analysis 1
The results of the simple search were then manually analyzed, compared to the ReLink results and validated. We focused on finding the links that the ReLink and the simple search had in common in order to evaluate the general appropriateness of the approach and on the links that are different in order to discover the simple search weaknesses. The results given in table II indicate that 74.1% of the links are equal to the ones made by the ReLink and all of them were correct ones. The 20.1% of the links that were not equal to the ones made by the ReLink and were incorrect presented a valuable subset to discover patterns and consistent errors in linking process.
The evaluation and manual validation of the results revealed that in each of incorrect links the Bug ID was adjacent to other alphanumeric characters forming a Bug ID with more digits or a different Equal Links Different Links
Bugs with one link Bugs with multiple links
Incorrect links
Potentially correct links Links From Different Repository identification code. This led us to the improved search criteria which will predetermine the bug ID surrounding characters so we constructed the following regular expression: the preceding character to be any non digit character, including the start of string the number to be the exact match of the Bug ID we are looking for the following character to be any non alphanumerical character, including the end of string In order to evaluate the performance of linking bugs and commits using regular expression given in (1), we performed the analyses 2 and 3. Unlike the HTTPD project, we managed to find the equal source code history for the Opennlp project even though we take it from different repository. Therefore, the linking is done upon the same input and it offers a good comparison basis. The correct links given by the benchmark dataset are compared with the links obtained by ReLink and with the links obtained by our regular expression search. The details are given in tableI.
The results obtained from the analysis 2 upon the HTTPD project indicates the regular search can link approximately the same number of bugs as the ReLink tool. The difference between these two approaches is due to the different source code inputs. Analyzing the links made by the ReLink tool and not by the regular expression search reveals that all of their links do have a bug ID number in the commit message. We looked for these commits in the GIT repository but did not find them. This means that the regular expression search would have performed just as good as the ReLink does if it had the same source code input.
In order to prove this theory, the analysis 3 is done upon the Opennlp project comparing the results also to the benchmark dataset. The results show that our regular expression search approach actually outperforms the ReLink tool. We found all the correct links given by the benchmark and an incorrect one. The incorrectly linked commit is given in table III. The commit message clearly shows it has nothing to do with the Bud ID we are looking for. Furtheremore, the time elapsed between closing the bug and committing the supposed fix is greater than it usually is (from several days up to a month). Finally, the bug assignee does not correspond to the author of the commit. This link was made due to a single digit bug ID which is found in the name of Apache release. The bug IDs that are small in value present an obstacle to our search approach because they could be found in other identification numbers, dates, release tags or similar parts of a commit message.
The ReLink, on the other hand had no incorrect links, but managed to miss 12 of them, which is approximately 10%. They also did not link 5 bugs at all, which is approximately 6%. Several examples of missing links are given in table IV. It is unclear why the ReLink tool did not make that connection. All the presented missing links do contain the Bug ID in the commit message. The time elapsed between closing the bug and committing the supposed fix ranges from 0 up to 63. Besides the first commit linked to the Bug ID 367 which has 63 days and the commit linked to the Bug ID 471 which has 35 days, the remaining commits are within usual time interval. Finally, all the commit authors do correspond to the bug assignee role in the presented missing links.
The mistakes of the linking process reflect upon the files we indicate as fault prone as well. Our approach pronounces the same 141 files to be fault prone as the benchmark dataset does, because the incorrect link is connected to the already fault prone file. The ReLink failed to pronounce 9 different files as being fault prone, which is approximately 6%.
Integrated datasets such as Bug-Code fix datasets are very important for further development of the software engineering discipline. This datasets can provide us new insights into the software development processes and practices and the conclusions may lead to new developments.The creation of as much as possible Buge–code datasets that are reliable and comparable enough to minimize external threats to validity is an important task. However, addressing the data quality and dataset bias is not an easy task.
In this paper we show that already a simple traditional approaches with help of regular expressions may work well. However, they may not be equally effective in all environments. One should adapt the regular expression to each particular repository and even to each project. From the performed study we may observe that the data quality issues may be lower for bigger, longer and more complex projects. Therefore, we assume that very advanced linking techniques involving learning rules might not be not only ineffective but also not applicable for the complex project environments. The very good results obtained by our regular expression bug-code linking, evident in the case of benchmark datasets, encourage us to expand this research. Our future work is to replicate it on more complex projects from different open source communities and to expand this the selection bug-code linking techniques. The comparison of techniques will include the datasets provided by other researchers and try to find statistical evidence in favor or against these statements.
The main contribution of this study is the presentation of an experiment and the preparation for more advanced and more demanding experiment that would lead to development of algorithm adaptable to open source repositories and projects aiming to build as much as reliable bug–code datasets.