Software development projects receive many change requests each day and each report must be examined to decide how the request will be handled by the project. One decision that is frequently made is to which software developer to assign the change request. Efforts have been made toward semiautomating this decision, with most approaches using machine learning algorithms. However, using machine learning to create an assignment recommender is a complex process that must be tailored to each individual software development project. The Creation Assistant for Easy Assignment (CASEA) tool leverages a project member's knowledge for creating an assignment recommender. This paper presents the results of a user study using CASEA. The user study shows that users with limited project knowledge can quickly create accurate bug report assignment recommenders.
Copyright is held by the author/owner(s). EICS’16, June 21-24, 2016, Bruxelles, Belgium.
bug report triage; assignment recommendation; machine learning; recommender creation; computer supported work
H.5.m [Information interfaces and presentation (e.g., HCI)]: Miscellaneous; I.2.4 [Programming Languages and Software: Expert system tools and techniques]; I.2.7 [Natural Language Processing: Text analysis]
Large software development projects can receive hundreds
of bug reports per day [
Bug triage takes significant time and resources [
Conceptually, the creation of an assignment recommender
using machine learning is straightforward [
This paper presents the results of a user study using CASEA to create assignment recommenders for a large open source project, and is the first such study. The study found that subjects could quickly create an accurate assignment recommender using the tool, despite the users having no specific knowledge about the software project.
To our best knowledge, CASEA is the first system to address the bug report assignment recommender creation problem, and this paper presents the first study of its use.
This paper proceeds as follows. First, an overview of CASEA is presented. Next, the results from a user study involving subjects creating assignment recommenders for the Eclipse Platform project are presented. The paper then concludes with a discussion of some of the threats to the validity of this work, related work and possible future improvements to make CASEA more practical for software development projects.
This section presents background information about bug reports, their life cycles, and machine learning.
Bug reports, also known as change requests, provide a means for users to communicate software faults or feature requests to software developers. They also provide a means for developers to manage software development tasks. Bug reports contain a variety of information, some of which is categorical and some of which is descriptive. Categorical information includes such items as the report’s identification number (i.e. bug id), its resolution status (e.g., NEW or RESOLVED), the component the report is believed to involve, and which developer has been assigned the work. Descriptive information includes the title of the report, the description of the report, and discussions about possible approaches to resolving the report. Finally, a report may contain other information, such as attachments or links to other reports.
All bug reports have a lifecycle. When a bug report first
enters a project’s issue tracking system (ITS), it is in a state
such as UNCONFIRMED or NEW. The bug report will then
move through different states, depending on the project’s
development process, and arrive at a resolution state, such
as FIXED or INVALID. The lifecycle of a bug report can be
used to categorize bug reports [
Machine learning algorithms Machine learning is the
development of algorithms and techniques that allow computers
to learn [
The Creation Assistant for Easy Assignment (CASEA) [
The first step in recommender creation is to gather the data to be used for creating the recommender. Specifically, bug reports are extracted from the project’s issue tracking system (ITS). The project member provides the URL of the project’s ITS, a date range for data collection, and an optional maximum limit for number of reports to gather. Reports that have a resolution status of RESOLVED, VERIFIED or CLOSED are gathered chronologically, with every tenth report selected as a testing report to create an unbiased set for evaluation.
Having collected the data from the project’s ITS, the next step is to filter the data to produce the highest quality training set. Two types of filtering are performed: automatic and assisted.
The automatic filtering performs three actions on the textual
data. First, terms that are stopwords (i.e. common words
such as ’a’ and ’the’) are removed. Next, stemming is
performed to reduce all of the terms to their respective root
values so that words such as ’user’ and ’users’ are treated
as the same word, ensuring a common vocabulary between
the reports. Finally, punctuation and numeric values are
removed, except where the punctuation is important to the
term, such as URLs or class names (e.g. “org.eclipse.jdt”).
CASEA assists the user with two types of filtering: label
filtering and instance filtering. To assist with label filtering,
CASEA presents the user with a label frequency graph. For
an assignment recommender, this graph presents bug
fixing statistics, a type of activity profile [
Instance filtering is done using project-specific heuristics. The heuristics have two parts: a grouping rule and a label source. The grouping rule is used to categorize the data into groups for which the label source will be used for labeling the instances. For an assignment recommender, the grouping rule is a bug report lifecycle (called “Path Group” in Figure 2) and the label source (i.e. data source in Figure 2) is either a field from the bug report, such as the assigned-to field, or other labelling information that can be extracted from the bug report, such as the user that last attached a patch or the developer who marked the report as resolved.
Figure 2 shows an example of instance filtering in CASEA. All of the training reports are used to determine the specific bug report life cycles for the project and the user is presented with a statistical summary of the categories. The figure shows that for the data set for the Eclipse Product project, 30.6the reports have a NEW!FIXED (NF) lifecycle, followed by 22.6% NEW!FIXED!VERIFIED (NFV), and 15.6% being NEW (N). Using this information, the user can create heuristics for the most common occurring categories. In this case, “FixedBy” was chosen for the NF category, “Resolver” for the NFV category, and “Reporter” for the N category. The user can also choose the number of heuristics to be applied, to a maximum of ten; six was chosen in Figure 2.
After filtering the data to create the set of training and
evaluation instances for the recommender, the data is then
formatted for use with the machine learning algorithm. Once
the user has filtered the data and the data is formatted, the
recommender is created using a multi-class Support Vector
Machines (SVM) algorithm with a Gaussian kernel. SVM is
a commonly used algorithm for assignment
recommendation [
Once the user starts the recommender creation process, the user is moved to the Analysis tab that presents the recommender evaluation results (Figure 3 ). The user can then return to the Configuration tab, adjust the values for label and instance filtering, and create a new recommender. This process continues until the user is either satisfied with the created recommender, or the user has determined that an assignment recommender cannot be created with a high enough accuracy to benefit the project. At any time the user can save the recommender configuration and return at a later date.
CASEA uses the metrics of precision and recall to evaluate a created recommender. It presents results for the top recommendation (top-1), the top 3 recommendations (top3), and the top 5 recommendations (top-5). Figure 3 shows an example of the evaluation results for a recommender after eighteen trials. It shows that the first four configurations did not create very accurate recommenders as the activity threshold was too low, but when the threshold was raised, a more accurate recommender was produced. After about five more trials, a good heuristic configuration was determined that produced an assignment recommender with a high top-1 accuracy and reasonable top-3 and top5 accuracies. Further experimentation was done with the heuristics with varying results, before determining that the configuration from trial #10 was the best configuration.
A small user study was conducted to assess the potential
for CASEA to assist software projects in creating
assignment recommenders. Specifically, the study sought to
answer qualitative questions such as “What aspects of the
recommender creation process and the CASEA interface
do users find helpful? challenging? or confusing?". This
study was similar in intent to Stumpf et al [
The user study was conducted in the following manner. First, subjects were asked to complete a prior knowledge and experience survey. Specifically, subjects were asked about their prior knowledge and experience in two areas: technical experience and technical knowledge. For technical experience, subjects were asked about their level of experience with issue tracking systems, open source projects and software testing. To assess prior technical knowledge, subjects were asked about their familiarity with bug reports, machine learning algorithms, classifiers or recommender systems, user interface design principles, and data mining. A Likert scale (Very Experienced/Familar, Some Experience/Familarity, Little Experience/ Familarity, Heard Of, No Experience/Familarity) was used for the self-reporting of their experience level. After completing the prior experience and knowledge survey, subjects were asked to create an assignment recommender for the Eclipse Platform project within fifteen minutes. Once the subjects expressed that they were done using CASEA, a debriefing interview was conducted. The subjects were asked to explain their approach to creating an assignment recommender using CASEA and what they recommended as improvements to the tool, as well as any other comments about their experience with CASEA.
Figure 4 shows a summary of the responses from subjects regarding their prior experience. As shown, most of the subjects had prior experience with issue tracking systems, with four reporting some experience and four reporting little experience. Overall testing experience was a bit less, with six subjects reporting little experience, two reporting “heard of” and one reporting no experience. Subjects had the least overall experience with contributing to open source projects, with two reporting little experience, three reporting “heard of” and four reporting no experience.1 For technical knowledge, Figure 5 shows a summary of the responses. Most of the subjects reported either being very familar (2 subjects) 1Not all subjects provided answers to all of the questions. or having some familiarity (6 subjects) with user interface design principles from either recently or currently taking an undergraduate course about this topic. Half of the subjects reported familiarity with machine learning algorithms, and slightly more (5 subjects) reported familiarity with classifiers and recommender systems. The knowledge in these areas came from either taking an undergraduate course in computational intelligence or from other course projects. Subjects reported the least familiarity with the bug report lifecycle and data mining.
For technical knowledge, Figure 5 shows a summary of the responses. Most of the subjects reported either being very familar (2 subjects) or having some familiarity (6 subjects) with user interface design principles from either recently or currently taking an undergraduate course about this topic. Half of the subjects reported familiarity with machine learning algorithms, and slightly more (5 subjects) reported familiarity with classifiers and recommender systems. The knowledge in these areas came from either taking an undergraduate course in computational intelligence or from other course projects. Subjects reported the least familiarity with the bug report lifecycle and data mining.
Table 1 shows the quantitative results from the eight subjects. The first column identifies the subjects. The next two columns present both the number of trials a subject conducted before creating their most accurate assignment recommender, and the total number of trials that the subject conducted in creating an assignment recommender using CASEA. The next three columns show the Top-1, Top-3 and Top-5 precision and recall values for the best Eclipse Platform assignment recommender created by the subject. The last three rows of Table 1 show a summary of the results, presenting the maximum, minimum, and median values for the columns.
Table 2 shows the threshold and heuristic configurations for the best assignment recommender created by each of the eight subjects. The first two columns list the top ten path groups for the data set and how much of the data set is covered by the path group. As shown by the table, 77% of the data set is covered by the first five path groups, and most of the subjects specified heuristics for six or fewer path groups. Also, with one exception, the “Assigned” data source was used for the remaining 11%-27% not covered by the specified heuristics. Half of the subjects chose values less than 10 for the threshold and the others used values greater than 20. The results show that the subjects were usually able to create a reasonably accurate assignment recommender in 10 trials or less. The two most accurate recommenders (created by Subjects #7 and #8) had the same configuration (i.e. threshold and heuristic values), as shown in Table 2.
Based on observations during the study and responses from the debriefing interview, subjects were found to employ two strategies for assignment recommender creation using CASEA. Some subjects were found to be very experimental in their approach, making many changes before creating a new recommender. Other users were more methodical, making small changes and testing the results. Figure 6 shows a categorization of the different types of changes (heuristic change, threshold change or both) made by each subject. As expected, subjects changed the heuristic configurations the most, and most subjects only changed the threshold three times or fewer.
One subject commented that the best strategy was to make small incremental changes, and that CASEA made it easy
Top (1%) Top (3%) Top (5%) Precision Recall Precision Recall Precision Recall to employ this strategy. Another subject observed that creating an assignment recommender using CASEA was similar to trying to get a high score in a game, where the score was the precision and recall values.
As part of the recommender evaluation, CASEA provides information about how long it takes to create a recommender. This led some subjects to work towards an incorrect goal of minimizing the recommender creation time. Although subjects were provided with a brief tutorial of CASEA and a high level explanation of the recommender creation process at the beginning of the study session, subjects encountered a number of problems related to understanding terminology or concepts. Specifically, the term “Path Group" was used to describe the categorization of bug reports, and subjects found this term unintuitive. This led to some initial confusion about the options in the heuristic configuration panel. Also, the meaning of the precision and recall metrics was not initially well understood by subjects. However, once their meaning was understood, subjects felt that they made more intelligent choices about the
As was mentioned, the subjects did not have specific knowledge about the project, such as who formed the core group of developers. This led to some subjects choosing a low activity cutoff so as to not exclude developers, and resulted in recommenders that were not accurate and took longer to create. This behavior would not be expected from an actual project member using CASEA, as they would have knowledge about the core development team.
This section highlights some of the threats to the internal validity, external validity, and construct validity of this work. Threats to the internal validity of this work relate to the potential sources of error in the evaluation of CASEA. A potential source of error is with the creation of the data set used for the evaluation. Although a random sample of bug reports was examined to establish that the data collection procedure was correct, there may have been some bug reports that contained incorrect data.
Threats to external validity relate to the generalizability of the results to other projects or user groups. In this work, subjects with no project-specific knowledge were used to evaluate the usability of CASEA. Therefore, these results would not generalize to those with project-specific knowledge, but could be considered as a lower-bound for such a group.
Threats to construct validity refers to the suitability of the
evaluation measures. The method used to determine the
set of developers that could have fixed a bug report, used
for calculating precision and recall, is known to
overestimate the group [
This section presents related work in the areas of assisting with triage, assisting with recommender creation, and explaining machine learning.
Like CASEA, Porchlight [
SkyTree Infinity [
One avenue toward making the use of recommender systems practical is to assist in their creation and evaluation. This is the approach taken by CASEA. An alternative approach to making their use practical is by explaining their results.
Poulin et al. [
Kulesza et al. [
Basilio Noris developed a visualization tool for machine
learning called MLDemos [
This paper presented the results of a pilot study of CASEA, a tool to assist in the creation of bug report assignment recommenders. CASEA assists a user in labelling and filtering the bug reports used for creating a project-specific assignment recommender, as well as providing feedback on the effectiveness of the configured assignment recommender. The study found that users with little to no project-specific knowledge were able to quickly create effective assignment recommenders for the Eclipse Platform project.
Based on feedback and the results of the user study, a
number of future improvements were idenitified for CASEA,
including having CASEA first attempt to tune a
recommender automatically and then have the user tweak the
configuration, extending CASEA to assist with the creation
of other triage recommenders, supporting other machine
learning algorithms, and providing other evaluation
metrics, such as F1. An improved version of CASEA, called
the Creation Assistant Supporting Triage Recommenders
(CASTR) [