- Product based organizations have diverse product offerings that meet various business needs. The products are in turn integrated to create integrated product suites. Rigorous product engineering is a must for creation of high quality integrated software products. Adequate measures must be taken to improve quality of the integrated product before every release of its module or sub-product. It is hard to imagine upgrading an integrated software product with unidentified defects prior to its release. In this paper, we share our observations on implementing a defect dependency metric to identify the dependency of a defect over a real-time industry defect dataset of an integrated software product. This defect dependency metric was captured and analyzed during release cycle(s) to avoid surprise issues post product launch.
Academic research in areas such as software architecture, automation frameworks and implementation methods has seen a tremendous growth in recent years and it has been observed that software industries apply them in real-time business to achieve better results [1][2]. Many software practitioners are currently trying to use methods and technologies proposed by academia to create products to the best of their abilities. There were many lessons learnt from industrial case studies over the past decade [3].
All new products are created with the intent of delivering better functional and quality objectives that meet or exceed end user expectations. Most software firms are now deliberately framing their mission statements with a ‘grow fast or die fast’ strategy before they hit the market with a high quality product. As per Gartner’s 2015 Magic Quadrant for Enterprise Integration Platform as a Service survey [4] most of the software industries that work on developing integrated software products still follow traditional approaches to develop and maintain quality standards of their existing products. As per their study, most of the new start-ups are concentrating on new trends in research for a better product(s) of similar class.
In most cases, it is easier for start-ups or new development projects to implement new trends in research on to software production. However it is a challenge for well-established and equipped products to adhere to these changes as it requires massive planning and human effort. Especially in integrated software, individual sub-products which are commonly referred
The primary author of this paper has been working in this domain for many years and has contributed to the integration of the integrated product suite in various roles. The primary author is also pursuing graduate studies on a part-time basis. Hence the authors could gain access to all the artifacts and the original data. Due to non-disclosure clauses, the name of the integrated product suite, its product pillars and the organization is being withheld. The product information shown in Table 1 makes use of alternate names to the existing (real) names. However the defect dataset presented in table II shows exactly the same numbers as present in the defect database for the various products and versions of the integrated software product.
The rest of the paper is organized as follows: Section II provides details of industrial examples of software quality related to our work, section III explains the background of defect dependency with an example along with study design of our work, and section IV details the implementation setup of defect dependency metric on an industry defect dataset. Section V talks about results of our implementation and observations identified during every new release of our integrated software product. Finally in section VI we discuss the threats to validity and present some insights about future work.
Software Quality Assurance (SQA) in integrated software
products is a major activity during software production cycle.
Advanced SQA practices were proposed by various researchers
over past decade that became standard approaches in today’s
software production release cycle. Functional integration
approaches, strategies and methodologies to integrate software
by its features were initially proposed [7]. Cost based effort
estimation method [8] for integrated software architecture
model-COTS was proposed and deduced quality measures to
choose right resource for right task. Fedrik et al. proposed
quality based methods to improve software integration [9]. In
[
Researchers have presented interesting methods on
implementation of integration in global software projects and
veracious trends in integration [
Large-scale software products are complex and as such are prone to defects. Software quality teams have to perform rigorous checks before releasing a fix to a defect. This includes ensuring that the fix will not cascade new defect(s) into the product. The setup can be simple in case of small products but not for complex software products or an integrated product suite. Quality teams mostly face integration issues with incorrect control flow and data flow between the sub-products or submodules with in entire integrated product. It is also tough to detect and track the source of a defect in a complex integrated system as this involves various other quality teams from different sub-products. Firms that integrate products due mergers and acquisitions have different set of challenges as these products may have evolved independently but not in an integrated fashion. In such a scenario, it is essential for product owners to understand the impact the defect so as to mitigate possible surprise defects from other modules of the integrated product. We introduced defect dependency metric to address this specific concern in our previous paper [5]. We proposed a Defect dependency metric (D*) to calculate defect dependency by demonstrating the application of Generalized Dependency degree (Г) using rough set theory [6].
Defect dependency can be defined as a metric to study the
widespread of a defect with unknown impact and unknown risk
over a module(s) or component(s) or sub-product(s) of a
software product(s). Defect dependency can be calculated for
any software of any size, however heuristically it is more
applicable for complex systems as it is difficult to comment on
widespread of a defect without any evidence. Generalized
Dependency degree (Г) is a mathematical approach to calculate
the dependency between the equivalent classes generated by
equivalence relation using disjoint sets. Initial study using this
approach was proposed in Rough Set theory and was later
studied by Halxuan et al [
If we redefine above definition in terms of a defect dependency approach, consider a defect dataset (D) of a large scale complex software product (L). Then:
Learning Management
System (LMS) Human Resource
System (HRS) Business Intelligence
System (BIS) Work force Manager
(WFM) Web Services Manager (WSM)
Sub-product Admin Mgmt.
Learner mode Manager mode
Hire Mgmt.
Compensation Mgmt.
Succession Mgmt.
Performance Mgmt.
BI Dashboards Data Downloader
Data Uploader Attendance Mgmt.
Payroll Mgmt.
Reimbursement Mgmt.
Export Mgmt.
Integration Mgmt.
Web Service Admin mode If P1, P2, P3, P4 …… PN are sub products of L, then consider DP1, DP2, DP3, DP4…DPN are defect subsets of respective subproducts of a universal defect dataset D. S = (D, De) is an approximation space, where D is a nonempty finite defect set and De is a equivalence relation defined over all defect subsets DPi where {i ϵ 1,2,3….n} To calculate the dependency of a defect subset attributes over another subset, we will evaluate the value for Г (Generalized dependency degree) which is defined as D* = Г(O, H) = 1 |D| ∑ |O(x) ∩ H(x)| |H(x)| (1)
Here O & H are two equivalent classes generated over an equivalence relation framed from some disjoint sets of universal set D. We have utilized this method to find dependency of a defect on our industrial defect dataset. It is a simple mathematical approach to understand the dependency of a one set over another. Each data point in the dataset contains collection of attributes that are pre-processed such that it can be applied over dependency metric. If we map this method to our real time dataset, D is the total defect dataset of our enterprise software product, O and H are two equivalent classes of equivalent sets which constitutes defects of two different subproducts O and H. In case there are more than two sub-products, we need to generate equivalent sets of all the defect product subsets, constructs equivalence class and apply this formula. There is no definite scale to the defect dependency metric, however the value varies between 0 and 10.
Our industry defect dataset contains defects of an Integrated Human Resource Integrated System (IHRIS) product with 5 primary product pillars (as shown in Table I) that are integrated as a single product suite. Each product pillar has sub-products that are implemented in an integrated mode. As stated earlier, due to non-disclosure clause, we are use the common derived names of product and their sub-products instead of the original product names.
This integrated product is deployed as Software-as-Service, Stand-alone Hosted and On-premise subscription for most of the fortune 500 companies. New service pack is released and deployed (includes feature changes or major fixes to the defects) once every 2 months in a calendar year to all the customer instances. Also a maintenance pack is released twice a month in a calendar year that includes minor fixes for the defects reported between the release timeline. All the above products once crosssold and deployed as individual products are now deployed as an integrated suite, i.e. all users accessing the integrated suite will be able to access respective product(s) or sub-product(s) as per their role permissions defined by the global administrator of the product suite.
The defect dataset constitutes defects from all the products and sub-products of the integrated suite that are extracted from the defect database of a defect tracking tool called JIRA™. Dataset contains defects raised by QA teams every sprint cycle along with defects reported by customers post product deployment. The authors worked with quality assurance teams and customers to extract the defects from the sprint cycles and evaluated the data using product managers’ inputs.
To understand the need of studying defect dependency, we provide a real time industry scenario consisting of three defects reported in three different sub-products of IHRIS software: Scenario: A manager uses the performance management sub-product to perform an employee’s year-end performance assessment. The Manager rates employee’s performance (between 0-5) along with comments. As per the manager rating, a pre-defined compensation hike shall be added to the employer salary in compensation sub-product along with relevant tax calculations as per policy in payroll sub-product. Defects: The sensitiveness of appraisal data necessitates encryption while storage. So, decryption was necessary to view the data in other modules. Defect #191 is raised, as the decryption method is not honored by the numeric data in manager comments. Later defect #278 and #286 were recorded due to defect #191 but were practically difficult to trace within a complex product without performing a defect dependency study. Observations: These three defects appear to be linked, however software quality teams normally would not have proactively identified defect #278 and #286 unless customers reported them. Defect#191 caused malfunction to compensation and payroll calculation. In cases like these, defect dependency study helps in detecting such defect spread and help product managers to prioritize defects accordingly.
Module (Product)
Below are the details of study workflow and teams involved.
The study was conducted over three service packs along with five maintenance packs of the above provided integrated software suite. The study was done over a period of 9 months between September 2014 and July 2015.
The entire defect dataset of integrated product has been chosen and equivalence classes have been generated for all the sub-products and products.
Defect dependency metric is applied over the equivalence classes and the metric value is calculated for all the defects identified by quality assurance (QA) team during every weekly sprint cycle.
These defects include defects recorded during sprint cycle and defects raised by customers together. The metric results are combination of two sources (QA team and customers).
QA team will evaluate the results of the metric over post release defects and compare them with the current defects recorded during sprint cycle for regression. Primary aim of this exercise is to avoid the possible spread of defects in upcoming release version.
The value of defect dependency metric is the indicator for improvement study. QA teams progressively compare the metric values every release and sprint cycle.
It has to be noted that there is no specific scale for this metric as it always depends on size of the defects and attributes (products chosen to evaluate) from dataset. QA Team shall present the results to product management team so that defects can be prioritized and an executive decision can be taken on implementing a plan for a new feature for a stable product(s) or subproduct(s) in upcoming service packs.
This section describes the steps involved on calculating the metric using the industry dataset with specific. Each defect in this dataset is a data point. All subproducts are considered as subset i.e., there are 16 subproducts spread across 5 product pillars (shown in Table I). For example, if Web Services Manager is a pillar product, Export Mgmt., Integration Mgmt., and Web Service Admin mode are its subsets.
Each set contains defects of its sub-product and they are entitled to be calculated together. Let D superset which contains defects of all sub-products i.e.
D = {p1 U p2 U p3 U…………….. U p16} pi represents 16 sub-products from the enterprise product suite under union of D the superset.
Equivalence relation is constructed using all the pi sets considering all the entities of the individual sets Equivalence classes are created for each pi set generating the classes of values that are common to all the pi sets.
All equivalent classes of pi sets are now passed to calculate Г(p1, p2,…, p16) to generate overall defect dependency metric D* D* is now the metric standard for all the input pi set of defect for a specific release. This activity needs to be continued for every release to understand the dependency of a defect over pi sets used to calculate D* Post every release (including service pack and maintenance pack), D* values are compared and reviewed to identify the improvement.
All the above steps are programmatically implemented using .NET 4.0 and SQL. Additional details in this regard are provided in the next section.
In addition to the standard testing process, QA team and product managers executed the below implementation and evaluation plan for of the defect dependency metric. Fig. 1 shows the implementation flow of the study setup. JIRATM is hosted against Microsoft SQL Server 2008 R2 at database level. Below ‘D’ is the JIRA defect database which stores defects raised by customers post product release and QA team during sprint cycle.
Using a data extract package (designed using Microsoft SQL Server Integration Services 2008 R2), we extract desired defects from available sub-products from the entire product suite. The data extract package contains SQL query logic to extract the defect dump for all the sub-products. This package pushes the defect dump to a testing database (T). We use this testing database to implement defect dependency metric. We construct another package called metric package (M) that contains the SQL query logic to construct equivalence relation and equivalence classes of sub-products chosen for metric calculation. Using .NET Code and SQL, D* is calculated and stored in testing database.
JIRA Defect Database
(D)
Data Extract Package (E) Testing Database (T)
Metric Package (M)
Reporting Solution (R)
The implementation cycle is repeated during every release and every sprint cycle so that our QA teams can analyze and compare the metric results for taking fair decisions on improving product quality and defect prioritization. Product Managers and QA teams depend on Reporting tool (R) to visualize the trend of the metric periodically to understand and decide whether the results are conflicting or making real sense in practice.
We found interesting results across different version releases of our integrated software product. Table II contains the detailed trend data of metric values captured per product across entire produce suite specific to the released versions. Here {V1, V2, V3} being the service pack releases and {V1.1, V1.2, V1.3, V2.1, V2.2, V3.1, V3.2, V3.3} are the maintenance pack releases. V1 is the considered as major service pack release and V1.1, V1.2 and V1.3 are its subsequent maintenance pack releases. Apart from these values, our QA team captured the metric values for every sprint release separately and for customer defects on weekly basis.
If we carefully observe, we can find the defect dependency values to be high in initial version V1. This was the base version of the implementation. We first calculated the metric value for V1 version to analyze the health of the current integrated software suite and found that it had high defect dependency value of 6.78. Human Resource System. (HRS) product was found to have high defect dependency value across overall product suite whereas Web Service Manager (WSM) was found to have low values. We started implementing the approach across different releases and found a significant changes in the quality of product and also a downtrend in the values of overall metric result for every product within a given specific version i.e. if we consider an example, in case of Learning Management System (LMS) the metric dropped down from 1.84 to 0.99 from service pack version V1 and by end of release of maintenance pack V1.3 which signifies improvement and stability in the product. Similar trend was identified across other product pillars in the enterprise suite. Our QA team has found significant improvement in terms of quality of product as the widespread of defects are diminishing by end of stable release as observed by the decrease in metric value for the products in below table.
Fig. 2 is the graphical representation of values from Table II highlighted in bold and italic, provides the trend analysis of the metric values across all products across version. We find a significant downtrend during the end of every version i.e. from V1 to V1.3, V2 to V2.2 and V3 to V3.3. We were able to minimize the various dependent issues across the integrated suite raising the quality levels of the entire product. This methodology helped QA teams and Product Managers to prioritize and de-prioritize defects with developers. For example, the Sustenance Engineering team responsible for providing fixes by end of upcoming release of a service pack or maintenance pack was able to select a particular defect that needed fix in a particular release cycle.
As per Fig. 2, from version V1 to version V3 we find a rise in dependency issues on every standard service pack release i.e. V2 and V3. We studied causes of this increase and found that rise in metric is due to dependency among the new features introduced in the respective pillar products. However, as the maintenance pack(s) were released with subsequent fixes, we found downtrend in metric results within a version, i.e., V2.1 and V2.2. At the end of every version, we were able to determine the impact of most of the defects. This led to prioritization of addressing high defect modules thereby easing the dependency of the defect to specific part of the product and decreasing it’s widespread.
We present our observations partially based on the retrospective session conducted between Product Managers and QA teams for trend analysis. It became tough to gain confidence from Product managers in initial sprint cycles, as the defect dependency was too high which brought down initial confidence levels. Also as the approach was mathematical (based on rough set theory), the QA team didn’t seem to comprehend the methodology in the beginning. As a result we had to spend some time negotiating for adoption of the approach within the Quality assurance team. 6.78 5.04 4.27 QA teams have come up with improved test cases as part of future integrating testing, as traditional test cases are no longer contributing towards product quality.
In summary, defect dependency metric was one of the key contributors along with our standard processes for stabilizing our integrated software product to a greater extent. The QA team gave informal feedback that the metric was of great value and product managers stated that it has helped improve customer success across customer subscriptions.
Our approach to calculate degree of Defect dependency metric is based on rough set theory. We implemented it against a real time defect dataset to improve and evaluate the quality of our large scale integrated software product during every release cycle since September 2014 to July 2015. We were successful in improving the integrated product suite. The main concern with our case study just like other case study papers is the possible extension and applicability of the work to other defect datasets. Given that we have applied it only to a single product suite, we can’t convincingly state that it’s applicable to other product suites too. However, it needs to be noted that our case study was based on an integrated software product that is used by most of the fortune 500 companies. It would be interesting to see if this methodology is adopted in tools used to build integrated software from mid-size software industries to large scale industries to understand its significance in reality. We believe that apart from defect dependency metric, heuristic approaches can also be used to solve our day-to-day quality issues. However we suggest fellow software practitioners to adopt our approach to improve software quality of their products. The scope of defect dependency metric is only to identify dependency of defect i.e. it’s widespread; however an integrated software product can still be un-stable with no defect dependency. This can be because of poor functional and architectural design or due to control/data flow issues.
On the other hand, organizational constraints and its corresponding influence on the accuracy of metric can be questioned. However a series of evaluation by quality teams and meetings with product manager and key stake-holders of the project(s) helped us evaluate the efficiency of the metric during every release. Influence of teams with lack of process knowledge, skill set or technology used can be argued and the results may be interpreted differently at times. To limit this issue, the evaluation of this metric has to be attributed to only key decision makers within the organization.
In current study, we have implemented this metric only on product & sub-product defects. As an extension to this study, we will be working on alternate methods to identify dependencies and widespread of defect on various other artifacts at different levels of software production like requirement analysis, resource planning, integration strategy, maintenance and design. This will help an integrated software company to address quality issues at all levels. Lessons learnt by conducting such studies can address some of the open challenges and help take efficient decisions to produce better complex products. As a future work, we will be assessing the metric more comprehensively by getting feedback from developers and quality teams on how significant this method helps them to prioritize the defect as part of regular work. We will have to work on testing strategies while adopting this approach in real time so as to improve test cases and address proactive defects especially during maintenance phase.
We thank all the members of product management, quality assurance and deployment teams at SumTotal Inc. for providing the valuable assistance, suggestions and feedback on implementing our research.
REFERENCES