Reliability is one of the key attributes of software product quality. Capability for accurate prediction of reliability will allow software product industry to have better market acceptability and enable wider usage in high integrity or critical applications domains for their product. Software Reliability analysis is performed at various stages during software product development life cycle. Popular software reliability prediction models proposed in literature are targeted to speci c phases of life cycle with certain identi ed parameters. However, these models seem to have certain limitations in predicting software reliability in an accurate and acceptable manner to the industry. A recent industrial survey performed by the authors identi ed several factors which practitioners perceived to have in uence in predicting reliability. Subsequently we conducted a set of experiments involving diverse domains and technologies to validate the perceived in uence of the identi ed parameters on software product reliability which was evaluated using SonarQube. In this paper, we present our evaluation approach, experimental set up and results from the study. Through these controlled experiments and analysis of data, we have identi ed a set of in uential factors a ecting software reliability. This paper sets direction to our future research on modeling software product reliability as a function of the identi ed inuential factors.
Quality is de ned as \capability of a software product to conform to
requirements." as per ISO/IEC 9001[
In the past many models for predicting software reliability have been
developed and studied extensively. However, these models are applicable under
certain assumptions and for speci c phases of development life cycle[
This paper evaluates the signi cance of such identi ed in uential
environmental [
These experiments were performed in a large software development
organization in India, which has laboratory setup necessary for performing such
experiments. These laboratories basically serve as training centers for employees, both
for new recruits as well as part of continuous learning. We have identi ed a set
of independent input parameters [
Paper organization: Section 2 discusses the background and some details regarding experimental context. Section 3 gives methodology followed for performing experiments. In section 4 the experimental results are presented along with discussion. Section 5 gives conclusion based on analysis done in previous section. 2
In this study, we hypothesize reliability to be a function of defect leakage,
postdelivery defects, schedule variance, e ort variance, productivity, technology,
commercial o the shelf(COTS) complexity, design complexity, unit test defects,
integration test defects, system test defects, execution time and skill level of
developer/ tester. These factors were identi ed as the most in uential ones as
perceived by stakeholders such as product users, coder, tester, designer,
product managers, a ecting software product reliability during the industrial survey
conducted by the authors[
Experiments involved studying impact of these factors on reliability. Reliability is computed by varying one factor while maintaining all other factors constant. For example, skill level can be varied (from high to low) while keeping all other factors constant.
For each application, we performed minimum 30 combinations. For example:
If skill level was identi ed as variable factor then all other factors such as
techhhLiioggwhh,, medivuemry, cBoamsepdlexointy emxepaesrutrejsudgments and
C#, .NET, C#, .NET used in one application
Sharepoint, ASP and considered as one level
0-5 Simple Complexity of COTS is based upon
6-10 Medium a. # of internal interfaces b. Impact
11-15 Com- factor c. # of calls through main
plex program
nology, hardware, rmware, tools were kept constant. For four applications, we
performed overall 120 experiments in the laboratory in which 560 people of
different roles and skill levels participated. Reports on reliability [
In this section we also present reference of activities performed before
entering experimentation area. Activities were targeted at software product reliability
literature review and also conducting survey with practitioners and experts
identi ed in the industry across globe. In the literature survey [
The realized software projects have been developed and managed as per
ISO 9001:2015 standard and CMMI measurement and analysis, project
monitoring and control issues. In these experiments, we captured data related to four
products, which have been used for commercial purpose across the globe. All
considered products can be classi ed as application in di erent domains and are
listed in Table 3. Industrial standards proposed by Halstead were considered for
categorizing design complexity of the applications[
For collecting data, we took help of di erent tools such as Jira, Rational Team Concert (RTC) and Team Foundation Server (TFS). These tools were used for collecting defects in requirement and design phase. E orts and Schedule related data was captured using Microsoft Project Plan (MPP). We used GIT for conguration management and Rational Functional Tester (RFT) and Quick Test Professional (QTP) for testing automation. Other code quality related parameters were captured using PurifyPlus. 3
Experimentation is powerful tool in software engineering. The main objective of
performing experiments is to nd cause and e ect relationship [
Run SonarQube and capture reliability R(Cijk)
Yes Are Runs Over for (Ai,Fj )? No For every application and every independent factor repeat experiment Ranking of factors and Validation through MTBF
Yes
Prepare graph of R versus independent factor Fj (e.g. skill) Calculate Correlation between Reliability & Fj
Correlation >= 0.8?
No Correlation < 0.5?
No
Yes High Correlation
CMoerdreiulamtion LCoowrrelation
were conducted in a multinational software product organization having centers
across the globe. Series of experiments were conducted in controlled environment,
where one parameter is considered as variable and other parameters are taken
as constant [
The methodology used for performing the experiments is shown in Figure 1. For example, in experiments to study impact of skill on reliability, one functionality was identi ed of an application and task of developing it was assigned to software developers having varying skill levels. Minimum of 10K lines of code was the criteria set for developing the application. The design document was provided to all developers. Design complexity (input variable) along with other identi ed input parameters were kept constant. SonarQube was run on error free code to give the reliability factor for each skill level. To statistically conclude, more than 30 data points were recorded. By using this methodology, experiments were preformed for other identi ed factors. 4
In this section we present our experiment ndings and rank attributes in uencing
reliability. Chi-Square Test was used to statistically test whether parameters
are having any impact on reliability [
Table 4 summarizes output of "R" for di erent skill levels for various technologies. In all above cases, probability value (p-value) for acceptance of null hypothesis is calculated and if the p-value is less than 0.05 then the null hypothesis is rejected. It can be seen from Table 4 that irrespective of the technology used, there is good correlation between skill level and reliability.
As a sample, in Figure 2 we show the scatter plot of skill versus reliability. The
pattern of the resulting points reveals that there exists correlation [
Validation of results is also done using mean time between failure observed during testing and operational phases. This method is adopted for each application. Mean time between failure is judged based on operational discontinuity of an application. It is assumed that in case of critical, very high, high and medium type defects, application can behave di erently and reliability of an application can be hampered. Sometimes during use, an application does not execute certain part of the code which can be a threat to the overall experimenting exercise. We covered maximum code during execution and checked all branches and nodes in the code to con rm the reliability gure. This is done through writing test cases for each branch and node identi ed for all features mentioned in the applications.
By careful design of the study and involving a broad spectrum and su ciently large number of respondents across the organization, we have been able to eliminate most usual issues of external validity and reliability in empirical studies. However, one cannot rule out the possibility that we might have omitted yet another important factor from our list.
Table 5 summarizes the impact of other factors on reliability showing only absolute values of correlation factor. It can be concluded that reliability is strongly correlated with Skill factor, Post Delivery Defects and Review E ciency, whilst reliability has good correlation with COTS complexity, System Test Defects, Design Complexity and Load Condition. The last two columns show the average and rank of the factor based on its in uence on reliability.
Current models are considering defects from eld and internal in testing phase. However they are not considering factors like skill of developer and tester or review e ciency in development process. 5
One of the noteworthy ndings from these experiments are factors like postdelivery defects, skill and review e ciency contributes signi cantly towards software product reliability and hence should be included in its prediction. With the help of this exercise, we could also eliminate (or at least keep on backstage) some parameters such as process metrics (Schedule Variance, E ort Variance and Productivity), Unit Test Defects, Integration Test Defect, System Test Defects. These experiments also indicate that load condition, Design complexity and COTS (in the order of increasing importance) could be signi cant in de ning software product reliability. Though further detailing is needed, we consider this as a good starting point for de ning an appropriate prediction model of software reliability. 6
We would like to thank Dr. Yogesh Badhe, data scientists from Persistent Systems Ltd. for his valuable support in data analysis. Also, we would like to thank Dr. Ramprasad Joshi for his valuable inputs for documentation while performing experiments. Punnekkat acknowledges support from FiC(SSF) Project.