Reliability assessment and prediction of the number of faults/defects is an important part of the software engineering process. Many software reliability models assume that all detected are removed with certainty and no new faults are introduced. However, the introduction of secondary faults during software updates has become quite common in software development practice, which can be explained by the enormous complexity of modern computer applications. In the paper we consider different scenarios of introducing secondary faults and how to predict number of such faults. Finally, we discuss how different SRGMs like Jelinski-Moranda, Exponential, Schick-Wolverton, Musa and Lipov models can be modified to account secondary faults in order to improve accuracy of software reliability prediction. We use an industrial case study to demonstrate applicability of the proposed approach. Our results show that considering secondary faults helped to considerably improve accuracy of software failure rate prediction.
Software (SW) has become a ubiquitous component and an important part of modern information and communication systems. As a result, software failures caused by faults made during software development and overlooked during the testing process can have severe consequences and lead to largescale outages of computer systems, significant financial losses and even human casualties.
Thus, assessment of software reliability is one of the key issues that arise during SW development,
verification, and validation. An importance of software reliability assessment is emphasized by the need
to comprehensively account its’ impact on reliability of computer systems for critical and
businescritical applications [
2022 Copyright for this paper by its authors. over time due to ageing, while software cannot age physically) and can predict the mean time to failure (MTTF) based on failure statistics collected during software testing and operation.
A large number of software reliability models have been proposed over the past decades. A model is a description of the relationship between two or more variables (e.g. the number of already detected faults/defects and the time to the next failure).
Model assumptions are used to simplify model development. They denote a set of used conventions,
choices and other specifications on which the model is based. Many SRGMs contain an assumption that
in the process of eliminating already detected (primary) faults, new (secondary) faults cannot be
introduced [
The paper proposes a new technique for prediction the number of secondary faults. This technique has been incorporated into Jelinski-Moranda SRGM to improve accuracy of software reliability assessment. An industrial case study was used to justify applicability of the proposed approach and to demonstrate its efficiency.
Over two hundred different software reliability models have been developed over the past decades. These models define the set of assumptions about the number of faults (finite or infinite), fault rate, time between failures, etc. and offer analytical equations forecasting the number of residual failures, the failure rate and mean time to failure. A large set of different assumptions is caused by the wide range of different factors in the process of software development, testing and operation affecting conditions of faults creation, detection and manifestation [14-23].
There have been quite a few publications attempting to systematize and classify existing SRGMs
based on the assumptions they use:
• Hecht’s classification [
Ultimately, we can conclude that software reliability models can be divided into three main classes: (i) empirical models, (ii) statistical models and (iii) probabilistic models. The class of probabilistic models seems to be the most suitable to consider and take into account probability of inserting the secondary faults during the process of removing the primary faults. We selected a subset of probabilistic models which risk functions can be modified to account probability of inserting the secondary faults: Jelinski-Moranda, Simple exponential, Schick-Wolverton, Lipov and Musa models. In the next section we will consider scenarios of introducing secondary defects during the software life cycle and how the risk functions of the software reliability models mentioned above can be updated to account for secondary faults in order to improve the accuracy of software reliability predictions.
It is assumed that software contains Nd initial faults and undergoes a series of updates/upgrades during its lifecycle. The number of faults Mi left in the software after the i-th update/upgrade can be estimated using (1) depending on one of the following assumptions: 1. All faults Ni are removed with certainty and no new faults are introduced; 2. All faults Ni detected by the i-th update step are removed with certainty; however, Ki new (secondary) faults are introduced after software is updated; 3. Not all faults out of Ni detected by the i-th update step are removed after updating the software; ∆Ni faults are left unfixed, but no new faults are introduced as a result of software update; 4. Not all faults out of Ni detected by the i-th update step are removed after updating the software; ∆Ni faults are left unfixed; in addition, Ki new (secondary) faults are introduced as a result of software update; 5. All faults Ni detected by the i-th update step are removed with certainty; however, K*i new faults are introduced after upgrading software functionality (i.e. adding new functions); 6. Not all faults out of Ni detected by the i-th update step are removed after updating the software; ∆Ni faults are left unfixed; in addition, K*i new faults are introduced after upgrading software functionality (i.e. adding new functions); 7. All faults Ni detected by the i-th update step are removed with certainty; however, Ki* new faults are introduced after upgrading software functionality (i.e. adding new functions); in addition, KiB interaction faults (i.e. faults occurred during interaction of upgraded software modules with nonupgraded ones) are introduced into the program. 8. Not all faults out of Ni detected by the i-th update step p are removed with certainty; however, Ki* new faults are introduced after upgrading software functionality (i.e. adding new functions); in addition, KiB interaction faults (i.e. faults occurred during interaction of upgraded software modules with non-upgraded ones) are introduced into the program. (1) where λ(t) – risk functions, Fi-1 – the total number of corrected defects at the time, Xi – test time from ti-1 (time of detection of i-1 software defect) to ti.
The process of predicting the number of secondary SW faults can be split into the following steps. Step 1. Collecting the SW defects statistics.
As an input this step uses the following information [
The output of this stage is statistics about failure occurrence and fault detection. It is obtained based on the analysis of testing and validation reports and code reviews.
Step 2. Analyzing failure occurrence and fault detection statistics as a function of time.
Failure occurrence and fault detection statistics in each time interval collected at the previous stage is used to analyze its time correlation.
Step 3. Selecting the regression function.
Step 4. Evaluating regression coefficients.
Step 5. Predicting the number of secondary SW faults. 4.1.
Below we define steps for predicting the number of secondary SW defects.
1. Calculating the module of the difference between the actual number of software defects detected in the specified time interval and the value predicted by the regression function for the same period of time.
2. The number of secondary SW faults in the i-th testing interval can be estimated as the difference between the results obtained in the previous step and the standard deviation of the fault statistic in the specified time interval multiplied by the coefficient (2) and rounded to the nearest whole number. where n – is the total number of testing intervals.
As a result, we can derive the following equation:
1 n + 1 − i
, = y −
− b − a x
1 n + 1 − x y , (2) (3) where y – is statistics of software faults detected in each testing interval; x – is the sequence number of the testing interval; a, b – are coefficients of the regression function; n – is the total number of testing intervals; y – is the standard deviation of y; – is the estimated deviation value.
The number of secondary faults then can be estimated by rounding to the nearest whole number.
RadICS’s (see Fig. 2) is the digital instrumentation and control platform used in safety and control systems applications in operating nuclear power plants [23]. It is a modular platform which includes a set of replaceable standardized modules such as logic module, digital and analog input/output modules. The functionality of each module is driven by the program logic implemented in the on-board FPGA(s). FPGA (Field-Programmable Gate Array) is an integrated circuit designed to be programmed through the use of hardware description languages, e.g. VHDL or Verilog [26, 27]. FPGA code is written in a description language, then is interpreted, synthesized, and ultimately produces hardware. As such, faults in the FPGA program code can introduce logic errors into the system.
In this section we report fault detection statistics (see Table 2) for RadICS’s Logic Module collected during its functional testing and apply the discussed approach to predict the number of secondary faults. The Logic Module serves as the core of the entire RadICS platform. It implements the application logic and is used as a communication node linking other modules installed in the chassis together, performs self-diagnostics and controls communications with external chassis and systems.
Figure 4 depicts the fault detection statistics (NS_1) collected during functional testing and the power regression function (4) used for theoretical approximation of the testing results (NS_2). Values of the parameters of the regression function were estimated by the least-squares technique. y = 12.66/x + 2.31 (4) 16 14
The number of secondary software faults nin was calculated using (3) by pairwise comparison of NS_1 and NS_2 series. Obtained results are summarised in Table 3.
The number of predicted secondary faults can now be used to estimate the software failure rate λd with the help of software reliability growth models. Table 4 compares software failure rates predicted using Jelinski-Moranda SRGM with and without considering secondary software faults. secondary faults. It is shown that the total number of faults detected during the testing period is equal to 67. The number of predicted secondary faults is 8 (approx. 12% of the detected faults).
Thus, one can assume that the potential number of software faults equals 75 which means that the software needs to be further tested to detect and remove the rest of the faults. 1,667813 2,031909 1,108970 1,123758 0,706515 0,063928
Testing intervals, month 1 2 3 4 5 6 7 8 9 10 11 12
This paper discusses an importance of software reliability assessment. A great number of models have been proposed to predict software reliability (probability of failure, failure rate or mean time to failure) and the number of residual software faults.
Different models use different approaches to evaluate software reliability. For instance, one of the first attempt to predict the number of software faults was made by Maurice Halstead in 1977 in [23] where he proposed a number of program code’s ‘complexity’ metrics used as predictors of program defects.
In this paper we consider a class of software reliability growth models which use fault detection statistics collected during software testing. These models put forward different assumption about statistical distribution of failures (e.g. time between failures follows the exponential distribution), number of faults (e.g. finite or infinite) and other limitations to simplify the process of software reliability assessment.
However, some of these assumptions might not be very realistic and, hence, affect accuracy of the reliability prediction. For example, many SRGMs assume that all detected faults are removed with certainty and no new faults are introduced in the software. However, our industrial experience and other studies show that the introduction of secondary faults is quite common and, hence, needs to be accounted during software reliability modelling.
In the paper we discuss different scenarios of introducing secondary faults and how to predict number of such faults. Finally, we discuss how different SRGMs like Jelinski-Moranda, Exponential, SchickWolverton, Musa and Lipov models can be modified to account secondary faults in order to improve accuracy of software reliability prediction. We use an industrial case study to demonstrate applicability of the proposed approach. Our results show that considering secondary faults helped to increase accuracy of software failure rate assessment up to 5%.
1977.