Combinatorial robustness testing (CRT) is an extension of combinatorial testing (CT) to separate test suites with valid and strong invalid test inputs. Until now, only one controlled experiment using artificial test scenarios was conducted to compare CRT with CT. The results indicate advantages of CRT when much exception handling is involved. But, it is unclear if these advantages are also valid in the real-world. In this paper, we present the results of a case study conducted to compare the fault detection effectiveness of CRT and CT by testing an industrial system with 31 validation rules and 13 injected faults.
Robustness is an important property of software. It describes "the degree to which a system [...] can function correctly in the presence of [invalid inputs]" [1]. Invalid inputs are caused by external faults, i.e. faults in other systems or made by users interacting with a system. Examples are inputs to the system under test (SUT) that contain invalid values like a string value when a numerical value is expected, or invalid value combinations like a begin date which is after the end date. When invalid inputs remain undetected, they can propagate to failures in the SUT resulting in abnormal behavior or crashes [2].
Developers attempt to improve robustness of systems by implementing exception handling (EH) to detect and recover from invalid inputs. Unfortunately, EH is itself a significant source of faults (cf. [3,4]). Therefore, it is important to test the exceptional behavior as well.
Combinatorial testing (CT) is a black-box test method that is based on an input parameter model (IPM) [5]. When considering the exceptional behavior, an IPM must describe invalid values and invalid value combinations that trigger EH. Unfortunately, invalid values and invalid value combinations can cause input masking (cf. [6,7,8]). When a SUT is stimulated with an invalid input, the EH is expected to detect it, to respond with an error message, and to terminate the SUT without resuming the normal behavior. Consequently, the remaining values and value combinations of the test input remain untested as they are masked.
To avoid input masking, combinatorial robustness testing (CRT) is developed as an extension to CT using a robustness input parameter model (RIPM) being an extension of an IPM with additional semantic information to annotate values and value combinations as invalid [7]. With this semantic information, valid test inputs can be selected which do not cover any invalid value or invalid value combination. Further on, strong invalid test inputs can be selected which contain exactly one invalid value or one invalid value combination.
Due to the separation of valid and strong invalid test inputs, the input masking effect can be avoided when testing the normal behavior and the exceptional behavior. However, in comparison to CT which does not separate valid and strong invalid test inputs, CRT requires effort to model the additional semantic information.
Despite the presence of input masking, CT can still be effective in detecting faults as a previous controlled experiment indicates [8]. Nevertheless, the fault detection effectiveness (FDE) of CT decreases for systems with much EH. Even for high testing strengths and large test suites, the FDE of CT deteriorates. For systems with much EH, CRT is a promising approach that can achieve a higher FDE while requiring fewer test inputs than CT [7]. For systems with little EH, CRT is at least as effective as CT.
Although, the current assessment is solely based on one controlled experiment with artificial test scenarios (cf. [7]). Therefore, our objective is to further compare CRT with CT guided by the following two research questions. To answer these research questions, we conducted a case study. According to Kitchenham et al. [9], a case study helps to evaluate the benefits of methods and tools in industrial settings. When applied to compare methods and tools, a case study is of explanatory nature "seeking an explanation of a situation or a problem" [10]. As Runeson & HΓΆst state, a case study "will never provide conclusions with statistical significance" [10]. But it can provide sufficient information to help you judge if specific technologies will benefit your own organization or project" [9]. Since a case study has, by definition, a higher degree of realism than a controlled experiment [10], a case study that compares CRT with CT can provide additional insights that complement and extend the findings of the previously conducted controlled experiment.
The paper is structured as follows. Section 2 introduces basic concepts of CT and CRT. Related work is discussed in Section 3. Next, the design of the case study is introduced (Section 4) and its results are presented (Section 5). Afterwards, threats to validity are discussed (Section 6) before the paper is concluded in Section 7.
In the following, CT and CRT are briefly introduced. For more information, please refer to [11,5,7].
CT is a black-box test method [5]. It is based on an input parameter model (IPM) which declares π parameters and each parameter is associated with a non-empty set of values. A schema is a set of parameter-value pairs for π distinct parameters [12]. A schema with π = π parameter-value pairs is a test input. A schema π covers another schema π if and only if schema π includes all parameter-value pairs of schema π.
Real-world systems are often constrained and certain values should not be combined to schemata and test inputs [5]. These schemata are irrelevant because they are not of any interest for the test. Test inputs that cover irrelevant schemata are irrelevant as well and their test results have no informative value. Hence, they should be excluded from testing.
Constraint handling is often used to exclude irrelevant schemata [13]. Therefore, irrelevant schemata are explicitly modeled by a set of logical expressions (called exclusion-constraints). A schema is relevant if it satisfies all exclusion-constraints. A schema is irrelevant if at least one exclusion-constraint remains unsatisfied.
A coverage criterion is a condition that must be satisfied by a test suite. A test selection strategy describes how values are combined to test inputs such that a given coverage criterion is satisfied [11]. Test suites resulting from a test selection strategy that supports constraint handling, e.g. I P O G -C [13], satisfy the π‘-wise relevant coverage criterion. This criterion is satisfied if the relevant test inputs of a test suite cover all relevant schemata of degree π = π‘ that are described by an IPM [11,5].
To avoid input masking, CRT is developed as an extension to CT that separates valid and invalid test inputs [7]. To better separate the concepts, we say that CT relies on IPMs while CRT relies on robustness input parameter models (RIPM). A RIPM contains additional error-constraints which is another set of constraints to annotate relevant schemata as invalid. A relevant schema is also a valid schema if it satisfies all error-constraints. A relevant schema is an invalid schema if at least one error-constraint remains unsatisfied. Further on, an invalid schema is a strong invalid schema if exactly one error-constraint remains unsatisfied.
Test selection strategies like R O B U S T A [7] not only consider exclusion-constraints to exclude irrelevant schemata, they also consider error-constraints and exclude invalid schemata from valid test inputs. Further on, strong invalid test inputs are selected such that each invalid value and invalid value combination that is modeled by error-constraints appears in strong invalid test inputs.
Valid test inputs are selected to satisfy π‘-wise valid coverage. The π‘-wise valid coverage criterion is an extension of the π‘-wise relevant coverage criterion. It is satisfied if all valid schemata with a degree of π = π‘ that are described by a RIPM are covered at least once by a valid test input.
Strong invalid test inputs are selected to satisfy π-wise strong invalid coverage where π denotes the robustness interaction degree. Without robustness interaction (π = 0), the coverage criterion is called single error coverage (cf. [11,7]). It is satisfied if each invalid schema that is described by an error-constraint appears in a strong invalid test input. With robustness interaction (π β₯ 1), each described invalid schema is combined with all valid schemata of degree π = π. The coverage criterion is satisfied if all combinations of invalid schemata and π-sized valid schemata are covered by strong invalid test inputs.
Following these brief introductions of CT and CRT, the conceptual difference between the two approaches should become clear. CT and CRT use the same parameters and values. But CT does not distinguish between valid and invalid schemata. Instead, both types of schemata are mixed and the FDE purely relies on the combinatorics, i.e. different testing strengths π‘. In contrast, CRT distinguishes valid and invalid schemata to avoid the effect of input masking. Here too the FDE relies on combinatorics but the avoidance of input masking has an additional influence.
8th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2020)
CRT requires the effort to model error-constraints. Test selection strategies that consider error-constraints also become more complex. This raises the question whether the avoidance of input masking outweighs the additional effort and complexity of CRT. Until now, only artificial test scenarios are used to compare CT with CRT (cf. [7]) and it remains unclear if indicated advantages of CRT can be transferred to real-world scenarios. Therefore, this case study was conducted.
To the best of our knowledge, Sherwood [6] first mentioned invalid values in the context of C A T S which is a test selection strategy and tool for CT. Cohen et al. [14] and Czerwonka [15] also acknowledged the necessity to separate valid and strong invalid test inputs. They also published test selection strategies and tools and the IPMs contain semantic information to distinguish relevant from irrelevant schemata and to distinguish valid from invalid values. However, invalid value combinations are not directly supported. Therefore, we proposed R O B U S T A and the structure of RIPMs with error-constraints [7].
Many studies exist that demonstrate the usefulness and effectiveness of CT (cf. [16,17,18]). But most studies do not distinguish between relevance and validness and focus on testing the normal behavior.
One case study by Wojciak & Tzoref-Brill [19] reports on applying CT and also considers testing with invalid inputs. They report that single error coverage was not sufficient because EH depended on interactions between invalid and valid values. In particular, "the same [exception] would often be handled differently depending on the firmware in control [...] or depending on the configuration of the system". A further remark is concerned with the ratio of valid versus invalid test inputs: "Since a lot of attention was given to [robustness] testing [...] where full recovery in the presence of [exceptions] was expected, the [test suite] contained a ratio of up to 2:1 [invalid test inputs vs. valid test inputs]. "
In this section, the case under analysis and the data collection procedure are introduced.
The case is a development project conducted by an IT service provider of an insurance company, where a new software was developed to manage the life-cycle of life insurance contracts. One subsystem of the software is concerned with the validation of insurance application data according to a set of validation rules and with forwarding the data when it satisfies the validation rules. It is the same project which we analyzed in a previous case study (cf. [18]).
Altogether, 31 validation rules are defined to check insurance application data. The order of the validation rules is predefined and all validation rules are traversed for each insurance application data. Whenever a validation rule is not satisfied by an insurance application, a corresponding error code is returned and the remaining validation rules are skipped. If all validation rules are satisfied, the subsystem returns S U C C E S S and the insurance application data is further processed. Although, the further processing is out of scope for this case study.
Each validation rule is built as an implication consisting of two parts:
The first part determines whether a given validation rule is applicable to the insurance application data or not. If a rule is applicable, the insurance application must not violate the rule, i.e. isValid(application). Otherwise, the validation rule is ignored.
Because details of the case are confidential, a generic example is given to provide further illustration of validation rules. The example depicts two validation rules to define maximum sums that can be insured depending on the permissions of the insurance agents. The first validation rule is applicable to all applications created by insurance agents with the highest level of permission. The second validation rule is applicable to all applications that are created by insurance agents with lower permission level.
The distinction between the two validation rules is made by the first part of the implication: Rule 1: i s A p p l i c a b l e (application) βΆ a p p l i c a t i o n .agent.permission = h i g h e s t _level Rule 2: i s A p p l i c a b l e (application) βΆ a p p l i c a t i o n .agent.permission β h i g h e s t _level
The second part of the implication is used to enforce the maximum insured sum. As an application may consist of several partial contracts, the individual insured sums of all partial contracts are collected first. Afterwards, it is checked whether the total sum exceeds the threshold. While the structure of both rule's isValid() parts is the same, different values for the m a x i m u m _insured_sum constant are used: This example shows that many parameters may be involved in a validation rule, that intermediate calculations may be required, and that intermediate calculations may be reused in different validation rules. Therefore, all validation rules should be tested thoroughly.
For this case study, we consider the current set of validation rules as correct and treat them as our specification. By browsing the source code repository, we have identified 13 changes that have been made to the validation rules in order to correct them. Each change documents a fault that existed previously but is fixed prior to release. Based on these 13 changes, we reconstructed 13 implementation versions of which each contains one fault.
The 13 faults can also be classified according to our robustness fault classification (cf. [7]). Five faults can only be detected by invalid test inputs, while eight faults can be detected by both valid and invalid test inputs. Two of these five faults can be classified as faults in error-signaling. To reveal them, invalid test inputs must trigger EH which responds with an incorrect error code. The other three faults can be classified as faults in error-detection conditions. The conditions are too weak and do not detect invalid test inputs. Hence, the SUT incorrectly continues with its normal behavior.
The remaining eight faults can be detected by both valid and invalid test inputs. They are faults in error-detection conditions. Four of theses faults have conditions that are too strong and therefore incorrectly detect exception occurrences for valid test inputs. The other four faults have characteristics of being too weak and too strict at the same time because wrong parameters with similar characteristics are used in the exception condition. As a consequence, an invalid test input may not violate the condition (too weak) while a valid test input may not satisfy the condition (too strong).
Data collection refers to the measurement and calculation of metric values from test execution. Therefore, metrics are defined in this section. Furthermore, the modeling of the IPM and RIPM as well as the selection and execution of test inputs is described.
The resources available from the software development project are not directly analyzed and compared. Instead, they are used to reconstruct the implementation versions for test execution and to create a RIPM and an IPM that represent variations of insurance application data.
Based on the RIPM and IPM, test inputs are selected using a CT and a CRT test selection strategy. Then, the test inputs are executed on the 13 reconstructed implementations to assess the effectiveness.
A common metric to assess the effectiveness is fault detection effectiveness (FDE) [11,16]
Since the FDE and AFDE metrics highly depend on the quality of the RIPM and IPM, a systematic modeling approach is necessary. We model the IPM first and later extend it with error-constraints to get a RIPM. The IPM is modeled iteratively for one validation rule at a time. In each iteration, parameters and values are added to ensure that test inputs with the following three characteristics can be detected: (1) test inputs that are not applicable; (2) test inputs that are applicable and valid; (3) test inputs that are applicable but not valid. In addition, some exclusion-constraints are introduced to ensure syntactic correctness of selected test inputs. The IPM is considered as complete once the IPM contains all parameters and values necessary to satisfy branch coverage of each validation rule.
For the RIPM, the modeling of additional error-constraints is required. The error-constraints are modeled iteratively and we add new or update existing ones until the separation of valid and strong invalid test inputs conforms to the responses of the SUT, i.e. the SUT returns S U C C E S S for each valid test input and the SUT returns an error code for each strong invalid test input.
In total, the IPM and RIPM consist of 32 parameters and 106 values. Most parameters have two, three, or four values each. But two parameters have six values each 8th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2020) and one parameter has even nine values. Three exclusion-constraints of which each restricts combinations of two parameters are required to ensure syntactical correctness of the insurance applications. Furthermore, the RIPM contains 31 error-constraints. 15 error-constraints annotate single values as invalid. The remaining 16 error-constraints annotate schemata with 2, 3, or 5 values.
The complete IPM and RIPM are described below in exponential notation. For parameters and values, π₯ π¦ refers to π¦ parameters with π₯ values. For exclusion-and errorconstraints, π₯ π¦ refers to π¦ constraints with π₯ parameters.
Parameters & Values:
Exclusion-Constraints: 2 3 Error-Constraints: 5 2 3 6 2 8 1 15
After creating the IPM and RIPM, both models are used to select sets of test inputs. Since we compare CRT with CT, two different test selection strategies are used. R O B U S T A is used to select test inputs for the RIPM and I P O G -C is used to select test inputs for the IPM.
To compare the FDE and AFDE of CRT with CT, test suites that satisfy different coverage criteria are used. We apply I P O G -C to select test suites that satisfy π‘-wise relevant coverage for π‘ β {1, ..., 5}. Furthermore, we apply R O B U S T A to select test suites that satisfy π‘-wise valid coverage with π‘ β {1, ..., 3} and that satisfy π-wise strong invalid coverage with π β {0, 1}.
To reduce the effect of accidental fault detection caused by ordering, the order of parameters and values of the input parameter models is randomly reordered and 20 different model variants are used to select test suites for each coverage criteria.
Table 1 depicts the average sizes of test suites that satisfy the different coverage criteria. Since R O B U S T A encompasses two coverage criteria (π‘-wise valid coverage and π-wise strong invalid coverage), the test suites are considered both, separately and combined.
The largest test suite is selected by I P O G -C which is required to satisfy π‘-wise relevant coverage with π‘ = 5 (15023.70 test inputs). The second-largest test suite is also selected by I P O G -C to satisfy π‘-wise relevant coverage with π‘ = 4 (2813.45 test inputs). The third-largest test suite is selected by R O B U S T A and satisfies π‘-wise valid coverage with π‘ = 3 and π-wise strong invalid coverage with π = 1 (2224.30 test inputs).
When comparing the test suite sizes of π‘-wise relevant coverage of I P O G -C with π‘-wise valid coverage of R O B U S T A , it can be seen that the error-constraints drastically reduce the number of valid test inputs.
After test input selection, the test suites are used to stimulate the SUT in 13 different versions. Therefore, the 13 reconstructed implementations of which each contains
In this section, the case study results regarding the computed FDE and AFDE values are reported and discussed. As can be observed, π‘-wise relevant coverage is not able to detect all faults reliably. The FDE values increase when testing strength π‘ grows. But even with π‘ = 5 (15023.70 test inputs), only 7 faults are detected reliably (FDE value of 1.00). Further on, fault no. 10 remains undetected (FDE value of 0) and faults nos. 9 and 13 are only detected by one out of 20 test suites (FDE value of 0.05).
The CRT coverage criteria are characterized by avoiding the invalid input masking effect. Since all invalid schemata are excluded by π‘-wise valid coverage, the faults 8th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2020) nos. 9 to 13 cannot be detected. But for all other faults, π‘-wise valid coverage has higher FDE values for the same testing strength π‘ when compared to π‘-wise relevant coverage. Because invalid input masking is avoided, a testing strength of π‘ = 2 is sufficient to detect faults nos. 1 to 8 reliably (FDE values of 1.00).
Using π-wise strong invalid coverage with π = 0, 11 out of 13 faults can already be detected reliably and the two remaining faults have high FDE values of 0.90 and 0.80. The effectiveness of robustness interactions is even higher and all faults can be detected reliably with π = 1.
Four faults that have too strong error detection conditions and that actually require valid test inputs to be detected are also reliably detected by π-wise strong invalid coverage. We could observe that a strong invalid test input that is expected to violate the error detection condition of the π-th validation rule is also expected to satisfy all prior validation rules from 1 to π β 1. Therefore, strong invalid test inputs can be considered as "partiallyvalid" test inputs that are able to accidentally detect faults that require valid test inputs. This effect is strengthened by robustness interactions because more test inputs are selected and more interactions are covered by them.
R O B U S T A combines π‘-wise valid coverage and π-wise strong invalid coverage and the FDE values show that test suites for both coverage criteria complement each other. Since valid and strong invalid test inputs are able to detect faults nos. 1 to 8, the FDE values are complemented by the combination of both test suites. For faults nos. 9 to 13, the FDE values are not complemented by the combination of both test suites. This is because test suites that only satisfy π‘-wise valid coverage cannot detect these faults. Therefore, the FDE values of the combined test suites are the same as the FDE values of the test suites that satisfy π-wise strong invalid coverage.
In order to detect all faults reliably, the π-wise strong invalid coverage must be selected because faults nos. 9 to 13 remain undetected otherwise. Either robustness interaction (π > 0) or the combination of π-wise strong invalid coverage with π‘-wise valid coverage is required to reliably detect faults nos. 1 to 8. Even though π‘ = 1 is only sufficient to detect three of the first eight faults reliably, the combination with π-wise strong invalid coverage improves the FDE and all faults can be detected reliably.
The discussion of the FDE shows which coverage criteria are appropriate to reliably detect different types of faults. Next, we discuss the AFDE over all 13 faults.
Because AFDE values are average values over a set of faults, AFDE allows making general statements about both the effectiveness and the efficiency of coverage criteria. First, we discuss the effectiveness in terms of AFDE values of different coverage criteria. Therefore, In direct comparison, test suites that satisfy π‘-wise valid coverage reach a maximum AFDE value of 0.62 as well. The same AFDE value can be reached because they prevent invalid input masking. However, the AFDE value cannot be further improved by increasing the testing 8th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2020) strength because faults nos. 1 to 8 are already detected reliably and faults nos. 9 to 13 cannot be detected by valid test inputs. Comparing the two coverage criteria for each testing strength individually shows that the AFDE value of π‘-wise valid coverage is always higher than the AFDE value of π‘-wise relevant coverage.
For π-wise strong invalid coverage, the lowest AFDE value is 0.98 (no robustness interactions) which is always higher than the AFDE values of π‘-wise relevant and valid coverage. Furthermore, π-wise strong invalid coverage with robustness interactions has an AFDE value of 1 and therefore detects all faults reliably.
Overall, the combination of π‘-wise valid coverage and π-wise strong invalid coverage performs the best and always detects all faults reliably.
When putting the AFDE values in relation to test suite sizes, it can be noted that π‘-wise relevant coverage has the worst efficiency as it requires 15023.70 test inputs for an AFDE value of 0.62. In contrast, π‘-wise valid coverage only requires 48.30 test inputs for an AFDE value of 0.62.
The best efficiency is offered by the combination of π‘-wise valid coverage with π‘ = 1 and π-wise strong invalid coverage with π = 0 which requires 308.00 test inputs for an AFDE value of 1.00. When using an AFDE value of 0.92 as a lower boundary (12 out of 13 faults), π-wise strong invalid coverage with π = 0 is sufficient and only requires 301.00 test inputs for an AFDE value of 0.98.
This discussion about efficiency is, of course, influenced by the characteristics of the 13 faults and cannot be generalized. But as more general statements, it can be observed that π‘-wise relevant coverage requires more test inputs to reach a similar AFDE value than π‘-wise valid coverage, π-wise strong invalid or the combination of both. At the same time, the combination of π‘-wise valid coverage and π-wise strong invalid coverage always has an AFDE value of 1.00 while at most 2224.30 test inputs are used. This finding is also consistent with our prior experimental evaluation (cf. [7]).
Therefore, we draw the conclusion that π‘-wise valid coverage, π-wise strong invalid coverage, and the combination of both perform as well as or better than π‘-wise relevant coverage in terms of effectiveness and efficiency. Although, the findings are only derived from one particular case. Therefore, we do not consider this to be true for all SUTs but for SUTs with many validation rules.
We compare the effectiveness of CRT using an implementation of the R O B U S T A test selection strategy with CT using an implementation of the I P O G -C test selection strategy. To ensure an unbiased implementation, both implementations follow the guidelines of Kleine & Simos [20]. Further on, the source code of the test selection strate-gies is published as part of the c o f f e e 4 j open-source test automation framework 1 .
The effectiveness of CRT and CT highly depend on the IPM and RIPM. Furthermore, the effectiveness depends on the faults that are considered in this case study.
Unfortunately, details of the case, i.e. source code of the validation rules and detailed descriptions of the faults, are confidential. To improve transparency and reproducibility, we describe the faults and make the characteristics of the IPM and RIPM explicit.
To avoid any bias, both the IPM and RIPM are modeled systematically and share the same set of parameters and values. To prevent falsified results due to accidental fault triggering, the orders of parameters and values are randomized and 20 different variants are used in test input selection. All presented FDE values are average values.
Since this is a case study with only one case, it is difficult to generalize the findings [10]. Further on, it has to be noted that the archival data of this case study is only a snapshot and the ground truth, i.e. the existing and previously existing faults, is unknown. Hence, the data can be biased towards simpler faults that are easier to detect. To prevent too far-reaching conclusions, we describe the characteristics of the SUT and also limit our conclusions to similar systems with many validation rules.
CRT extends CT to generate separate test suites with valid and strong invalid test inputs in order to avoid input masking that is caused by EH. Therefore, CRT requires additional effort to model error-constraints and introduces additional complexity to test selection strategies because error-constraints must be considered. This raises the question about the usefulness of CRT and whether the avoidance of input masking outweighs the additional effort and complexity. Until now, only artificial test scenarios are used to compare CT with CRT and it remains unclear if indicated advantages of CRT can be transferred to real-world scenarios.
In this paper, we therefore present the results of a case study based on a real-world system with 31 validation rules and 13 previously existing faults. To compare CT with CRT, we construct a IPM and a RIPM, select test inputs, and stimulate 13 implementations of the realworld system of which each implementation contains one of the 13 previously existing faults. For the subsequent discussion, we introduce the FDE and AFDE metrics.
To summarize the findings of this case study, we discuss both research questions individually.
Research Question 1: Our results indicate that the CRT test method is applicable in real-world test scenarios. This case study demonstrated that RIPMs with 32 parameters and 31 error-constraints can be constructed. Further on, the R O B U S T A test selection strategy is capable of selecting test suites for RIPMs with 32 parameters and 31 error-constraints.
Research Question 2: The comparison of CRT with CT is consistent with the findings of our previously conducted controlled experiment with artificial test scenarios (cf. [7]). Since the case under analysis has much EH, CRT performs better than CT in terms of FDE. Further on, it requires fewer test inputs to achieve better AFDE values than CT.
Therefore, we draw the conclusion that π‘-wise valid coverage, π-wise strong invalid coverage, and the combination of both perform as well as or better than π‘-wise relevant coverage in terms of effectiveness and efficiency.
Although, the FDE and AFDE values are influenced by the characteristics of the 13 faults and cannot be generalized. Therefore, we do not consider this to be true for all SUTs but for SUTs with much EH.
In future work, we plan to conduct further case studies to learn more about the FDE of CRT and CT.