System test cases allow to verify the functionality of a software system. System testing is a basic technique to guarantee quality of software systems. This work describes, analyzes and compares five proposals to generate test cases from functional requirements in a systematic way. Test cases generated will verify the adequate implementation of those functional requirements. The objective of this analysis is to determine the grade of mature of those proposals, evaluating if they can be applied in real projects and identifying which aspects needs to me improved.
System testing phase begins when the building of the software system is finished. The objectives of system testing phase are to test the system in depth and verify its global functionality and integrity, running the system in an environment as similar as the final production environment. This verificatio n is based on observation of a controlled set of executions called testing cases. Test process can be expressed like a searching problem. Its main objective is to discover and to correct most bugs as soon as possible [Binder1999].
Nowadays, use cases are the most used tool to express functional requirements. Use cases are also a good artefact to generate system tests cases [Fröhlich2000]. Actually, there are several proposals that describe how to generate test cases from use cases. This work resumes a comparative analysis among five proposals.
This comparative analysis started in March of 2004 and it is still active. Today this comparative includes 12 proposals. The objective of this comparative is to evaluate in depth the state of the art in system test cas e generation and to build the basis of a new proposal.
Some results from this comparative analysis have been included in [Gutierrez2004]. This work includes a different set of proposals and more comparative characteristics. Conclusions in this work are more precise due this work includes more proposals.
section section three analyzed proposals are introduced. In the state of the art is described. section 4 results from the analysis a r e s h o w e d .
Briefly, process of system test generation from functional requirements consists of build d a t a , a system model from functional requirements.
From that system model, input e v e n t s a n d e x p e c t e d r e s u l t s a r e g e n e r a t e d [Jacobs2004].
T h i s p r o c e s s i s described in
SCENT [Ryser2003] is a methodological proposal divided in two blocks. In first block, SCENT describes a process to define use scenarios. In second block, SCENT describes how to systematically generate system-testing cases from scenarios obtained in block one. Generation of test cases accomplishes intervening a three steps process. In first step, each test case is defined, indicating what it goes to test. After that, test cases are generated from the distinct paths that can be gone over in the state diagram. Finally, test cases obtained are refined and completed with more test cases developed by classical methods, like stress tests, user interface tests, etc.
Test Cases from Use Cases (TCUC) [Heumann2002] develops a method to obtain a set of system test cases from use cases in three steps. First, all possible path of execution are generated from every use case. Every possible execution path generates a test case. Finally, test values for every test case are identified. Test values include valid and invalid values and outputs expected.
AGEDIS [Hartman2004] is an investigation project financed by the European Union concluded at the beginning of 2,004. AGEDIS main objective has been the development of a set of tools for the automatic generation and exe cution of tests to verify systems based in distributed components. Although AGEDIS can be applied to any kind of system, better results are obtained applying AGEDIS to control systems, as communication protocols, than to information transformation systems, like compilers. AGEDIS focuses in two products: A system model written in a modelling language called IF, and a set of UML class and state diagrams. These products allow automatic generation of sets of tests and groups of test case objects to link system model and its implementation. This one allows executing tests with system model and system implementation and comparing outputs from model with outputs from implementation.
Use Case Path Analysis (UCPA) [Ahlowalia2002] describes a process composed by 5 steps. The starting point is a textual description of a use case. A flow chart is built from the use case. Using path analysis process a set of test cases is generated.
Requirements by Contract (RBC) [Nebut2003] is divided in two blocks. In first block, this proposal shows how to extend UML use case diagrams adding pre conditions, post-conditions and parameters. In second block this proposal describes several algorithms to generate test cases from extended use case diagrams. At the end of this process, a set of test objectives is obtained. A test objective is a sequence of instantiated use cases. A test case generator has to be used in order to produce concrete test cases from those test objectives.
Eleven factors were evaluated for each proposal. Table 1 shows the most relevant factors and next paragraphs describe those factors.
New notation indicates if a proposal proposes its own notation or diagrams. SCENT introduces a proprietary usage diagram and UCPA a proprietary notation for flow diagram. RBC introduces a proprietary notation to extend use case diagrams . AGESIS uses IF Language to model the system. TCUC uses natural language only .
Full systematized indicates if a proposal describe how to perform all steps indicates. If proposal is not fully systema tized some steps, like build system model form requirements, are not systematically detailed. Only AGEDIS if fully systematized and has a complete set of tools to perform the whole generation process.
Practical cases indicate if there are real project reports in which a proposal has been applied. Only SCENT and AGEDIS includes references to real projects.
Automated level measures the grade in which that proposal can be implanted in software tool. AGEDIS is the only proposal with full level.
Use of standards indicates if a proposal is based in diagrams largely used like UML diagrams. Only AGEDIS and RBC use UML diagrams.
Supporting tools indicate if, nowadays, there are tools to support the proposal.
Difficulty of implantation is based in quantity and diffic ulty of transformations to realize in each step. A low difficulty indicates a simple proposal to realize, without specific preparation. A medium difficulty indicates that there is new notation or some process that needs a previous preparation. A high difficulty indicates that a proposal cannot be applied without a depth study of its elements.
Application examples indicate if a proposal includes examples, aside from practical cases. All proposals, except RBT, include a practical case.
Coverage criterion indicates the method to generate test cases from system model. Several means that proposal exposes different coverage criterions.
Test values indicate if proposal exposes how to select test values for test cases generated.
Test case optimization indicates which proposal describes how to select some of test cases generated without losing of quality or coverage.
Multiple use cases indicate if proposal can generate test cases that involve more than one use case or proposal can only generate test cases from one use case in isolation.
Test case order indicates if proposal describes the order to execute test cases generated.
This section describes briefly main strong and weak point of each proposal.
SCENT offers a detailed method to manipulate and organize use scenarios. It includes two references to real projects where it has been successfully applied. However, it is necessary to make a very drawn-out job, 16 steps, with scenarios before generating test cases.
TCUC works with use cases written in natural language, instead of formal use cases . This makes suitable to rapidly obtain test cases, but difficult the automated of process by tools.
UCPA proposes a technique to determine which execution paths are most frequent and critical and which execution path are useless for testing purposes. This technique allows decreasing number of test without affect quality. Weak points are: flow chart notation is simple and hard to apply over complex requirements, and UCPA does not detail how to build test cases once identifies execution paths.
AGEDIS is the most complete proposal. AGEDIS includes generation and execution of test cases. This proposal provides references to five real successful projects. It has a complete tool kit to support all steps of the process. However, AGEDIS cannot be applied to all kind of projects, just only projects that flow controls is more important that information transforming. Tool kit is free for educational purposes only. AGEDIS obligates to adapt to AGEDIS tools.
RBC is supported by a prototype open-source tool called UCTSystem [Generating2003]. RBC generates test cases that include several use cases and describe several coverage criterions to concrete types of system. However, RBC extends use cases ignoring extension rule s proposed by UML Consortium. RBC treats use case as black boxes. This means that this proposal cannot generate test cases to verify the whole set of interactions in a use case in isolation.
Comparative analysis shows that AGEDIS is the most modern and complete proposal but, due its weak points, it is not the definitive solution. None of proposals are definitive; all proposals have some advantage over the rest and some weak points. There are main elements which are not described with enough detail to be applied in practice. Examples are: coverage criterions, or how to use storage requirement to derive test values. None of analyzed proposals describe clearly the result of the generation process. None of analyzed proposals describe de grade of detail of results. All proposals generate test but: how test cases are expre ssed? can they be directly implemented?. Examples included into proposals describe test cases as tables, but none of proposals defines clearly how to express a test case. Except AGEDIS, none of proposals defines detail level of generated test cases. They also do not generate test cases, only test descriptions that must be refined an implemented.
Actually we are working in answer these questions. Our goal is not to define a new proposal, but takes all elements in exiting proposals and complete their lacks, like several coverage criterions, a formal specification to a test cases, how to generate test code from that specification, etc.