This paper is devoted to technology of automating full cycle of software development from natural language processed formalized requirements and their analysis with symbolic verification to tests generation and execution. Suggested technology is capable of checking semantic consistency of requirements in generated code of target application. Primary target areas of technology applicability are telecommunication and distributed systems, but other types of applications can also be developed using the technology.
One of the main problems in the development and test automation of industrial applications’ software is handling of complicated and large scale requirements specifications. Documents specifying requirements specifications are generally written in natural language and may contain hundreds and thousands of requirements. Thereby the task of requirements formalization to describe behavioral scenarios used for development of automatic tests or manual test procedures is characterized as a task of large complexity and laboriousness.
Agile methodologies are used more and more often by the software developers.
There are different ones: Scrum[
In this paper, an approach is proposed for creating an effective verification and
automated testing technology, which should be used all along the development process.
Tests are a part of a project, just as the system code is, and recommended by every
agile methodology listed above. Usually the mandatory checks are run on unit tests,
which are verifying small parts of the code before integration into main development
branch, and functional tests, which are verifying typical scenarios and use cases from
the user point-of-view [
This is a common situation, when software developers just do not have enough time or resources to properly test the system in non-standard cases, estimate the dangers of “copy-paste” from one module to another and other types of behavior where checking could take a lot of time and requires a thoughtful code analysis of possible negative outcome.
Because in agile methodologies the development of design specification is omitted, the requirements will take the brunt and thus must be created as good as possible. Thereby the task of requirements formalization to describe behavioral scenarios used for development of automatic tests or manual test procedures is characterized as a task of large complexity and laboriousness.
Applicability of formal methods in the industry is determined to a great extent by
how adequate is the formalization language to accepted engineering practice which
involves not only code developers and testers but also customers, project managers,
marketers and other specialists. It is clear that no logic language is suitable for
adequate formalization of requirements which would keep the semantics of the
application under development and at the same time would satisfy all concerned people [
In modern project documentation the formulation of initial requirements is presented either in constructive way, when checking procedure or scenario of requirement fulfillment can be constructed from the text of this requirement in natural language, or unconstructively, when functionality described in the requirement does not contain any explanation of its checking method.
For example, behavioral requirements of telecommunication applications in case of described scenario of requirements coverage are constructively specified and allow using of verification and testing to check requirements implementation in software. Non-behavioral requirements are usually unconstructively specified and require additional information during formalization which allows reconstructing the checking scenario, i.e. converting of non-constructive format of requirements specification into constructive one.
The procedure of requirement checking is exact sequence of causes and results of some activities (coded with actions, signals, states), which analysis may prove that current requirement is either covered or not. Such checking procedure can be used as a criterion of coverage of specific requirement, i.e. it can become a so-called criteria procedure. In the text below a sequence or “chain” of events will be used for criteria procedure.
Tracking the facts of criteria procedure coverage in system’s behavioral scenario (hypothetical, implemented in the model or real system), it can be asserted that the corresponding requirement is satisfied in the system under analysis.
Procedure of requirement checking (chain) is formulated by providing the
following information for all chain elements (events):
• conditions (causes), required for activation of some activity;
• the activity itself, which shall be executed under current conditions;
• consequences – observable (measurable) results of activity execution.
Causes and results are described with signals, messages or transactions, commonly
used in reactive system’s instances communications [
Problems with unconstructive formulating of requirements are resolved by development of requirement coverage checking procedures on user interfaces or intercomponent interfaces.
Thus, chains containing sequences of events can appear as criteria of requirements coverage; in addition, it is possible that criterion of some requirement coverage is specified not with one, but with several chains. 3
Usually initial requirements in technical documentation are formulated in natural language and can be presented in one of the following form: • in form of behavioral requirement, scenario (procedure) of requirement checking can be retrieved based on requirement text; • in form of non behavior requirement, only structure and sense of requirement can be retrieved without information about requirement checking.
Any behavioral requirement can be formalized with further automatic analysis; for
this purposes VRS/TAT technology [
In VRS/TAT technology Use Case Maps (UCM) notation [
UCM model (Fig.1) contains two interacting instances model description. Each path on the graph from the event “start” to the event “end” represents one behavioral scenario. Each path contains specified number of events (Responsibilities). Events on the diagram are marked with × symbol, while Stub elements which encode inserted diagram – with ♦ symbol. As a result, each scenario contains specified sequence of events. Varieties of possible scenarios are specified with variety of such sequences.
In these terms a chain is defined as subsequence of events which are enough to make a conclusion that the requirement is satisfied. A path on the UCM diagram, containing the sequence of events of some chain, is called trace, covering the corresponding requirement. Based on the trace the tests can be generated which are needed for experimental evidence of the requirement coverage. 4
Verification project requirements formalization starts with creation of Traceability
matrix (TRM) [
The distinctive feature of VRS/TAT technology – special criteria of each requirement’s coverage checking. Below all criteria related to requirements are listed in ascending order of their strength: 1. Events criterion - coverage level in generated scenarios of subset of events used in criteria chains. 2. Chains criterion - coverage level in generated scenarios of subset of chains (consisting of events and states of variables) with at least one for each requirement. 3. Complex criterion - coverage level in generated scenarios of the whole set of chains specifying integral criteria (combined from criteria 1 and 2) of requirements coverage.
Criteria development shall be adaptive to specific project [
Model-checking based technique is used to generate symbolic and concrete traces
(STG and CTG) [
There are two ways of guides creation: automatic and manual. Pros and cons of both
pocesses are described in [
The process of tests generation with usage of VRS/TAT tools can be divided into following phases:
Manual creation of formal model for requirements in UCM language. Key actions
are refined from initial requirements. Based on these actions a set of chains, which
will be translated into behavioral scenarios are created. On the next step objects which
interact in the scope of scenarios and structure are refined. Metadata for data flow are
added into formal model. More detailed description of the phase is presented in [
The next phase is conversion of UCM model into BP notation [
Guides which satisfy branch criterion are also generated based on initial UCM model. In process of generation all branches are traversed with VRS/TAT toolset.
On the next phase formal model in basic protocols notation and guides are used in trace generator of VRS tool for verification and automatic traversal of behavioral tree for symbolic scenarios generation.
After scenarios generation process finished it is necessary to analyze coverage of guides by traces with goal to define which guides are still not covered and identify the reasons. Such analysis can be performed in EVA module. Based on analysis results UCM model or guides shall be corrected. The amount of generated test scenarios may be very big and thus running the whole toolchain may require a lot of time, which is not desirable and can significantly slow down all development process.
The most obvious way to speed the things up is to run tools in parallel. To do
this, each of the processing steps was divided on separate parts:
• Parallel traces generation became possible thanks to guides [
Concrete traces are processed in different threads, as they are independent [
Theoretically, independent process parallel launching will give linear scalability. However, there are still limits on executing on CPU/server, because machine has limited number of cores/threads. Therefore, you will only have some work running, while the rest of it will wait for its turn. Of course, you can improve the performance of the system by adding another set of computing nodes. By bringing more and more computing power, we are getting closer to the term of high performance computing (HPC).
The use of supercomputer can help to extend the number of testing instances to the point where we can run all tests from suite simultaneously (or at least a lot of them).
Test suite was divided into several parts (by the number of nodes), which were processed individually on each node, which includes launching testing toolset and running the SUT. 6
Project contained stats: 2500 basic protocols, 42000 symbolic traces, 160000 concrete
traces and tests was released by S.Petersburg Polytechnic University’s Supercomputer
[
The result of this work is improved technology which integrates verification and testing of software projects and provides: • Full automation of industrial software product development process with the control of requirements semantics realization. • Generation of application’s model and symbolic behavioral scenarios, which fully (100%) cover behavioral features of the application. • Automated concretization of symbolic traces in accordance with test plan. • Automated execution of system and regression testing phases. • Supercomputer resolves problem of testing performance and we can run all tests from suite simultaneously • High level of automation for the process of development and managing of software product quality.
Results of integrated design and testing technology appliance in the development of wireless telecommunication applications demonstrated 26% time-saving in software product creation.
Acknowledgments. The work was financially supported by the Ministry of Education and Science of the Russian Federation in the framework of the Federal Targeted Programme for Research and Development in Priority Areas of Advancement of the Russian Scientific and Technological Complex for 2014-2020 (№ 14.584.21.0022, ID RFMEFI58417X0022).