Features of using the regression test selection method for automated testing of the graphical user interface in the development of information systems that consist of a set of modules are considered. The source of the need to create additional test environments required in the development of multi-module information systems that are using databases is specified. The three most popular approaches to organizing test environments - Copying, Scaling, and Scaling with synthetic data generation - are considered. The positive and negative sides are considered in terms of implementation, using, and resources spent on creating and maintaining resources, as well as in terms of the reliability of the results obtained in the process of testing models created using these approaches. The positive aspects of checking the quality of complex multi-module information systems from the point of view of the graphical user interface by various testing methods and, in particular, in the process of performing regression testing are presented. The positive aspects of using regression testing automation in conditions of lack of resources using various software platforms are indicated. The advantages of using the dynamic selection method for regression tests for automated testing are also given, as well as recommendations for implementing the selection method in existing and beginning projects.
The process of developing information systems (IS) [
Of course, each of these stages is important and cannot be excluded from the IS
development process. Depending on the chosen model of IS development, these
stages can follow strictly one after another or be performed cyclically [
The testing stage, in turn, is also divided into several stages. Usually, a test
strategy that plans the types, categories, and order of testing is developed for each
project separately, but there is a list of tests that are mandatory for any of this – unit
testing, integration testing, smoke testing, release testing, system testing, regression
testing, interaction testing [
Regression testing is a special type of IS quality control aimed to detecting errors in areas and modules that have not been modified specifically but may have lost their correct functioning due to changes made to adjacent areas or modules. The complexity of conducting a correct RT procedure is caused by the need to understand the IS architecture when selecting tests for RT, which is especially difficult in the case of developing a multi-module is with possible interaction of modules not directly, but indirectly. In addition, it is necessary to have a good knowledge of various methods for selecting scenarios for RT, which may not be available in the context of saving resources for IS development and insufficient qualification of testing engineers.
The frequency of RT execution depends on the specifics of the developed IP, the
selected IS development process [
One of the simplest and most effective ways to check the quality of IS which are
developed using the most popular process continuous integration/continuous delivery
(CI / CD) [
In simple monolith ("solid") IS [
For small sites on the Internet [
For both types of described IS, the task of organizing an isolated environment [
Multicomponent IS are becoming increasingly popular for various reasons [
Coping
This approach is only applicable for small multi-module IS with a small incoming data flow.
2. Using a scaled-down industrial instance model-using fewer resources, copying a certain percentage of data from the industrial instance database (Fig. 2).
This approach allows you to save the resources which are necessary for the operation of a test instance of the IS, while maintaining the ability to interact with modules of different levels. At the same time partial copying of data from an industrial database allows you to perform testing on data with a high degree of similarity to real data.
3. Using a scaled-down industrial instance model with generating synthetic data
in the database and using the necessary modules (Fig. 3).
This approach allows you to drastically reduce the resources necessary for
deploying the test infrastructure, isolate one or more connected modules, and generate
a certain range of data, which allows you to check the correctness of the system in
simulated situations (in situations that occur extremely rarely in industrial operation,
as well as in emergency situations). The application of this approach is significantly
simplified when using automatic control systems for containerized applications, such
as Kubernetes [
Using the most common and simple methods for selecting regression tests (selection by priority, functionality, result of previous execution, date of execution) leads to the formation of a fairly stable list of tests. In the process of developing an IS with a regular CI/CD functional increment, regression testing is performed regularly, meaning that testing engineers have to regularly perform the same scenarios, which can lead to a decrease in concentration when performing such tests and as a result a decrease in the effectiveness of searching for errors in the IC. To avoid such situations, as well as to make more efficient use of testing engineer’s resources, modern regression testing projects use automation for executing test scenarios [19].
Once an automated test can be run an unlimited number of times in the future, almost without the participation of an engineer. Tests can be developed and run using, for example, TestComplete, Selenium, or Robot framework environments. The automated launch of RT can be combined with the release of a new version of the IS using, for example, the software system CI / CD Jenkins [20].
In the case of a long process of IS development, the number of automated tests and regression tests can reach large amounts and, consequently, the execution of all automated tests can take a very long time. If the IS is a small application, site, or mobile device app, you can solve the problem of running automated tests for a long time by running these tests on multiple devices in parallel.
Running a significant number of automated processes for complex multi-module is with a database can take a very long time, while running several processes in parallel may not be possible due to the presence of only one test infrastructure, which can also be used for manual testing during the working day. It is possible to partially solve this problem by dividing all automated tests into groups and running tests in groups according to a certain schedule (Fig. 4).
Such approach allows you to implement the execution of all automated RT but requires significant costs for the correct formation of the list of RT and the schedule for running these tests. 4
In projects of the development of multicomponent IS with the regular release of new versions with high functional growth, the issue of prioritizing, selecting and minimizing the list of RT is very acute not only because of insufficient time to conduct all the necessary checks, but also because of the high employment of the test infrastructure. It is necessary to consider that the practice of CI/CD can lead to results new version of is almost every day, which entails the need to conduct all possible checks in the shortest terms and, therefore, to spend time on manual selection of automated RT is very desirable for the reason that there are a significant number of methods of selection of RT, which are based on different principles of determining whether to run the test and correct their application often requires a large amount of resources. In addition, when testing multicomponent IS, it is highly desirable to consider the possible mutual impact of changes in modules and, more importantly, the impact of changes on the UI. The method of selecting and ranking test scenarios considering an arbitrary number of selection methods allows you to automate the selection of RT.
Let us assume that the selection of tests for RT is made using a certain number of selection methods. For each of these methods, you can determine the significance of this selection method in terms of the importance of running the test and whether the test can detect an error. For convenience of calculations, you can use the value of the test significance in the range from 0 to 100.
Thus, the methodology for calculating the significance of the test for an arbitrary number of RT selection methods can be defined as follows:
Ia=∑〖(I1+I2+⋯+In)〗, (1) where Iа – the total value of the test; I1 – significance of the test selected by method 1; I2 – significance of the test selected by method 2; In – significance of the test selected by method n.
Using (1), we get the significance of the test in terms of the possibility of detecting errors in the IS – the higher the received significance of the test, the higher the priority of the test execution.
Separately, it should be noted that (1) is correct for the significance of the test (ST) for each selection method in a certain General range, for example, from 0 to 1 or from 0 to 100. If it is impossible to fulfill this condition for various reasons, it is suggested to use additional normalization by weight coefficient (WK).
Ia= 1×I1+ 2×I2+⋯+ ×I , 1+ 2+⋯+ (2) where Iа – total ST; w1 – WK for test group 1; I1 – ST, selected by method 1; w2 – WK for test group 2; I2 – ST, selected by method 2; wn – WK for test group n; In – ST, selected by method n;
Thus, using (2) it is possible to get a queue of texts selected using an arbitrary number of selection methods, which allows us to take into account the potential mutual influence of IS modules, ranked by the priority of test execution, taking into account the use of the UI. Having data in the test about the IS modules that are necessary for the correct execution of this test will allow you to limit the use of test infrastructure resources only to those modules that are necessary.
In the future, the resulting list of automated tasks can be entered into the configuration file of the software system for ensuring the continuous integration process and will be performed in a period of time when the test infrastructure is not involved in higher-priority tasks. 5
To implement the described method of RT selection, a software package was developed that allows working with various testing management systems via the API. Development was carried out in the object-oriented programming language JAVA. The UI of the software package provides the possibility to use an arbitrary number of test selection methods for RT, specify the WK for each of the selection methods, and an area for displaying the test pool, ranked in descending order of ST (Fig. 5).
This software package was used to create a pool of manual regression tests for two different IS development projects on a daily basis, which made it possible to quickly assess the potential IS regression in the conditions of daily updating of the system modules.
The effectiveness of this method was evaluated by comparing the RT results (the number of errors found), which were formed by manual selection of test scenarios and using a software package for IS versions similar in number and volume of functional growth, for the same RT period (10 working days), on an identical number of tests (103 tests).
For a multi-module IS that evaluates the quality of the provided broadband data and TV content service (the number of modules is more than 70) with a high functional increase (over the period of 10 days, changes were made to the code of 22 modules), the number of detected errors detected by RT using an automated selection method increased by 11%.
For a modular IS that controls the car's multimedia information system (the number of modules is 8) with an average functional increase (over a period of 10 days, changes were made to the code of 3 modules), the number of detected errors detected by RT using an automated selection method increased by 6% (Fig. 6).
At the same time, the time spent on forming the RT pool was reduced by 2 times when using the automated selection method – from three hours to one and a half hours (in total – when forming the RT pool daily). 6
This study has shown the possibility of applying the dynamic selection method of regression tests for automated testing of graphical interfaces of multi-module information systems. Integration of this methodology into existing IS development projects provides the following advantages: 1 Allows to significantly reduce the time spent on the selection of automated RT; 2 Allows to perform automated RT on a priority basis, the results of which can be concluded with a high degree of probability that there are errors in the most priority IS modules from the point of view of the UI; 3 Allows to take into account the potential mutual influence of different modules in a multi-module IS when performing automated RT; 4 Allows to save the use of test infrastructure resources by prioritizing the implementation of automated test scenarios.
Axelrod, A.: Complete Guide to Test Automation: Techniques, Practices, and Patterns for Building and Maintaining Effective Software Projects. Apress (2018).
Laster, B.: Jenkins 2: Up and Running: Evolve Your Deployment Pipeline for Next Generation Automation. O'Reilly Media (2018).