The active use of test automation approaches in software development causes further automation of this process, in particular, in the field of results analysis. This paper presents the results of applying the methodology for creating information technologies of multi-level intelligent monitoring to provide data for decision-making processes by a testing engineer. Methods of text mining and machine learning are combined to build a methodology for creating intelligent multi-level monitoring systems. On the example of a set of reports, which were divided by an expert into 4 classes, the processes of coordination of interactions of different types of methods for forming an array of informative features, typical aggregates for the synthesis of classifier models at each stage of monitoring were investigated. The task of classifying the results of text research was solved to obtain the reason for the failure of the tests. A set of n-grams for each class of tests formed an array of input data for the synthesizer of models of the monitoring information system. The classification of these data was provided using an ensemble of models, where the resulting value was obtained combining predictions by meta-learning using stacking. The effectiveness of the described methodology has been experimentally proven.
Intelligent monitoring of automatic software
testing processes provides identification of the
causes of errors to speed up their processing by
the tester. Based on the testing results, the
monitoring information system (MIS) [
The introduction of test automation into the software development process is now a must, the number of available test results is increasing significantly.
Jobs, or agents who have the role of a QA
engineer and perform tests, can work around the
clock, seven days a week, and in addition, the
number of test cases accumulates during each
sprint. Thus, more results are obtained for
management and analysis. A significant amount
of data requires approaches to their processing
[
If the time spent examining the test results
exceeds the time saved by running automated
tests, then the very meaning of its
implementation is lost. To reap the benefits of
automation, it is important to know how to
properly handle the growing number of test
results. Continuous Delivery requires constant
testing that slows down the pace of work. In the
fast pace of product development and testing,
optimizing (both failed and successful) test cases
is critical [
At the same time, there are not so many
applications for tracking the history of
automated tests, they are limited in functionality.
And standard reports are always hard to read
since the appearance is not informative enough.
The most common test frameworks are NUnit
and TestNG, the latter of which was used in [
Machine learning of classifier models is
based on the analysis of previous runs of
autotests. The main stages are [
1) coordination of the form and content of information about the properties of objects of observation; 2) determination of the list of diagnostic signs; 3) the formation of an array of input data (MVD); 4) construction and training of models by the MIS synthesizer;
5) testing and using models to identify the causes of errors. The set of the obtained classifier models is entered into the model knowledge base, forming its hierarchical structure.
The task of ensuring the information content
of the Ministry of Internal Affairs, sufficient for
constructing useful models by the available
methods and means, implemented in the
synthesizer of the MIS models, is always
relevant. The effective organization of
observation of research objects provides for the
attraction of the latest scientific achievements in
the subject area, in which the monitoring
technology is implemented [
Now there are not many implemented
readymade products for classifying the results of
automated testing. Most of them are test
developments and student projects [
Thus, the use of machine learning technology of models through MIS for analyzing the results of Automation tests is relevant and allows you to increase the efficiency of QA engineers, allows you to increase the coverage of web applications with new tests, and improves the quality of applications. It also reduces the time it takes to make decisions to improve on failed tests.
A hypothesis was formulated that the list of
classification features for identifying the causes
of errors in programs should be obtained by
using Text Mining methods [
Thus, at the first stage of building a monitoring information system, it is necessary to determine the format of the input and output data.
Input data for the system - are error messages, and their example was shown on Figure 1.
The initial data will be a table with the results of automatic testing, the application of machine learning algorithms, and detailed analysis will be carried out directly by the testing engineer. They establish the reason for the fall of the test. That is, the input data was analyzed by an expert, who allocated 4 classes of system responses to the results of automation testing. However, if necessary, this list can be expanded, that was shown on Figure 2.
For the analysis of the reports, the text Mining approaches were used. Tokenization was carried out (with the selection of the optimal length and mixture of n-grams. In addition, from the preprocessing approaches, the text cleaning was used. The main purpose was to clean the text from specific characters (primarily specific punctuation and symbols). Also, the most frequently and rarely used words were removed, do not allow define the specifics of the text and classify it.
Algorithms for the synthesis of classifier models
were constructed using artificial intelligence and
machine learning [
In particular, the CountVectorizer provided by
the scikit-learn library [
After that, you can see the resulting function vectors for each sentence based on the previous dictionary, that is, Bag-of-words. Each document is represented as an n-gram vector. You can then use these vectors as function vectors for the machine learning model.
Thus, the list of informative signs contains a mix of n-gram and the reason of the route cause, determined by the expert. The resulting value for the entire test case was determined based on the Bayes' formula: ( ) = ∑
Thus, at the first level of the MIS, a random forest classifier with optimal parameters is created by creating samples based on the bootstrap from the verified elements. Another first-level model was decision tree “stumps”, which were used as a baseline model for running with optimal parameters. The generalization of the results obtained at the previous levels was due to the logistic regression model using the Softmax function.
The idea behind the MIS is to automate the process of analyzing test reports by an engineer and provide support for making the final decision. To do this, you will need to use two roles: the admin side and the engineer side. The administrator has access to all points and the ability to edit root causes, create and train the model. The engineer has access to all reports, can view and analyze tests. All the functionality that is on the Report list page, open to the engineer. The functional diagram of the algorithm is shown on Figure 4. will be with the most likely critical root causes decision rests with the engineer.
Each test has its name and status, which are displayed on Figure 5. If the test has not yet been analyzed using the classification algorithm, the “Prediction AI” column will be empty. To view more detailed information about the test, you must click on it.
For unanalyzed tests, predictions will be displayed in a special column. When analyzing, an engineer will need to first of all pay attention to those tests that will be with the most likely critical root causes (Defect, Framework issue, etc.). The final decision rests with the engineer.
Thus, the engineer pays attention, first, to those tests that have critical root causes and, in more detail, examines the causes of their occurrence and enters the results into the database. These detailed reports will also be the basis for training the model on subsequent launches. The results of auto-analysis and engineer's analysis can be seen as a result in the table with tests. An example of these results is on Figure 6.
The developed system was tested based on reports of automatic testing of the website development process. The model test results are demonstrated using an error matrix and are shown in Figure 7. It is worth noting that the result c6an be influenced by different class sizes within the studied array of input data.
From the 62 studied reports, the reasons were correctly identified in 55, that is, the number of correctly identified causes of errors was 89%.
The use of MIS to provide information to decision-making processes in the field of automation testing allows you to successfully solve the problems of classifying the analysis of the input standard reports of automated tests based on the data of the expert assessment of automation engineers.
The developed system can have a wide practical use for solving problems in the process of developing and testing software products. To quickly identify critical root causes of test failures, chronological analysis and help the engineer make decisions. The introduction of this system can lead to an increase in the ability of engineers to cover web applications with new tests and will lead to a higher quality of applications. It will also reduce the time it takes to make decisions about failed tests. The obtained accuracy allows us to assert that the proposed approach for the formation of the Ministry of Internal Affairs makes it possible to form it sufficiently informative due to its application in conjunction with the proposed architecture of the ensemble.
The effectiveness of using the proposed architecture of the ensemble classifier for the synthesis of models-classifiers based on an array has been experimentally confirmed.