The article considers the problem of test automation software. The authors analyze testing tasks that can be automated. They also cite cases where the use of automation is inappropriate. The key factors of using automation are time and cost savings. According to the authors, the advantages of automated tests are: the ability to check the latest changes in the application (regression testing), speed of execution, saving the time of testers, the ability to create selftests by developers. The disadvantages of automatic tests are: insufficient reliability, need for support, fewer errors detected, a false sense of product quality. The following processes are identified, which can be automated: background processes, file logging, database entry, registration and payment systems, load tests, data entry operations, long-end-to-end scripts, checking complex mathematical calculations, checking correct search. The article provides statistics on the use of programming languages for developing automated tests. A comparative analysis of ready-made software products for automated testing is offered. Based on research analysis and experience, the authors believe that human intelligence is always required to validate the program. So, the authors justify the need to perform a manual and automated test.
Testing is a very important stage in software development life cycle (SDLC). This
process ensures that most errors are found. However, one of the principles of testing is
that all errors cannot be found – exhaustive testing is impossible [
As the research and practical experience shows, the software development lifecycle
takes place during the following stages – analysis, design, development,
implementation, testing, deployment, and support (Fig. 1) [
As the speed of software development is increasing today, there is a need for quality
and timely testing. But some of the testing tasks are too time-consuming to perform
their manually. In addition, more and more companies are moving to work according
___________________
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
to agile methodology. An agile process begins with initiating the project; in the
activities that follow, the software will be developed and deployed into the user
environment through multiple iterations. In most agile methodologies, maintenance
does not appear as a separate phase, but is rather performed through further iterations
of the main development phases [
In article [
Today the situation has changed. Almost every company wants to have a automated tester, no matter if it's a big company or a small one. Today, automation is not only relevant in startups. As a result, writing self-tests is usually not time consuming. In general, startups may not even have test cases. The growing numbers of communities, forums, channels where automated testers discuss their problems, can also testify to the great development of automation. 2
Automated testing is now evolving rapidly. This is due to the fact that automated,
debugged processes will require less money, which is one of the key factors in IT. As
experience in the early days of the introduction of automation shows, it requires more
material costs, while fewer tests are performed instead. However, over time, it is
possible to significantly increase the number of test runs without significant investment.
The popularity of automated testing is also explained by the fact that more and more
“manual” testers want to develop in programming. Automation is a next stage of their
development. “The job was pretty boring. We were just supposed to look at flagged
differences and decide if they were a big deal or not. I wasn’t happy with that. I wanted
to know what caused the differences, so I started digging a little deeper", – John Sonmez
talks about it in the article “Going from QA (or Another Technical Role) to Software
Developer” [
Systematic and automated approaches have shown capable of reducing the
overwhelming cost of engineering automated tests. Industrial success cases have been
openly reported and academic interest continues to grow as observed by the increasing
number of researchers in the field. While there exist various trends on evolving the
automation in software testing, the provision of sound empirical evidence is still needed
on such area [
The authors of [
The purpose of the article. The dynamic pace of development of automated testing requires the answer to the question “can we all automate?” What is the role of manual testers in modern software development processes?
Software testing can be performed for different purposes. According to the purpose
there are such types of testing [
All these types of tests can be performed both manual and automated.
Researchers identify such phases of the software testing process [
Automated testing or testing automation is a method of testing software. The method involves the use of special software tools to control the performance of tests. Then the actual test results are compared with the predicted ones. Most operations are performed automatically, with little or no intervention by the test engineer. Automated testers write scripts (so-called automated test cases) that have a set of actions and checks. Properly written automated tests can have many benefits and can be very useful for the project and organization. However, there are some disadvantages of automated tests that you should also be aware of. 3.2
Regression testing. Automation testing is the most common for this type of testing.
Regression testing is a type of testing aimed at checking changes made to an application
or environment (debugging, code merging, migration to another operating system,
database, web server), to confirm that existing functionality is still working. Regression
can be both functional and non-functional tests. Typically, regression testing uses test
cases written in the early stages of development and testing. That is, regression
automated tests are performed at a predetermined time interval. They are usually
downloaded after every successful compilation (in small projects) or every night or
every week. This guarantees that the changes to the new version of the program do not
damage the already existing functionality. In the paper [
Speed of execution. Automated verification scripts may take some time. However, it takes less time to complete them than the person who would perform these checks manually. Therefore, self-tests help to provide quick feedback to the development team.
Time saving for testers. Test automation frees up testers' time. Therefore, they may be more focused on exploring new features. Automated checks can be started automatically with minimal supervision, or without any supervision, or manually. Usually when you do not want to use automated tests there is a cyclical situation of lack of time (Fig. 3). Ability to create automated tests by developers. Automated tests are usually written in the same language as the product being tested. As a result, the responsibility for writing, conducting and performing tests becomes a shared responsibility. Everyone in the development team, not just testers, can contribute to the quality of the software. 3.3
False sense of product quality. For this reason, it is worth paying special attention to successfully passed automated tests. This is especially important for user-level (UI) or system-level functionality testing. The automated test only checks what is programmed to be tested.
All automated tests in the test suite may pass, but some mistakes may be not identified. The reason for this is that this test was not designed to detect these specific failures.
Insufficient reliability. Automated checks may not be successful due to many factors. For example, automatic checks can be broken by changing the user interface, shutting down a service, or having a network problem. These problems do not directly affect the program being tested, but may affect the outcome of automated tests.
Need for support. It should be understood that automated tests require maintenance. Automated checks are short-lived. Failure to update them will cause crashes. It is also possible that some checks are no longer relevant or that they do not correspond to new implementations of the software. These failures can affect the test results.
Writing an automated test case is not a one-time effort. To get the most out of automated tests, they need to be updated and up-to-date. This usually takes time, effort and resources.
Fewer errors detected than manual testing. Most errors are detected “accidentally” or during exploratory testing. This type of testing involves the simultaneous study of a software product, the design of tests and their execution. Its specificity is that at each exploratory testing session there is an opportunity to test the application in different ways.
On the other hand, automated checks always follow the set path, sometimes with the same test dataset. One can mention here one of the principles of testing – the “paradox of the pesticide”. It is that by performing the same tests over and over again, we face the fact that they find fewer and fewer errors. This is due to the development of the system, as a result of many found defects are corrected and old test cases no longer work. This in turn reduces the likelihood of finding new defects in the product.
“Test automation is not always testing”. Here, it is understood that testing is a research activity. Testing requires specific knowledge, a purposeful mind, and a willingness to learn the application. Unfortunately, many people are wrong about the importance of test automation. They get a test automation tool and want to get rid of all the “manual testers”. But this is not possible because testing is not simply about performing a set of predefined testing steps and comparing actual results with expected results. The most recent tasks are automated checks.
Ability to group failures into clusters. In the paper [
A person’s intelligence is always required to validate the program. Here’s how Tommy Wyher says it: “Every day, we see hundreds of new apps and products out in the market. A lot of testing takes place before they are released to the public. Automated testing can speed up the process and is often seen as a replacement for manual testing. However, manual testing still has a critical role in the QA process. By dedicating your QA resources to only one of these approaches, you’ll miss many opportunities to improve quality. Using manual and automated testing together will lead to a higher quality, more stable product” [15]. 3.4
Let’s look at software components and processes that can be automated. 1. Hard to reach places in the system: background processes, file logging, database entry. 2. Frequently used functionality where there is a high risk of errors: payment systems, registrations, etc. Automation of critical functionality checks ensures fast errors, since the test takes an average of several minutes. 3. Load tests that test the functionality of a system with a large number of requests. 4. Template operations, including data searches, input of forms with many fields, checking their preservation. 5. Validation messages: fill in the fields with incorrect data and validation check. 6. Long end-to-end scenarios. For example, an online store scenario that involves: user registration, product detail page (PDP), shopping cart, shopping cart, product purchase, and confirmation of purchase. 7. Verification of data requiring accurate mathematical calculations, eg accounting or analytical processing. 8. Checking the correctness of the data search.
Although newer automated testing tools are emerging, it is still difficult to test the functionality of the user interface. It is also not possible to automate the testing of a new feature without at least one manual test run. Therefore, to say that you can automate everything, for now, will be an exaggeration.
Consider which tools are most often used to write automated tests.
Automated tests are written in various programming languages, the most common
being Java. Figure 4 shows the distribution of automated programming languages in
the world [16]:
In addition to programming languages, there are other automation tools available in
software testing. Their comparative analysis is given in the Table 1 [
Web applications
Windows, Linux
Python,
JavaScript, Ruby Ruby PHP, C#,
C++ No required. Advanced Recommended skills needed for advanced to integrate test script various tools Easy setup and Require use. Need advanced training to skills to properly user install and the tool use
Programming skills needed Recommended Skills to integrate for advanced various tools test script
Require inaEsntaadslleUaotsifeon iansdskvtiaalllnlscateondd Easy suesteup and use
Recommended for advanced test script Complex in installation.
Need training to properly user the tool
Each of these tools has its own peculiarities and scope. The most common tool for automation is Selenium. This is because it is free and scripts can be written in many programming languages, unlike other tools. Developed over 15 years ago, it has evolved over the next decade. It is now a web browser automation tool. In most cases, it is used to test Web applications, but this does not limit its scope.
JMeter is used for load testing. Initially, this application was intended to test the operation of the Apache Tomcat servlet container, which is essentially a web server. With the development of Jmeter, the user interface has been improved and additional features have been added that have made it an effective tool for performance testing and load testing of web applications.
It is impossible to say exactly what kind of tools will be popular in the future, as they are in rapid development. In our opinion, this will greatly simplify testing automation.
The authors of the article [
Faster feedback from tests execution allows to catch issues early on, saving huge amounts of costs. 2. Low-level tests are executed much earlier in the QA automation pipeline. Usually, unit tests are run before each commit test. If this is true, then testers prevent bugs and do not let them get into the project repository. 3. Low-level tests are much more stable than high-level ones. It’s need to remember that high-level tests should be only the third defense shield, for catching all the remaining issues that were not caught on the first two levels. 4
In the process of the research, we found out the importance of testing in the software development process. We found out what automation is and what role in testing. We analyzed tools for writing automated scripts, provided statistics on the popularity of programming languages to test automation across regions.
Test automation is evolving very quickly, new tools and technologies are emerging almost every week, but they are not yet fully replaceable, and everything cannot be automated. Without human intervention, it is impossible to construct the testing process, since testing requires intellectual intervention. In our opinion in the near future, there will be a trend of increasing number of automated testers and a gradual decrease in manual ones. It will become an obvious educational goal to become an expert in automation. After training in the practical use the testing tools, manual testers will be able to perform automated testing. Therefore, every tester who wants to be more in demand in the IT should learn at least one programming language. So learning programming and testing are processes that depend on each other.
It cannot be said that automated testing will completely replace manual testing. As a result, the conclusion is one: in the world of software testing, there will always be room for both manual and automatic testing. The choice of testing methods is left for every worker in this area.