zet Test, yazlm kalitesinin byk bir parasdr. Testin nemi gn getike artmaktadr. zellikle byk uygulamalar test etmek iin zaman gerekten nemlidir. Test mimarisinde, gml sistemleri test etmek iin takip edilecek birok teknik vardr. Regresyon testi bunlardan biridir. Regresyon testi, modiye edilmi‡ ve dzeltilmi‡ ortam yeniden test eden test tekni§i trdr. Bu teknikle, her bir yazlm versiyonuna yeni fonksiyonlar eklenir ve eklenen her fonksiyonu test etmek zaman gerektirir. Bu noktada, yeni eklenen fonksiyonlarn uygun zaman periyotlarnda mevcut fonksiyonlar bozmad§ndan emin olmak iin her versiyonda regresyon testinin uygulanmas nemlidir. Bu yazda regresyon testinin verimlili§ini artrmak iin yeni teknikler sunulmaktadr. Gml sistemlerin regresyon testi iin olan yakla‡mmz, zamandan tasarruf etmek ve daha fazla test yapmak iin hem test seim tekniklerini hem de zaman tasarrufu tekniklerini kullanmaktadr. Bu teknikler, test durumlarn benzer zelliklerine gre gruplandrmay ve test durumlarnda gerekle‡tirilen yaplandrmay geri almay ierir. Sonu olarak, nemli bir zaman tasarrufu sa§lanr. Bu sonu geli‡tirmeye ve dzeltmeye yksek fayda sa§lar.
Aects of Grouping Test Cases for Time Saving ?
Esentepe Mahallesi, Yakack Yolu No:111, 34870 Kartal/Istanbul, Turkey 2 Siemens Siemensallee 75, 76187 Karlsruhe, Almanya {engin.sacan; mujde.agra; aylin.kusku; rainer.heck; koestner.michael; daniel.schertler}@siemens.com
Software development lifecycle(SDLC) is the process which includes all phases of
software development from the beginning to end like planning, analysis, design,
production and maintenance [
In testing step of SDLC, defects and problems are found and testers inform
developers with details of the issue. If it is a valid defect which meets a set
of conrmed defect denitions by the development team, the developer will x
defect and create a new software version for retesting. All defects are xed until
product reaches specied requirements and quality measurements [
Software regression testing is one of the types of testing levels. In software,
a regression testing is performed after a new feature is implemented and nds
if this new implementation causes unexpected results. It is necessary to perform
regression testing after version change is made in software, not just to nd
defects. Because these changes made in system may have distorted the structure.
Thus, it is tested whether the functions lose their functionality or are harmed by
the newly added functions.When enough time is allocated for regression testing,
project can be tracked more closely and the success of the project is ensured
[
In today’s world, large number of embedded applications are used in
industrial communication area. Their size and sophistication are increasing as a result
of their wide usage. So that, eective regression testing of embedded applications
have great signicance[
Regression testing can be used for each level of software testing. It is used
in integration part of our software development. After build is released,
automated regression testing starts and validates that unchanged parts of code are
not aected by the code changes. Plenty of time is spent setting up the test
environment, analyzing test results, adding new features to test automation
system and dealing with the environmental issues. In this approach, maintenance
eort is decreased for existing test suite and more eort can be used for new
test cases. These all test cases will be nished earlier and results will be covered
faster. This result, not only improves the suitability of the developments, but
also it can provide great contributions to software quality [
There are three test levels as the White-box, the Black-box and the Grey-box testings. Integration test is related with the Grey-box testing. Software mostly consists of many modules. Integration testing shows how dierent modules of software product work together. 2.2
Regression testing is a type of testing ensures that changed or updated code has not broken any functionality of software. Tests of existing functionalities have been running again to be sure that they have not damaged anything. Regression testing can be done at any type of testing (e.g., unit, integration, system, acceptance etc.)
Regression testing is crucial for software development lifecycle; at the same time it has some major challenges. Since it involves running all old tests, it can consume much time comparibly other type of tests while running. It can be complicated to maintain big amount of tests. Moreover, when the number of tests increases, designing regression test environment is a hard task to achieve.
Test automation is an ecient way of running regression tests since manually running same tests repeatedly can make testers bored and less motivated. Moreover, updating tests regularly is a good way of running regression tests since it can be more dicult and complex later on. Furthermore, reviewing test design and test environment periodically to x errors and make improvements increase eciency of testing.
In our testing module, we applied regression test into integration test which means early feedbacks are given by daily builds. 2.3
Internal quality of a product is measured with the internal quality requirements. Internal quality of a product stays stable except design of product is not changed.
External quality of a product is measured with the testing of software after it is execution. During the testing period most errors can be found and xed.
Process quality helps improving product quality and product quality helps
improving quality in use. External and internal quality of software product can
be classied into characteristics which are functionality, reliability, usability,
efcieny, maintainability, portability[
Regression testing is not only about the suitability of the developments, it can also provide great contributions to software quality like reducing time consuming and getting quick feedbacks. 2.4
In the software development life cycle, maintenance is the most crucial phase
of the software delivered to the clients[
Retest all is a method for regression testing that all the tests are rerun in the
existing test suite. So the retest all method is much more expensive than other
regression testing methods which requires more time and budget[
This technique of regression testing prioritizes the test cases so as to increase a
test suite’s rate of fault detection that is how quickly a test suite detects faults
in the modied program to increase reliability.This is classied as general
prioritization and version specic prioritization. The meaning of general prioritization
is to select a sort of test cases for keeping the test coverage for every versions
of software orderly whereas the meaning of version specic prioritization does
concern a specic version of the software[
There are many Hybrid Approaches and dierent algorithms are proposed by
researchers for both Regression Test Selection and Test Case Prioritization.
1. Leon and Podgurski proposed a new hybrid approach which is combining
coverage-based and distribution based prioritisations. It is based on
observing the eects of basic coverage maximisation and repeated coverage
maximisation. Distribution based ltering techniques are more eective methods
to reveal errors according to coverage-based ltering techniques. These two
techniques are complementary in terms of nding dierent defects [
In regression test selection technique, tests are selected that are deemed
necessary from the existing test suite to verify the validity of the modied software. To
speed up and increase eciency of regression testing there are many algorithms
[
REGRESSION TESTING PROCESS COST ANALYSIS Regression testing is a repeating and growing process so that especially for large software programs it can be complex and costly in progress of time. Testers spend time for the following steps in a regression test process:
Testers develop new test cases to test new functionalities and requirements added to software. It sometimes takes weeks to add one test to regression test suite.
Testers spend time in the execution of test suite. While software is becoming larger, this costs more time and eort. Test automation is a good way to decrease eort for this step. Moreover, test automation can be executed in not working hours.
Testers investigate test results from execution. Investigating failures manually may take much time. Failure may be related with both software development and software testing.
Testers track failures until they are xed. This step related with issues such as the aect of failure, the experience of developer and it can be time consuming. Testers sometimes have to execute all test suites and validate all functionality of software. Execution and investigation of results can take long time.
Since regression testing is a time consuming process, there are many
techniques to decrease development and maintenance part of regression testing [
In the literture, there are many studies to increase eciency of regression
testing in terms of time, memory and cost. Data ow technique has been used
to specify tests for retesting after a change [
Unlike previous studies, our approach uses both test techniques and time saving method to enhance eciency of regression system. 3
The testing process begins in parallel with the requirement analysis in the Vmodel software development projects. Testing is a continuous activity throughout the software development process. Each software process has a corresponding test process. For this reason, the quality of test is as important as quality of software development process.
In our testing module, we applied regression test into integration test which means early feedbacks are given by daily builds. Re-running of full regression test suites every day can be extremely time consuming. This paper outlines grouping the tests and re-running these selected test groups in existing test suite with implemented time-saving methods. This approach not only provides time saving but also reduces the cost of regression testing. Unlike previous studies, our approach uses both test techniques and time saving method to enhance eciency of regression system.
In this study, we propose a test automation system which provides time savings without compromising the quality of the test. In the automation system, there are many scripts that have been automated to run every day. At the beginning of every script, there is a factory restart, which means that the conguration is completely erased so that the next script is not aected by the previous conguration in anyway. On average, a factory restart time lasts about 3-4 minutes, and it covers a considerable amount of time in many automation systems. For a quality testing process, if we get the same quality results in a short period of time, we will save time by removing the factory restart from the scripts. If there is a problem to congure device in scripts, we use again factory restart to avoid conguration failures. That is why factory restart could not be removed in automation system.
At this point, scripts which are related with the same content are combined with an .xml le in test automation. Each feature has a "group_order" node which is related to the "inside_group_order" node. These two nodes provide us to list our test scripts and groups. When the id of each group changes, automation performs factory restart to device. These steps are given below.
Feedback is really important for the development perspective. Test results should be as fast as it can be. It is hard to get test results faster because of these problem. The solution for these problems are grouping them and remove conguration after test script nish and do not perform restart. This grouping gives us around 20% time reduction for running test cases more to control and work on the test stations.(See Figure 1)
Generally, features are tested with unit test after every feature test system should reboot to clear conguration in devices. This helps clean start for device to check this feature behaviour.
Perl is one of the most powerful and practical programming language that can be used in the case of intensive text processing. The Perl programming language works in almost all operating systems. This specialty helps to work in independent environment.That’s why we used Perl in test automation system. It starts with reading a XML le and create a run list by order in "group_order" node than check "inside_group_order" lists that XML and runs test scripts. Automation uses Tool Command Line (TCL) to see the serial connection responses. TCL programing language uses expect extension to get responses from a terminal and sending commands to terminal. Generally, testing approach for test scripts ; even test script failed or passed, test automation send device to factory restart to eliminate conguration failure in new test script. This causes a lot of time loss. during unit tests. Avoiding this problem, automation perform restart with factory settings between each group. If there is a failure in the script in same group of tests, factory restart is applied and total run would not be eected by this failure. There could be an alternative for this approach which is:
Never use factory restart and try to undo conguration even test is failed. Continuing all test scripts with this method, could save time on factory restart. The one big problem about this situation is undo the conguration in failed scripts. There should be unexpected conguration change and this will eect all other test scripts.
Another benet of this method is, we can nd more related errors like even undo conguration about one test script would be an eect on other features and this situation decreases our vulnerability(Table 1 and Table 2 ). Device run duration be tested also with our approach with this technique, we will observe whether there is an error in the case of continuous operation.We also nd 10 failures in scripts which are in the same group because of undo conguration eect in the rmware.
This paper describes a regression test technique that uses both test selection method and time saving method. Our professional experience demonstrates that this technique helps testers about giving feedback about failures on time and also prove that related failures could be found even in integration part of the testing. This approach signicantly outperforms that easy applied time reducing technique with grouping related test cases. Testers could better understand and create better reproduction steps for defects in versions. Undo the conguration for test cases also can be a fundemental technique for our future work like mesuring software version stability without performing the factory settings to device.