Regression test suites of evolving software systems are often crucial to maintaining software quality in the long term. They have to be effective in terms of detecting faults and helping their localization. However, to gain knowledge of such capabilities of test suites is usually difficult. We propose a method for deeper understanding of a test suite and its relation to the program code it is intended to test. The basic idea is to decompose the test suite and the program code into coherent logical groups which are easier to analyze and understand. Coverage and partition metrics are then extracted directly from code coverage information to characterize a test suite and its constituents. We also use heat-map tables for test suite assessment both at the system level and at the level of logical groups. We employ these metrics to analyze and evaluate the regression test suite of the WebKit system, an industrial size browser engine with an extensive set of 27,000 tests.
Regression testing is a very important technique for maintaining the overall
quality of incrementally developed and maintained software systems [
Test suite maintenance is not easy and imposes high risks if not done
correctly. In general, the lack of systematic quality control of test suites will reduce
their usefulness and increase associated regression risks. Difficulties include their
ever growing size and complexity [
Hence, a more disciplined quality control of test suites requires that one is able to understand the internal structure of the test suite, its elements and their relation to the program code. Without this information it is usually very hard to decide about what parts of the test suite should be improved, extended or perhaps removed. Today, the typical information available to software engineers is limited to knowing the purpose of the test cases (the associated functional behavior to be tested), possibly some information about the defect detection history of the test cases and – mostly in the case of unit tests – their overall code coverage.
Furthermore, most of previous work related to assessing test suites and test
cases are defect-oriented, i.e. the actual amount of defects detected and corrected
are central [
In our previous work [
In this paper, we adapted the TAIME approach to use it for assessment
purposes and verified the usefulness of the method for the comprehension of the
test suite of the open source system WebKit [
were able to characterize the test suite both as a whole and its constituents and eventually provide suggestions for improvement.
In summary, our contributions are the following: 1. We adapted the TAIME method to provide an initial assessment of a large regression test suite and present a graphical representation of the metrics computed from code coverage. 2. We demonstrate the method on a large open source system and its regression test suite, where we were able to identify potential improvement points.
The paper continues with a general introduction of the analysis method (Section 2) and an overview of the metrics used for the evaluation (Section 3). Then, we demonstrate the process by evaluating the WebKit system: in Section 4, we report how functional units were determined, while the assessment itself is presented in Section 5, where we also introduce the heat-map visualization. Relevant related work is presented in Section 6, before we conclude and outline future research in the last section. 2
Our long-term research goal is to elaborate an assessment method for test suites,
which could be a natural extension to existing software quality assessment
approaches. The key problem is how to gain knowledge of overall system properties
of thousands of tests and at the same time understand lower level structure of
the test suite to draw conclusions about enhancement possibilities. In our
previous work [
z ?
eu
6
Set granularity eu ?
eu eu
uj
test suite and the program code into different groups. This can be done in
several ways, e.g. by asking the developers or, as we did, by investigating the tests
and the functionality what they are testing. After we have a set of features
we can decompose the test suite and the code into groups. Those tests, which
are intended to test the same functionality, are grouped together (test groups).
The features are implemented by different (possibly overlapping) program parts
(such as statements or procedures), which we call code groups. In the following,
we will use the term functional unit to refer to the decomposed functionality,
which we consider as a pair of associated test group and code group (see
Figure 2). The whole analysis process is centered around functional units, because
by this division the complexity of large test suites can be handled more easily.
The decomposition process (the number of functional units, the granularity of
code groups and whether the groups may overlap) depend on the system under
investigation. It may follow some existing structuring of the system or may
require additional manual or automatic analysis. More on how we used the concept
of functional units in our subject system will be provided later in the paper.
According to the GQM approach [
In this section we give an overview of metrics, coverage and partition that will be used for test suite evaluation. All metrics will be defined for a pair of a test group T and a code group P . Let T be the set of tests of a functional unit and P be the set of procedures which usually belong to the same functional unit. 3.1
Coverage Metric (cov) We define the Coverage metric (cov) as the ratio of the number of procedures in the code group P that are covered by test group T . We consider a procedure as covered if one of its statement was executed by a test case. This is the traditional notion of code coverage, and is more formally given as: cov(T; P ) = jfp 2 P j p covered by T gj : jP j
Code coverage measures are widely used in white-box test design techniques, and they are useful, among others, to enhance fault detection rate, drive test selection and test suite reduction. This metric is easy to calculate based on the coverage matrix produced by test executions. Possible values of cov fall into [0; 1] (clearly, bigger values are better). 3.2
Partition Metric (part)
We define the Partition metric (part) to express the average ratio of procedures
that can be distinguished from any other procedures in terms of coverage. The
primary application of this metric is to support fault localization [
Pp2P (j pj jP j (jP j 1) 1) :
The numerator can be also interpreted as the sum of j j (j j 1) values for all different partitions. It will be best (1) if test cases partition procedures so that each procedure belongs to its own partition, while it will be the worst (0) if there is only one big partition with all the procedures.
C = tttttt452367 BBBBBBBBB@ 110111
p1 p2 p3 p4 p5 p6 t1 0 1 0 0 0 0 0 1 1 0 0 0 11 10 10 00 101001 CCCCCCCCCA 1 1 1 1 1 1 0 0 1 0 0 1 t8 1 1 1 1 0 0
The cov and part metric values for our example coverage matrix (see Figure 3) can be seen in Table 1. The metrics were calculated for four test groups and for the whole matrix, the code group always consisted of all procedures. This example exhibits several cases where either cov is higher than part, or the other way around. Although in theory there is no direct relationship between these two metrics, with realistic coverage matrices we observed similarities for our subject system as described later.
The first two steps in the adapted TAIME method is to set the granularity and
determine the test and code groups. In this section we present the extraction
process of these groups from the WebKit system [
We chose to work on the granularity of procedures (C++ functions and
methods) due to the size and complexity of the system, but our approach is
generalizable to finer levels as well. (In [
The next step was to determine test and code groups. WebKit tests are maintained in the LayoutTests directory. It contains test inputs and expected results to test whether the content of a web page is displayed correctly (they could be seen as a form of system level functional tests). In addition to strictly testing the layout, it contains, for example, http protocol tests and tests for JavaScript code as well. Tests are divided into thematic subdirectories to group tests of particular topics. These topics are the different features of the WebKit system and this separation made by the WebKit developers gave the base of the decomposition of the program into functional units. Based on this separation, the decomposition of the test suite into test groups was a natural choice.
At the separation of the program code, it is important to note that the associated code groups were not selected based on code coverage information, rather on expert opinion about the logical connections between test and code. Code groups could be determined automatically based on the test group coverage, but in that case the coverage of code groups would be always 100%. Our choice to involve experts helps to highlight the differences between the plans of the developers and the actual state that the test suite provides.
The resulting functional units can be observed in Table 2. We identified a total number of 84142 procedures in WebKit, and the full test suite contains 27013 tests, as shown in the first row of the table.4 The rest of the table lists selected functional units and the statistics of the associated test and code groups, which are detailed in the following subsections. We asked 3 experienced WebKit developers – who have been developing WebKit at our department for about 7 years – to determine functional units of the WebKit system. 4.1
Test Groups As the experts suggested, the test groups were determined based on how WebKit developers categorize tests. The LayoutTests directory contains a separate directory for each functionality in the code that is intended to be tested. These directories (and their sub-directories) contain the test files. A test case in WebKit is a test file processed by the WebKit engine as a main test file, which can include other files as resources. The distribution of the number of tests in these main test directories can be seen in Figure 4. There are several small groups starting form the 12th largest directory in the middle of the diagram. We could either put them into individual groups, which would make the analysis process inefficient (with too many groups but less gain), or we could threat them as one functional unit breaking the logical structure of the groups. Finally we decided to exclude these tests, so about 91% of tests were included in our experiments.
executed by the test scripts. We used revision r91555 of the Qt (version 4.7.4) port of WebKit called QtWebKit on the x86_64 Linux platform. Also, only C++ code was measured.
Number of Tests in each Group
However, we found some inconsistencies in the test directory structure, so made further adjustments to arrive at the final set of groups. First, the fast directory contains a selection of test cases for various topics to provide quick testing capability for WebKit. This directory is constantly growing, and takes about 27% of all the tests in WebKit. Unfortunately, it cannot be treated as a separate functional unit as it contains tests that logically belong to different functions. Hence we associated each fast/* subdirectory to some of the other groups and a small, heterogeneous part is left uncategorized, marked as fast-remaining (starting with this point we cut the remaining test groups, see in the middle of Figure 4). Second, there exist three separate test groups for testing JavaScript code. This language plays a key role in rendering web pages, and the project adopted two external JavaScript test suites called sputnik and ietestcenter. In addition, the WebKit project has its own test cases for JavaScript in the fast/js directory. We used the union of these three sources to create a common JavaScript category called js in our tables. Once the features were formed, we asked the experts to assign source code directories/files to functionalities based on their expertise, but without letting them access the coverage data of the tests. This task required two person-day effort from the experts and the result of this step was a list of directories and files for each functional unit. The path of each procedure in the code was matched against this list, and the matching functional unit(s) were assigned to the procedures. We chose this approach to avoid investigating more than 84000 procedures one by one and so we determined path prefixes and labeled all procedures automatically. However, this method resulted in a file level granularity in our dataset, and increased the overlap between functional units. Eventually, out of 84142 procedures we associated about 28800 with one of the functional units, during code browsing only procedures strictly implementing the functionality of the units were selected. The omitted procedures usually included general functionality that could not be associated with any of the functional units unambiguously. At the end of this process, all experts accepted the proposed test and code groups.
We used several tools during this process. For the identification of all
procedures in WebKit (including non-covered ones) the static C/C++ analyzer of the
Columbus framework [
At the end of this process we determined the groups, executed the test suite and measured the coverage using the given metrics. 5
In this section, we present the last, analyze step of the proposed TAIME approach by first presenting our measurement results using the introduced metrics on WebKit, then at the end of the section we evaluate the system based on the result and determine possible enhancement opportunities. 5.1
Coverage metrics In the first set of experiments, we investigated the code coverage metrics for the different functional units to find out if a test group produces higher overall coverage on the associated code group than the others. In particular, for each test group - code group pair we calculated the percentage of code that the corresponding test cases cover, which we summarize in Table 3. Rows in the table represent test groups while code groups are in the columns, and each cell contains the associated coverage metric cov. For example, we can observe that tests from the canvas functional unit cover 26% of the procedures of unit css.
The first row and column of this table show data computed for the whole WebKit system without separating the code and tests of the functional units. The coverage ratio of the whole system is 53%, which means that about 47% of the procedures is never executed by any of the test cases (note that for computing system level coverage all tests are used including the 9% tests omitted from individual groups). This is clearly a possible area for improvement globally, but let us not forget that in realistic situations complete coverage is never aimed due to a number of difficulties such as unreachable code or exception handling routines.
We can interpret the results of Table 3 in two ways: by comparing results in a row or in a column. In the former case, we can observe which code groups are best covered by a specific test group, and in the latter the best matching test group can be associated with a code group. Using the latter it can be observed which tests are in fact useful for testing a particular piece of code. We used a graphical notation in the matrix of two parallel horizontal lines for rows and two PPPPPCPodPe Test WebKit canvas css dom editing http js svg m o d g v s .72 .45 .62 .57 .59 .63 .57 .63 .56 .62 vertical lines for columns, respectively, to indicate the maximal values. In order to have a more global picture of the results, we also present them in a graphical way in the form of a ‘heat-map’: the intensity of the red background of a cell is proportional to the ratio of the cell value and the column maximum, i.e. the column maxima are the brightest.
The most important to observe is that, except for tables, each code group is best covered by its own test group. This is an indicator of a good separation of the tests associated with the functional units. In the other dimension, there are more cases when a particular test group achieves higher coverage on a foreign code group, but the value in the diagonal is also very high in all cases. The dominance of the diagonal is clearly visible, however there are some notable cases where further investigation was necessary. For example code group http, which is very specific to its test group, and tables, which does not have a clear distinction between the test groups. We are going to discuss these cases in more detail at the end of this section. Another observation is that the coverage values in the main diagonal are close to the overall coverage of 53%. 5.2
Partition metrics The partition metrics for the functional unit pairs showed surprising results. As can be seen in Table 4 – that also shows the corresponding heat-map information –, the part metric values basically indicate the same relationship between the test and procedure groups as the coverage metrics. In fact, the Pearson correlation of the two matrices of metrics (represented as vectors) is 0.98. PPPPPCPodPe Test WebKit canvas css dom editing http js svg s s c m o d g v s
In general, the coverage and the partition metric values do not necessarily
need to be correlated, which is illustrated also in our example from Figure 3 and
Table 1. However, certain general observations can be made as follows. When
the number of tests and procedures are over a certain limit, it is more likely that
a unit with high coverage will include different tests and produce high partition
metric as well, but it can happen that a unit with high coverage consists of test
cases with high but very similar coverage values, in which case the partition
metric will be worse. On the other hand, if the coverage is low, it is unlikely
that the partition metric will be high because non-covered items do not get
separated into distinct partitions. In earlier work [
Characteristics of the WebKit test suite We summarize our observations on functional units that we found during the analysis of metrics in the previous sections.
Regarding the special cases, we identified two extremes in the system. On the one hand, there are functional units, where code groups are not really exercised by other test groups, while their test groups cover other code groups. One of these groups is http. By investigating this group, we found that most functionalities – i.e. assembling requests, sending data, etc. – are covered by the http test group, while other test groups usually use basic communication and small number of requests. The number of test cases in these groups could probably be reduced without losing coverage, but only with taking care of tests which cover the related code of the group.
On the other hand, there are groups like the tables group which is some kind of an outlier in the sense that this code group is heavily covered by all of the test groups. The reason for this is that it is hard to separate this group from the code implementing the so-called box model in WebKit, which is an integral part of the rendering engine. Thus, almost anything that tests web pages will use the implementation of the box model, which is mostly included in the table code group. Hence, the coverage data is highly overlapping. Although its coverage is the highest one tables maintains good part metrics. Highly covered by other test groups, tables should be the last one to be optimized among the test groups. The number of test cases in this group could probably be reduced due to the high coverage by other modules, however, more specific tests could be used to improve coverage.
According to our analysis there is room for improving the coverage of all code groups as they are around the overall coverage rate of 53%. Another general comment is that component level testing (unit testing) is usually more effective than system level testing (as is the case with WebKit) when higher code coverage is aimed. For example, error handling code is very hard to be exercised during system level testing, while at component level such code can be more easily tested. Thus, in a long term, introducing a unit test framework and adding real component tests would be beneficial to attain higher coverage.
In summary, the experience that we gatherd from analyzing the data of the
adapted TAIME method appoints two main ways to improve testing. The
proposed metrics can either be used to provide guidance for the developers during
the maintenance of the tests, or to focus test reduction on enhancing specific
capabilities of the test suite, e.g. fault detection and localization [
Although there exist many different criteria for test assessment [
There have been sporadic attempts to define metrics that can be used to
assess the quality of test suites, but this area is much less developed than that
of general software quality. Athanasiou et. al. [
Researchers started to move towards test oriented metrics only recently,
which strengthens our motives to work towards a more systematic evaluation
method for testing. Gomez et. al. provide a comprehensive survey of measures
used in software engineering [
Code coverage based test selection and prioritization techniques are also
related to our work as we share the same or similar notions. An overview of
regression test selection techniques has been presented by Rothermel and Harrold, who
introduced a framework to evaluate the different techniques [
Systematic evaluation of test suites to support various evolution activities is largely an unexplored area. This paper provided a step towards establishing a systematic and objective evaluation method and measurement model for (regression) test suites of evolving software systems, which is based on analyzing the code coverage structures among test and code elements. One direction for future work will be to continue working towards a more complete test assessment model by incorporating other metrics, possibly from other sources like past defect data.
We believe that our method for test evaluation is general enough and is not limited to the application we used it in. Nevertheless, we plan to verify it by involving more systems having different properties, and in different test suite evolution scenarios such as metrics-driven white-box test case design.
Regarding our subject WebKit, our observations are based mostly on the introduced metrics and we did not take into account the feasibility of the proposed optimization possibilities of the tests. We plan to involve more test suite metrics and investigate our actual suggestions in the future.