XML queries are broadly used in Web environments, but the existing approaches towards software quality based on testing have not deeply addressed them. Although there are some works oriented to generate test inputs for testing XML queries, the evaluation of expected outputs against the actual outputs resulting from the tests has not been tackled as far as we are concerned. In this paper, a research proposal is presented to deal with the absence of the expected outputs when testing XML queries, focusing the efforts on the definition of a feasible test oracle.
In recent years, Web Engineering has emerged as a new discipline motivated by the
swift growth of the World Wide Web. This growth has been accompanied by an
increasing complexity of Web applications (WebApps) as we can clearly see, for
example, in today’s WebApps formed by compositions of Web Services created, in
turn, with heterogeneous fast-evolving technologies. The development of these
complex WebApps, like other software systems, should involve quality criteria to
guarantee a certain degree of reliability based on testing, verification and validation
activities. Regarding testing, current approaches on WebApps are commonly centered
in checking static and dynamic navigational paths [
There are well-known technologies intended to carry out data access operations,
such as the SQL language for relational databases, but XML-based technologies for
data querying are becoming popular as is the use of XML-based formats for data
representation and interchange in the Web. The existing approaches and tools to test
data access operations on XML-based applications are commonly conceived to test
data repositories or query engines [
To address the lack of the expected outputs, some researchers state that there
should be an entity, called oracle, capable to determine whether or not a program
under test has behaved as expected during execution. The definition of such oracle
derives in the so called oracle problem, as oracles are usually difficult or even
impossible to obtain [
In this research proposal we present a line of work intended to tackle some of the problems concerning testing of queries for data access in XML-enabled environments (such as the Web), with a focus on the definition of a test oracle oriented to solve the absence of the expected outputs during testing.
The paper is structured as follows. In section 2 the aims and objectives of the research are outlined. Section 3 briefly describes the current work and, finally, in section 4 the expected contributions of the research and plans for future work are presented. 2
The current research activity in the field of test oracles is not significant in contrast
to the huge variety of the existing approaches involving test input generation.
Amongst the works encompassing test oracles, partial oracles seem to be the best
alternative to XML query testing. Such partial oracles are capable to determine
whether an actual output of a test case is incorrect while ignoring the correct output
[
With these considerations, our research hypothesis is intended to prove that a
partial oracle is a viable alternative when dealing with large volumes of data for
testing. In particular, we are aiming to prove that a partial oracle may be suitable to
test data access operations based on queries in XML-enabled environments. To cover
the hypothesis, the following objectives are in scope:
1. Define a feasible partial oracle to test XML queries. The automation of the oracle
will be one of the main points to consider.
2. Validate the proposal to prove its effectiveness in a real testing scenario, which
may include developing a proof of the concept to show the practical application of
the proposal.
3. Find possible alternatives to the proposed oracle if required. The proposed oracle
may not be suitable in every case. Because of that, other alternatives should be
studied, specially the existing ones that could be adapted to XML query testing,
such as metamorphic testing [
The complexity of the XML queries is a concern since there are many broadly used
alternatives oriented to query XML data. XML queries are commonly expressed using
the XPath [
With this abstraction, we propose the use of a partial oracle to test the query programs according to a specification composed by the following two elements: ─ Behavioural requirements, which establish a specification about the correct behaviour of a determined target query program under test. This specification is loose, which means that it need not be complete, and ─ Oracle constraints, which define invariant properties of the expected outputs that must be satisfied by every correct query program. Each oracle constraint is intended to check the presence a known type of fault.
Since behavioural requirements depend on the target program, the tester must provide them manually. In contrast, oracle constraints are invariant; thus, they could be embedded in the partial oracle internals and their evaluation could be automated.
Fig. 1 shows the structure of the oracle and its integration in the testing environment. As it can be seen, the oracle receives as input the behavioural requirements, as well as the test input and the actual output of the test, and yields a “Pass” or “Fail” response as a result of the oracle constraints evaluation.
Because oracle constraints specify general properties of the expected outputs, prior to their evaluation, they are particularized to the target programs under test by means of the behavioural requirements. Whenever an oracle constraint is not satisfied given a set of behavioural requirements, it is understood that the partial oracle has detected a fault, in which case, the partial oracle should respond with a “Fail” message.
In the paper in [
In order to ease the manual specification of the query program under test, behavioural requirements are loose, which means that they do not comprise a complete specification of the target program. However, the precision of the partial oracle will increase as the tester specifies these requirements closer to the complete specification. This provides flexibility to balance the relationship between the cost and the precision of the oracle.
The behavioural requirements are given by the tuple
S = < A , Ds , ts >, s where: ─ As is a set of XML nodes (XML elements or attributes) that are expected to be in the actual output of the target program, but are not included in the test input, ─ Ds is a set of XML nodes that are expected to be in the input, but not in the actual output. ─ ts is a relaxed specification-compliant query. It is defined as a function with the same domain and image as the query under test q . Then, the property qs (I ) ⊆ ts (I ) must hold for the definition of ts , where qs denote the correct implementation of the query that meets the specification, also in the domain and image of q , and qs (I ) is the expected output for the test input I . Note that qs it is not available for the tester. In practice, ts is expected to be a less expressive query than qs , and hence, it should be less error-prone and easier to specify. For example, suppose that we need to obtain the title of an XHTML Web page only if its body contains the string value "Contents". The specification-compliant query, qs would be represented with the XPath syntax as /html/head/title[../body = "Contents"]. Then, ts could be defined as the query /html/head/title, which always returns the title of the page, independently of the body contents. As seen, ts should be simpler and easier to code than qs .
The elements of the behavioural requirements tuple need not be defined as a complete specification. It is not required to specify every possible node in As or Ds , and the query ts does not need to be as expressive as qs in order to detect some types of errors. In the next section, the use of these behavioural requirements is shown. 3.2
Oracle constraints
Oracle constraints set necessary conditions for the correct behaviour of the queries under test. Since these constraints are intended to deal with operational aspects of the querying processes, they are domain-independent, and thus, can be embedded in the oracle (as shown in Fig. 1). Some of the oracle constraints we currently propose are presented and described below.
─ q(I ) I As = A . This constraint establishes that the XML nodes specified in s the set As must be contained in the actual output, q(I ) . ─ q(I ) I Ds = Ø . The constraint establishes that the actual output must not contain nodes from Ds . ─ q(I ) ⊆ ts (I ) . It checks that the actual output is contained in the relaxed specification-compliant query. If this constraint is satisfied, it means that data selection faults (such as mistaken predicates or bad node references) were not detected in the query under test, q .
Note that each constraint makes use of a different behavioural requirement from the tuple in Section 3.1, and every behavioural requirement could be defined with a custom precision by the tester.
After a test execution, if any of the constraints does not hold for a given query program, then a fault has been detected, and the oracle could notify the cause of that fault with a human-readable message for the tester. 4
The expected contribution of the research to Web Engineering consists on improving the quality of WebApps by means of testing XML queries. For this purpose, we focus on an approach to a test oracle whose main characteristics are outlined below: 1. Allows the tester to balance the cost and the effectiveness of the oracle. The tester can provide a loose specification (behavioural requirements) of the program under test, instead of giving a complete specification, which would be more costly and complex. 2. The proposed oracle is highly automatable. The mechanism of the oracle depends on the rules defined by oracle constraints whose evaluation could be automated.
At this moment, the proposed oracle can detect a relatively small set of faults. For future work, we plan to add/modify behavioural requirements and oracle constraints (as shown in the example in Section 3) to enhance the fault detection capabilities of the oracle. It is also necessary to define a systematic method for the evaluation of oracle constraints as this will result in the automation of the oracle. Besides, the oracle proposal should be validated to prove its effectiveness. We are planning to do the validation by using experimental techniques such as mutation, and by providing real/industrial case studies. 5 6 This work was partially funded by the Department of Education and Science (Spain) and ERDF funds within the National Program for Research, Development and Innovation, project Test4SOA (TIN2007-67843-C06-01) and the RePRIS Software Testing Network (TIN2007-30391-E).