This article introduces some ideas and challenges related to the testing of quantum programs. In particular, it approaches functional testing, white box testing (specially mutation) and model-based testing.
Quantum computers operate on cryogenic environments, at temperatures close to absolute zero. One of the characteristics of calculus made with quantum computers is the uncertainty of the results, that are not always the same: in fact, the same program executed two times may produce different outputs with different probabilities. As the number of executions grows up, the program output that could be considered as the actual output will be the most repeated one. This uncertainty introduces interesting challenges in quantum program testing. The situation is different if the quantum program is executed on a simulator, that produce deterministic results. However, the complexity of the quantum program may convert testing on a simulator in an unapproachable task.
In classic software engineering, a test case [
Thus, executing a test case on a quantum computer actually requires executing the same test case a number of times and, then, to compare the most repeated output to the expected output.
As well as this adaptation is required, quantum software testing requires both the adaptation of other techniques, as well as the creation of new ones. In this paper we discuss what can researchers do in this sense.
In practice, quantum programs provide quantum functionalities to conventional programs via some kind of interface. In Figure 1, the Functionaly Q calls the quantum program, which runs on a quantum computer, to perform some specific calculus.
In a functional testing, the test suite (i.e., the set of test cases) checks the different functionalities offered by the SUT. Depending on the test level, the test suite usually executes on the system the same operations that any of the actors could execute. For the system of Figure 1, the Test suite could replace the User, as illustrated on the left side of Figure 2. In the same way, a test suite for testing the quantum program will substitute the conventional system functionalities’ that require the services offered by the quantum program (right-hand side of Figure 2).
Going back to the three-parts structure of a test case, and recalling that the uncertainty requires that every test case must be executed several times: (1) the initial situation of a quantum test case will set up the initial status of the qubits, (2) as in conventional testing, the quantum circuit will be executed, and (3) finally, the test suite will save the obtained result, in order to calculate, in a further step, which is the most probable actual result. 3
In white box testing, the tester is interested in knowing which “fragments” of the
program are executed by the test cases. Thus, a program is completely tested if the test
cases have gone through all those “fragments”. Moreover, if the test cases do not
reveal errors, then the probability that the program is right is very high. The problem
here is in the granularity of the “fragment”. To measure it, there exist a number of
coverage criteria [
Thus, if a test suite runs all the statements of the SUT, it is said that its coverage is 100%. Obviously, some coverage criteria are more powerful than others: running all the methods is a coarse criterion that is subsumed by, for example, statements because if a test suite runs all the program statements, it runs also all its methods. Modified condition-decision (MC/DC) is one of the most powerful coverage criteria. It is fulfilled when every decision (understood as a set of conditions: for example, if A>B or B<C is a decision with two conditions, A>B and B<C) is executed in its two branches (true and false) thanks to the individual contribution of each individual condition. For this example, since the decision is composed by two conditions separated by OR, a complete test suite should lead the decision to the true branch thanks to A>B and to B<C (first two rows of Table 1), and to the false branch also thanks to each individual condition: since the operator is OR, this requires to do false both individual conditions, as shown in the third row. A>B B<C A>B or B<C Determinant condition true false true A>B (B<C has no influence) false true true B<C (A>B has no influence) false false false Both conditions influence
An additional white-box coverage criterion is based on mutation testing. In mutation testing, the tester creates a set of mutants, which are copies of the SUT. Each copy contains a syntactic change that, with the adequate test case, may produce an output different than the SUT’s output. Thus, the mutant can be seen as a faulty version of the original program. In this way, a mutation-based testing process finishes when the test suite has found all the faults inserted in the mutants and none in the original one. Some of the mutants produced are functionally equivalent, what means that it is impossible to find a test case that finding the inserted fault (in practice, this means that the change introduced in the mutant is not a fault, bust a code optimization or de-optimization). The coverage is measured in terms of the mutation score, which is the percentage of faults discovered (excluding equivalent mutants).
Artificial faults are inserted by means of mutation operators. A mutation operator
introduces some type of fault: some of the most typical mutation operators (see Table
2) are AOR (Arithmetic Operator Replacement), ROR (Relational Operator
Replacement) or UOI (Unary Operator Insertion). As it is seen, the artificial faults reproduce
errors that a competent programmer could commit [
if (a>b)
AOR ifr(eat>ubr)n a*b; if (a>b) return a-b; return a+b; ROR ifr(eat=u=rbn)a+b;
UOI ifr(e-tau=r=nb)a+b;
There are many different types of mutation operators for reproducing many different types of faults. There are also specific operators for specific programming languages (operators on the use of pointers in C and C++, for example) and specific technologies, such as mobile software.
Depending on the goodness of the mutation operators used, a given mutation score may subsume other coverage criteria. Mutation testing is quite suitable for quantum testing. In quantum programming, the programmer writes her/his programs directly using bits (actually qubits) and logic gates (actually quantum gates). Thus, the simple operation of adding two numbers requires to work at so low abstraction level.
Anagolum [
The program code, translated into OpenQASM, appears in Figure 3: it reserves 5 qubits for the first binary number (qreg a), 6 for the second one (qreg b), 5 for the carry bits (qreg c) and 6 classical bits (creg d) for saving the results. Initially all qubits are set to 0. Some of them (0, 2, 4 of a and 1, 3, 4 of b) are set to 1 via de x gate. Thus, the input numbers are those in Figure 3.
a b c 5 0
Then, the remaining code in the left column is composed by five blocks of three statements. Each block performs the following: 1. Via the ccx gate, if a[i]=1 and b[i]=1, then it changes the value of c[i+1]. 2. With cx, if a[i]=1, then it changes the value of b[i]. 3. The third statement changes c[i+1] if b[i]=1 and c[i]=1.
In these blocks of statements, the qubits change according to Figure 4.
5 4 3 2 1 0
ccx a[
Fig. 4. Code for adding two numbers
The code in the second column (blocks with five statements of gates cx and ccx) runs in a very similar way. The final statements (measure) read the qubit values and pass them to the classical register, d.
As it is seen, working at bit level is hard and fault prone: the programmer may be specially tempted to copy and paste for changing later the indexes of the qubits (and forgetting it), she/he may use a wrong gate, write statements in a different order…
All these possible faults are suitable of being reproduced with mutation operators. 4
In model-based testing (MBT), the test engineer models test cases with models, typically using UML. The UML specification includes the UML Testing Profile, an extension whose metamodel defines all the elements involved in tests definition (Figure 5). researched, but maybe it is possible to tailor the UML testing profile to this new programming paradigm.
This paper and its associated research is part of the project TESTIMO (SBPLY/17/180501/000503) that has been funded by JCCM Consejería de Educación y Cultura y Deportes, y Fondos FEDER.