Model-based testing is a compelling method for the end-to-end testing of microservices. However, when testing with a large number of services, state space explosion is a common problem. It is especially a problem since input-output conformance (ioco) is not compositional. We developed a novel and theoretically grounded testing method called model-based mocking (MBM) to end-to-end test microservice systems compositionally. We tested the MBM method using the Axini Modeling platform by inserting 20 mutants into an example microservice system. In our set of inserted bugs, MBM found more than half of the bugs faster compared to other methods and was slower for none of the bugs.
should be amenable to MBT: one can describe the entire system by modeling the individual services, and models can be combined to describe the entire system. However, since there is often a large number of services, state space explosions are common.
To avoid state space explosions, it is desired to test systems individually. However,
iocotesting is not compositional: even though individual SUTs may conform to their specification,
their parallel composition may not conform to the parallel composition of the specifications.
Some solutions have been proposed. Input-enabled specifications are for instance
compositional, and input-enabled specifications can be using
demonic completion [
Environmental testing tests if top-level services use their components correctly according to
their STS specification [
Contributions. In this paper, we develop a theoretical approach called model-based mocking (MBM) for testing microservices using ioco with on-the-fly ambiguous state detection. In particular, we show that when testing a specific microservice, the services that are used can be replaced by a mock (generated by the model), which simulates the implementation that adheres to the specification. In theory, this reduces communication and processing delays in the testing process. It is not required for the specifications to be input-enabled. Rather, it is suficient to check for (absence of) ambiguous states at runtime. We evaluate the theory by implementing the technique in AMP, and testing 20 diferent mutants of a microservice. We show that MBM often outperforms existing testing approaches.
Input-output conformance (ioco) is a theory for MBT based on labelled transition systems
(LTSs) [
where is a countable, non-empty set of states; and are countable sets of input and output labels, such that = ∪ , and ∩ = ∅. We write = ∪ { }, where /∈ is the internal action. ⊆ × × is the transition relation, and 0 ∈ is the initial state.
An input-output labeled transition system (IOLTS) is a 5-tuple ⟨, , , , 0⟩ We use the following notation. Let , ′ ∈ , ∈ , , ∈ and →− ′ for (, , ′) ∈ , →− if →− ′ for some ′, and ↛−
if ¬(→− the transition relation to weak transitions in the standard way, that is, we write if ∃1, · · · if there is a (possibly empty) sequence of -transitions from to ′; we write ⇒ →1− 2 ⇒ ′ for some 1, 2. We generalize this to weak traces by writing ==1=·..=.·=⇒ ′ , : = 1 =⇒1 · · · ===−⇒1 = ′, and ⇒ if there exists ′ such that ⇒ ′. ⇒ ⇒ ′
′ if ∈ * . We write ). We generalize input-enabled IOLTS is referred to as an input-output transition system (IOTS).
An IOLTS ⟨, , , , 0⟩ is input-enabled if ∀ ∈ , ∈ : ⇒ . An A state from which no output can be produced is called quiescent.
quiescent if for all
Let be an IOLTS ⟨, , , , 0⟩ and ∈ * a trace of . A state ∈ is ∈ ∪{ }, →− ̸ . We write () if is quiescent. We can make the observation of quiesence explicit by extending our IOLTS. We write Δ() = ⟨, , ∪ { }, Δ( ), 0⟩ for for ∪ { }. The suspension traces of are Straces() = { ∈ * | 0 ⇒Δ }. this IOLTS, where Δ( ) = ∪ {(, , ) | ∈ ∧ ()}. We write →Δ and ⇒Δ when we explictly refer to this transition relation or the corresponding weak transition relation, and use In the definition of ioco, we further use the following notation. The set of states in which the system can be after executing is denoted after = {′ | ⇒Δ ′}. The outputs that are enabled in state are () = { ∈ | ⇒ Δ }. We use after to denote 0 after .
Let be an IOTS and an IOLTS with input labels and output labels . Then ioco = ∀ ∈ () : ( after ) ⊆ ( after ).
Parallel composition (‖) is used to define how actions are ordered between two processes. Given two IOLTS 1 and 2, then the set of actions must synchronize (i.e. happen at the same time) is = (1 ∩ 2 ) ∪ (1 ∩ 2 ). Then all other actions (1 ∪ 2) ∖ are interleaved. is the minimal set satisfying the following inference rules (where ∈ ):
1 ∩ 2 = 1 ∩ 2 = ∅. If 1 and 2 are composable, then 1 ‖ 2 = ⟨, , , , 1 where = {(1, 2) | 1 ∈ 1, 2 ∈ 2}, = (1 ∖ 2 ) ∪ (2 ∖ 1 ), = 1 ∪ 2 , and ‖ 2⟩ For = 1, 2 let be IOLTS ⟨, , , , ⟩. 1 and 2 are composable if 1→− 1 ‖ 2→−
1′, /∈ 2 1′ ‖ 2 2→− 1 ‖ 2→−
2′, /∈ 1 1 ‖ 2′ 1→−
1′, 2→− 1 ‖ 2→− 2′, ̸= 1′ ‖ 2′ Note that, in this definition, if 1 and 2 synchronize on action , is an output of 1 ‖ 2.
Modern systems are often composed of multiple components running in parallel. Ideally, we
want to be able to test the individual components, as this allows for simpler descriptions, and
alleviates the state space explosion problem, while still being able to draw conclusions about
the system as a whole. However, in general, ioco is not compositional: if implementation
1 ioco 1 and 2 ioco 2, it is not generally the case that 1 ‖ 2 ioco 1 ‖ 2 [
Diferent solutions have been proposed to allow for compositional testing. For instance,
when all specifications are input-enabled, ioco is compositional [
For specifications without ambiguous states, ioco is compositional [
In this paper, we do not perform direct compositional testing. Instead, we use the results from Daca et al. to show that we can replace implementations that we do not want to test with their specification. This is formalized using the following result.
Theorem 3.2 (ioco-substitution for an ambiguous-free interaction). Let 1, 2 be IOTSs and 1, 2 be IOLTSs, with respective input and output labels (1 , 1 ) and (2 , 2 ). Let 2 be inputenabled for labels 2 ∖ 1 and 1 ‖ 2 contain no ambiguous states.
(2 ioco 2) ∧ (1 ‖ 2 ioco 1 ‖ 2) ⇒ 1 ‖ 2 ioco 1 ‖ 2 Note that this requires input-enabledness only for 2 ∖ 1 , which weakens the requirements needed to get compositionality for ioco in general. Now 2 only needs to be input-enabled for actions that are not part of the communication with 1. In practice, this is a useful requirement: if we can restrict 2 such that it only describes the interface-behaviour needed to communicate with 1, it does not have any other inputs. This is, e.g., typically the case in a client-server or module-submodule context.
Traditionally, in MBT we test whether an implementation conforms to its specification by generating inputs to the SUT based on the specification, and observing the outputs of the SUT and checking these are allowed by the specification. This requires an adapter to translate between actions in the specification and calls in the implementation.
We use the results from the previous section to obtain a testing setup using model-based mocking (MBM). The idea is efectively to replace the systems we are not testing with correct implementations, so called mocks, that are inferred from the specifications. However, we want to avoid manually developing the mocks. Instead, we use the specifications and existing adapters of the mocked components to allow the testing platform to act as mock. For this, AMP listens to the communication between the SUT and the mocked services, uses the adapter to intercept the calls from the SUT to the mocked service, translates the call to an action in the specification of the mocked service, generates a response based on the specification, and translates it back to a call that can be forwarded to the SUT. The approach is sketched in Figure 1b. This way, assuming the services that the SUT depends on are correct, MBM allows us to isolate the SUT, and focus the testing efort on it.
Note that our theory requires that 1 ‖ 2 does not contain ambiguous states. In our implementation, we handle this during testing. In practice, 1 will not wait for 2 to be ready to receive the message. If 2 receives an unexpected message, this is reported to the testing platform, and the test fails: either 2 is incomplete or we have detected a bug in the SUT. (a) (b)
The existing approach to isolated SUT testing in AMP is shown in Figure 1a. The tester considers transitions in the specification of the SUT. If the tester triggers an an input in some service, AMP sends the corresponding action label to an adapter, that translates it into either an HTTP or AMQP message for the relevant component. Messages that are received in return, such as an HTTP response, are received, and translated by the adapter into an action in the specification. The tester than takes the corresponding transition in the specification. If the response does not match the specification, the test fails, otherwise testing can continue. Internal communication is not listened to or interacted with.
MBM allows for a diferent setup shown in Figure 1b. The services that are used by the SUT are modelled in AMP. Calls that are made to these services from SUT are intercepted, and the existing adapter is used to translate messages into actions. These actions are matched to the specification of the service, and an appropriate response is selected based on the service specification. The response is again translated into a message using the adapter, and sent back to the SUT. When a synchronizing transition happens between a mocked service and the SUT, the transition is handled as if it is a external transition from the perspective of the SUT. Internal transitions between mocked services are considered hidden if the original transition was hidden, otherwise it is a regular mocked output. We do not require a priori checking of ambiguous states. Instead, ambiguous states between the SUT and the mock can be detected at runtime.
To study the efect of mocking on testing performance, we compare three testing methods. In naive parallel composition we first compute the parallel composition of all specifications, and use this specification to test all implementations together. To fully compare the testing methods, the specification for this method is precomputed, and added to AMP. Simultaneous testing is AMP’s default testing approach. Here all implementations are tested together, however, the parallel composition of the specifications is not precomputed. Instead, the current state of all the specifications is tracked on-the-fly. This should have the same testing efectiveness as naive composition. Model-based mocking is our new approach. It tests a single service; the remaining services are mocked.
As system under test (SUT) we use the eShopOnContainers project3. This is an open source
web shop built using .NET microservices. We model four services of the project: the basket
service, the catalog service, the ordering service and the payment service. When a user checks
out in the web shop, all services will be part of the processing of the checkout. The specification
model was derived from van den Brink’s model [
The basket API is the service the end user interacts with the most. The user triggers the
checkout from it, after which the rest of the services interact to finish the checkout. The basket
API is the service we want to test. For MBM, we mock all services, except the basket. To
compare the diferent testing methods, we consider 20 diferent mutants of the basket service,
and compare the time it takes for each testing method to find the bug in the mutant. The mutants
are those originally generated by van den Brink [
All tests have been run on a HP ZBook Studio G4 laptop with Intel Core i7-7700HQ CPU, 16GB of RAM and an NVIDIA Quadro M1200. All implementations of services were run on this same machine. To introduce latency, we use the tc tool to simulate a fixed round-trip latency of 100ms. We used a timeout of 500 steps. Each of the test runs is repeated 10 times.
Each of the testing approaches was able to find 19 out of 20 mutants. Mutant 11 was not detected
by any of the approaches. This is consistent with the findings from [
The models we use are complex, and we cannot assume a particular distribution of the data. Also, the data for diferent experiments are not independent. To statistically compare the testing methods, we therefore used the Kolmogorov-Smirnov test. The test compares the cumulative distribution function (cdf) of two distributions to see if one is significantly bigger or smaller at diferent points in the distribution. The cdf of our tests is calculated using the percentage of tests that has completed at a specific time, for instance after 30 seconds. If the cdf is significantly greater at any point in time than the other cdf, the testing method is a faster testing method.
The results are summarized in Table 1. The table shows in how many instances the test on the left was significantly faster than the test on the right, for < 0.05 and < 0.10. For example, the last row in the table shows that mocking was significantly faster than simultaneous testing in 6 out of 19 cases ( < 0.05), or 8 out of 19 cases (when < 0.10). The last column shows that mocking was never significantly slower than parallel or simultaneous testing.
If we look at the diferences between the methods for individual mutants, we observe that the tests using mocking often has fewer extreme outliers in the test. This can for instance be seen in the results for mutant 1 in Figure 2a. For mutant 13, whose detection requires interaction with the mocked services, mocking is significantly faster than simultaneous testing (both = 1.082 × 10− 4), see Figure 2b.
3https://github.com/dotnet-architecture/eShopOnContainers
(b) Mutant 13 From the results we observe that mocking is significantly faster than naive parallel testing most of the time, and it is significantly faster than simultaneous testing in about one third of the cases. Simultaneous testing is sometimes faster than naive parallel composition.
We have two reasons to think simultaneous testing was sometimes faster than naive parallel composition. First, higher memory consumption required by explicitly building the parallel composition can cause a performance decrease during testing. This is because garbage collection has to occur more often for the .NET applications, interrupting the regular flow of the program. Second, testing using parallel composition might require visiting diferent interleavings that represent equivalent paths. Simultaneous testing does not visit these diferent interleavings.
Regarding mocking, we should note that the mutants were not chosen specifically to require that they are on a path that includes mocked services. In fact, only mutant 13 requires interacting with mocked services to find the bug. Our results show that in particular in case of mutant 13, mocking is much faster than simultaneous testing. We expect that this is mainly due to lower communication and processing delays in the mock compared to the original simulation. We expect a similar performance improvement for other test cases that require mocked services.
All mutants except mutant 13 can be detected by the model following only transitions from the basket service. However, even in those cases, mocking is often faster. We expect that this is again due to the mocked services having lower communication delays. Even though the services are not critical in finding the bug, during testing they might still be used. We have developed model-based mocking (MBM). This is a strategy to test microservice architectures in AMP using on ioco-testing. We show that if the parallel composition of a mock and implementation does not have ambiguous states, the implementations of services can be replaced by a mock based on their specification. The approach was implemented in AMP. Our experiments show that MBM can speed up ioco-testing when targeting individual services, even in cases where the bug could be detected without making use of the mocked services. Thus, MBM provides a faster way to test individual components, guaranteeing that the composition is conformant given that its components are.
For future work, we want to test mocking on additional SUTs to see if our results generalize
to diferent systems. As a consequence of the limitations of composability on input labels of
parallel components, our approach does not allow for broadcast messaging. Alternative notions
of composability, allowing for the same input labels, and mutual acceptance were defined in the
setting for uioco [