In model-driven engineering (MDE), models constitute the main development artifacts. As a consequence, their quality significantly affects the quality of the final product. Thus, adequate techniques are required for ensuring the quality of models. We present a testing framework, comprising a test specification language and an interpreter, for validating the functional correctness of UML activities. For this purpose, we utilize the executability of a subset of UML provided by the fUML standard. As UML activities are employed for different purposes, from high-level process specifications to low-level object manipulations, the proposed testing framework not only allows to validate the correctness in terms of input/output relations, but also supports testing intermediate results, as well as the execution order of activity nodes. First experiments indicate that the proposed testing framework is useful for ensuring the correct behavior of fUML activities.
In model-driven engineering (MDE), models are used to specify the structure and
behavior of the system to be built. By using model transformations and code generation,
artifacts, such as source code, database schema, and deployment scripts, can be
generated from the models. This helps developers to abstract from technical details and
increase their productivity by automating parts of the development process [
One important quality aspect of models is functional correctness. For validating
the functional correctness of models, precisely defined semantics of the used modeling
language is a prerequisite. The semantics of a subset of UML [
Although the semantics of fUML is precisely defined, adequate means for systematically testing fUML models are missing. We argue that a unit testing framework for fUML may provide the same benefits as for code-centric approaches, as it enables to validate the functional correctness of models, helps to avoid regressions, and test cases on model level may serve as input for testing the artifacts generated from the models.
Providing unit testing techniques for fUML activities is a challenging task because they can be specified for different purposes and on different levels of abstraction. Hence, different means for validating their correctness are required. For instance, activities that serve as a high-level specification of processes cannot be tested adequately using assertions on input/output relations; constraints regarding the execution order of process steps modeled by activity nodes seem to be more adequate in such cases. For testing activities specifying low-level object manipulations and computations, assertions on input/output relations might be helpful. In addition, developers may also need to specify assertions on mutable intermediate results.
In this paper, we propose a dedicated test specification language and an accompanying test interpreter enabling the validation of the correct behavior of fUML activities. Using the test specification language the modeler can specify assertions on the execution order of the activity nodes, input and output values, and the runtime state of the model. The test interpreter is based on the reference implementation of the fUML virtual machine, which is used to execute the activities under test and to obtain execution traces that are used for evaluating the assertions defined in the test specification.
The remainder of this paper is structured as follows. In Section 2 we introduce a motivating example used to present our model testing approach throughout this paper. Section 3 gives an overview of fUML and the execution traces used for evaluating test cases on fUML activities. In Section 4 our test specification language for fUML activities and our test interpreter for evaluating them are presented. Related work is addressed in Section 5. Section 6 concludes this paper with an outlook on future work. 2
In this section, we introduce a simple example fUML model specifying the withdrawal functionality of an automatic teller machine (ATM), which serves as running example throughout this paper, and discuss some test cases for validating its correct behavior. From these test cases, we derive the requirements that we aim to address with the proposed testing framework for fUML activities which is presented in Section 4.
An excerpt of the class diagram specifying the structure of the ATM system is depicted in Figure 1. An ATM card (class Card) has a number and a pin and is associated with exactly one account (class Account) which has a unique number and a balance. For realizing the withdrawal functionality, the classes ATM and Account have dedicated operations, called withdraw, validatePin, and reduceBalance.
The activities specifying the behavior of the operations withdraw and reduceBalance of the classes ATM and Account are shown in Figure 2. The activity specified for the operation withdraw (cf. ATMWithdrawActivity in Figure 2) requires as input the client’s card as well as the PIN and the amount of money to be withdrawn entered by the client. First, it checks whether the PIN is valid by calling the operation validatePin. If the PIN is valid (i.e., the operation validatePin provides true as output), the operation reduceBalance is called for the account associated with the provided card. If this operation returns true, the ATMWithdrawActivity provides also true as output indicating the successful withdrawal; otherwise it returns false. The activity specifying the behavior of the operation reduceBalance (cf. AccountReduceBalanceActivity in Figure 2) takes as input the amount of money to be withdrawn and checks whether it exceeds the account’s balance. In case the balance is not exceeded, the balance is accordingly updated and true is returned; otherwise false is returned. Example – class diagram
For validating the correct behavior of the activity ATMWithdrawActivity, which specifies the withdrawal functionality of the ATM, we state the following test cases. Test case 1. If the correct PIN is provided and the amount of money to be withdrawn does not exceed the balance of the client’s account, the balance of the account should be reduced by the withdrawn amount and the activity should return true. Furthermore, the validation of the PIN should happen before the balance is reduced. Test case 2. A withdrawal should also be possible if the amount of money to be withdrawn is equal to the balance of the account.
Test case 3. If the amount of money to be withdrawn exceeds the balance of the account, the balance must remain unchanged and the activity should return false.
A testing framework for fUML activities has to provide the means for expressing and evaluating these and similar test cases. Therefore, it has to fulfill the following requirements.
R1: Execution order. It should be possible to test the chronological order in which activity nodes are executed during the execution of the activity under test. Also other activities that are called from the activity under test should be considered. Furthermore, it should be possible to state the relative execution order of activity nodes without having to state the order of all activity nodes that are expected to be executed. 2 R2: Input / output validation. The testing framework should enable to check whether an input of an activity results in a given output. Further, the same should be possible for activity nodes contained by activities to allow for testing intermediate results. R3: State validation. Assertions regarding the runtime state of the tested model, consisting of objects, their feature values, and links, should be possible for any point in time as well as for time periods of the execution of the activity under test. R4: Test input data. The testing framework should allow to specify input data for the parameters of the activity under test in order to test different execution scenarios.
The fUML standard [
Whereas the standardized fUML virtual machine provides the facilities to execute
activities and retrieve the output values for their parameters, it lacks in providing means
for analyzing the performed model execution. To address this shortcoming, we extended
the reference implementation of the virtual machine1 in previous work [
In summary, the trace of an executed activity enables to reason about the execution order of activities and activity nodes, inputs and outputs, and the runtime state of the executed model at a specific point in time of the execution. It therefore builds the crucial basis for the proposed testing framework for fUML activities presented in the following.
In this section we first present a dedicated test specification language for expressing test cases on fUML activities and illustrate its usage on the ATM example introduced
Trace model in Section 2. Subsequently, we present a test interpreter capable of evaluating test cases as well as the result from evaluating the ATM test cases.
We provide an implementation of our testing framework integrated with the Eclipse
Modeling Framework (EMF) [
checked. The temporal quantifier always denotes that all snapshots captured before or after the stated node should be checked, whereas the quantifierexactly denotes that only the single snapshot captured directly before or after the execution of the stated node should be checked. The properties of the object that shall be validated are defined in the state expression of a state assertion, which can either be an object state expression for validating the state of a complete object (i.e., all its properties), or it can be a property state expression for validating the value of a single property of an object.
Listing 1 shows the specification of the test cases for the ATM system defined in Section 2 using the proposed test specification language.
Listing 1: Test suite for the ATM system
After specifying the fUML model which we want to test using an import declaration (line 1), we define a test scenario (line 2–13) composed of oneATM object, one Account object, one Card object, and one link between the Account and the Card object.
The first test case (line 14–26) tests the activityATMWithdrawActivity and provides as input the Card object defined in the test scenario and the Integer values 1234 and 300 for the parameters card, pin, and amount, respectively. In this test case, we first declare a variable account (line 16), which refers to the object provided as output by the action readAccount through its output pin result. This variable can now be used in state assertions. In line 18, an execution order assertion is specified defining that the action call validatePin should be executed before call reduceBalance, whereas accepting the execution of any other activity node before, in between, and after these nodes using *. The state assertion in line 19–21 checks that before the operation reduceBalance is called, the balance of the account is always 800. The state assertion in line 22–25 checks that after the execution of the last activity node sucessTrue the balance of the account should always be 500 (i.e., it is and remains updated to 500 accordingly). Furthermore, this state assertion defines that the output (parametersuccessful) should be true.
The second test case (line 27–33) implements, similar to testcase1, the assertions regarding the correct behavior for a withdrawal of 800; hence, it is expected that the account’s balance is updated to 0.
The third test case (line 34–40) tests the behavior of the activity ATMWithdrawActivity for the case that the amount to be withdrawn from the client’s bank account (900) exceeds the account’s balance (800). Accordingly, we assert that the balance of the client’s account is not modified and that the output of the activity isfalse. 4.2
Test Interpreter For executing and evaluating test cases on fUML activities specified in the presented test specification language we make use of the fUML virtual machine, as well as of execution traces obtained from executing the activities under test (cf. Section 3).
The process of executing and evaluating tests is shown in the Figure 4. The input provided to the test interpreter consists of the fUML model to be tested and the test suite. Each test case in the test suite is evaluated by executing the activity under test using the fUML virtual machine with the parameter values defined in the test case. From this execution, we obtain an execution trace, which is used to evaluate each assertion of the test case. Finally, a test report is generated which provides the test verdict.
To evaluate execution order assertions, we simply investigate the ActivityNodeExecution instances contained in the execution trace, which represent the executed activity nodes, as well as the links between them defined for the referenceschronologicalPredecessor and chronologicalSuccessor.
These snapshots maintain all different versions of the object that existed during the entireTeexesctuetixoencouftitohen afcrtaivmityewunodrekr test. In fUML, objects can only be modified by certain kinds of actions, which all provide the modified object as output. Consequently, in the execution trace, an ActionExecution instance that represents the execution of such an action also refers to the ValueSnapshot instance representing the modified object fUML Model Classes Activities
Test Suite Test Cases test t1 activity a1 { var v = act1.result }assaecrtt5St{atve::bxef=or3e00 }
Test
Scenarios
Model Execution
Execution Trace
Test
Evaluation as output (cf. Output, OutputValue). Thus, to evaluate a state assertion, we obtain the ValueSnapshot instances of the ValueInstance representing the object of interest, which are referenced as outputs by the ActionExecution instances that have been executed in the time period specified in the state assertion (e.g.,always after actionX). Whether the modification took place within the respective time period can be easily derived from the chronologicalPredecessor/Successor of the activity node defined in the state assertion. The resulting set of snapshots is then checked concerning the specified condition.
When executing the test suite defined for the ATM system in Listing 1, the test interpreter reports failures for all test cases.
For test case 1, a failure is reported for the state assertion defined in line 24–27 because the account’s balance was updated to -500. The bug causing this failure resides in the activity AccountReduceBalanceActivity: the incoming edges of the input pins of the action call subtract have to be switched to calculate the account’s new balance.
When evaluating test case 2, the execution order assertion, as well as the second state assertion (validating that the balance was set to 0) fail, because the decision node of the activity AccountReduceBalanceActivity defines that a withdrawal is only possible if the amount of money to be withdrawn is smaller than the account’s balance (amount < account::balance). However, according to the test case, the withdrawal should also be possible if the amount is equal to the account’s balance (amount ≤ account::balance).
Test case 3 fails because no output value was provided for the output parameter successful of the ATMWithdrawActivity. The bug leading to this failure was introduced at the action successFalse of the activity ATMWithdrawActivity. This action has two incoming control flow edges, whereas only one of them can provide a control token but never both, which is, however, required to execute this action.
In summary, our testing framework fulfills the requirements identified in Section 2. It enables to assert the execution order of activity nodes (requirement R1), to evaluate the input and output of activities and actions (R2), as well as the runtime state of the tested model at a specific point in time of the executing (R3), and it enables to define test input data for testing different execution scenarios (R4).
Gogolla et al. [
Dinh-Trong et al. [
Pilskalns et al. [
The described approaches differ from our testing framework in the following respects. (i) For defining behavior in UML models, the presented approaches use their own formalisms. Thus, the execution semantics they apply is different from fUML’s semantics. However, they are not restricted to the fUML subset. (ii) The presented approaches only evaluate invariants and pre- and post-conditions defined within the UML model for specified scenarios, whereas our approach enables the specification of arbitrary test cases that are separated from the UML model. (iii) The presented approaches only provide the possibility to validate changes on the system state caused by the execution of a whole operation, while our testing framework enables to also validate the state changes caused by distinct actions contained by the activity defining the operation’s behavior. Furthermore, no validation of the execution order of operations or even actions is possible in these approaches.
Regarding the specification of test cases, the UML testing profile (UTP) [
We presented ongoing research towards developing a testing framework for validating the correct behavior of UML activities based on fUML. Our testing framework provides a dedicated test specification language for specifying the expected behavior of fUML activities in a set of test cases as well as a test interpreter which enables to evaluate the test cases by executing the activities under test and analyzing their actual behavior by utilizing execution traces.
First experiments with the proposed testing framework indicate its usefulness for ensuring the correct behavior of fUML models. In-depth case studies are necessary in future work in order to confirm this first impression. The experiments revealed the following interesting extensions of our testing framework left for future work. Parallelism. fUML activities provide modeling concepts for specifying concurrent execution flows (e.g., fork nodes). The current implementation of our testing framework only supports the evaluation of test cases for one possible sequential execution order of concurrent flows and not all possible sequential execution orders.
Object-centric testing. Another possible extension of our framework is to support
object-centric testing. User should be able to specify state validation expressions
without referring to the activity under test directly, but by specifying how the state of the
system changes during the execution. This would enable to express test cases on the
expected behavior of a fUML activity, without coupling the test case with the activity.
In this respect OCL might be a valuable extension of our testing framework.
Corrective feedback. Our testing framework provides as output the information
regarding the success or failure of each assertion. An interesting line of future work is
to investigate techniques for slicing fUML models (e.g., [
Model-based testing. Another possible future research direction is to apply
modelbased testing approaches to generate test cases for fUML activities based on coverage
criteria. We are currently investigating the approach proposed by Holzer et al. [