The Families to Persons case is a well-known example problem for bidirectional transformations. This paper proposes an implementation of this case within the recently developed Benchmarx framework [2], based on previous conceptual work [1]. Bidirectional transformations (bx) are transformations which may be executed from source to target and vice versa. They are required in a variety of scenarios, including bidirectional data converters, round-trip engineering, or view updates [5]. A wide variety of bx languages and tools have been developed which di er with respect to underlying data models, supported scenarios, incremental behaviour, etc. To compare these approaches, the need for benchmarks has been recognized for long [5]. In response to this need, a conceptual framework for bx benchmarks was proposed in [1]. Only recently, however, the conceptual proposal was materialized into an implemented infrastructure called Benchmarx [2]. The tool transformation case described below exploits this framework to provide for an implementation of a well-known bx case dealing with the transformation between heterogeneous databases. One of these maintains a collection of families and their members; its opposite contains a at collection of persons. Section 2 describes the Families to Persons case. Section 3 provides a survey of the tool architectures supported by the Benchmarx framework. Section 4 explains the test cases against which the solutions are executed. Section 5 sketches the steps to be performed for implementing the benchmark in a speci c tool. Section 6 is devoted to the evaluation of the proposed solutions. Section 7 concludes the paper.
The Families to Persons case is part of the ATL1 transformation zoo2 and was created as part of the \Usine Logicielle" project. The original authors are Freddy Allilaire, Frederic Joault, and Jean Bezivin, all members of the ATLAS research team at the time it was created (December 2006) and published (2007). The variant proposed for the TTC is described below.
We selected this case for several reasons: (1) The underlying data structures are rather simple. Thus, the case can be implemented e.g. in tools operating on trees rather than on graphs. (2) The case is rather small and simple to understand. (3) Furthermore, it may be implemented with acceptable e ort. (4) Finally, it demonstrates a number of problems typically occurring in the transformation between heterogeneous data structures (information loss, attening of hierarchies, di erent representation of the same information, etc.). 0 ..* families
Family name : String 0 ..1 mother 0 ..* sons 0 ..1 father 0 ..* daughters
FamilyMember name : String (a) Families metamodel
PersonRegister 0 ..* persons
Person name : String birthday : Date
Male
Female (b) Persons metamodel
Although solutions to the case need not be implemented in model transformation tools and do not have to live in a speci c technological space such as EMF, we assume EMF models in the following explanation. Thus, the data structures to be manipulated are de ned by Ecore models in Figure 1.
We assume a unique root in each model. A family register stores a collection of families. Each family has members who are distinguished by their roles. The metamodel allows for at most one father and at most one mother as well as an arbitrary number of daughters and sons. A person register maintains a at collections of persons who are either male or female. Note that key properties may be assumed in neither model: There may be multiple families with the same name, name clashes are even allowed within a single family, and there may be multiple persons with the same name and birthday. Furthermore, all collections are assumed to be unordered.
A families model is consistent with a persons model if a bijective mapping between family members and persons can be established such that (i) mothers and daughters (fathers and sons) are paired with females (males), and (ii) the name of every person p is \f:name, m:name", where m is the member (in family f ) paired with p.
After running a transformation in any direction, it is required that the participating models are consistent according the de nition given above. However, this requirement does not su ce to de ne the functionality of transformations in a unique way. Below, we rst consider batch transformations, where the target model is created from scratch.
The functionality of a forward transformation is straightforward: Map each family member to a person of the same gender and compose the person's name from the surname and the rst name; the birthday remains unset. The backward transformation is more involved: A person may be mapped either to a parent or a child, and persons may be grouped into families in di erent ways.
To reduce non-determinism, we introduce two boolean parameters controlling the backward transformation, resulting in a con gurable backward transformation: PREFER CREATING PARENT TO CHILD controls whether a person is mapped to a parent or a child. PREFER EXISTING FAMILY TO NEW controls whether a person is added to an existing family (if available), or a new family is created along with a single family member. If both parameters are set to true, the second parameter takes precedence: If a family is available with a matching surname, but there is no matching family with an unoccupied parent role, the member is inserted into an existing family as a child. Please note that the con guration parameters do not completely resolve non-determinism in the presence of multiple matching candidate families if PREFER EXISTING FAMILY TO NEW is set to true.
In this scenario, information loss occurs in both directions: In forward direction, the distinction between parents and children is lost as well as the aggregation into families. In backward direction, birthdays are lost. Altogether, this case constitutes an example of a round-trip engineering scenario, where both models may be edited and changes have to be propagated back and forth.
For incremental transformations, updates such as insertions, deletions, changes of attribute values, and move operations have to be considered. In forward direction, insertion of a family has no e ect on the target model. Insertion of a member results in insertion of a corresponding person; likewise for deletions. If a family is deleted, all persons corresponding to its members are deleted. If a member is renamed, the corresponding person is renamed accordingly. If a family is renamed, all persons corresponding to family members are renamed. If a member is moved, di erent cases have to be distinguished. If the gender is retained, the corresponding person object is preserved; otherwise, it is deleted, and a new person object with a di erent gender is created whose attributes are copied from the old person object. A local move within a family does not a ect the corresponding person's name; a move to another family results in a potential update of the person's name.
In backward direction, the e ect of an update depends on the values of the con guration parameters, which may be changed dynamically. Please note that the parameter settings must not a ect already established correspondences; rather, they apply only to future updates. Deletion of a person propagates to the corresponding family member. If a person is inserted, it depends on the con guration parameters how insertion propagates to the families model (see above). Persons cannot be moved because the persons model consists of a single, at, and unordered collection. Changes of birthdays do not propagate to the opposite model. If the rst name of a person is changed, the rst name of the corresponding family member is updated accordingly. Finally, if the surname of a person is changed, this change does not a ect the current family and its members: The family preserves its name even if it does not contain other members; thus, the update has no side e ects on the existing family. Rather, the corresponding family member is moved to another family, which may have to be created before the move; the precise update behaviour depends on the parameter settings. 3
The Benchmarx infrastructure has been designed such that it supports a wide range of tool architectures. To
this end, tools are considered as black boxes with respect to the data structures maintained by the tools. As
mentioned earlier, the data structures to be synchronized may be represented in a tool-speci c way; EMF models
merely serve as an example in this paper. Furthermore, the tools may be based on di erent architectures which
are sketched below (see [
source s source´ corrs c corrs´ target t target´
The shaded region in Figure 2 comprises the input data that may be used for update propagation. Tools may rely on correspondences between source and target elements (horizontal arrows). Furthermore, they may assume deltas between old and new states (vertical arrows). A delta may be operational (o-delta, a sequence of operations from the old to the new state) or structural (s-delta, de ned in terms of di ering elements). The output produced by a tool comprises at least a new target state, as well as optional correspondences and deltas.
In [
Selection of a tool architecture constitutes a classical engineering trade-o . On one hand, the preciseness of update propagation grows with the amount of available input data. For example, stored correspondences may be used to improve update propagation when correspondences cannot be inferred from source and target alone (e.g., when there are two family members with the same full name). On the other hand, a tool relying on input data such as deltas and correspondences may be used only in contexts where these data are available. For example, a tool relying on stored correspondences may not be applicable if source and target have been created independently. In this sense, there is no optimal tool architecture.
Figure 3 depicts a feature model for bx tool architectures used to classify the di erent bx approaches. The nodes of the tree are the \features" that a given bx tool architecture can possess. Features can be optional or mandatory, children features can only be chosen together with their parent feature, and children of the same parent can be either exclusive or non-exclusive alternatives. Features are abstract (grey ll) if they can be implied from the set of concrete features (white ll) in a given product. The feature model depicted in Fig. 3 yields exactly the bx tool architectures described above in terms of involved computation steps.
bx tool architecture batch
incremental (alignment-based) state-based (alignment computed internally)
delta/corr-based (external alignment is required and used) delta-based
corr-based operational structural
Legend or alternative (xor) concrete feature abstract feature
As an extension of the feature model for bx tool architectures, Fig. 4 depicts a feature model for benchmarx test cases. Every benchmarx test case must state the required bx tool architecture (cf. Fig. 3), its direction to be fwd (forward), bwd (backward), or both (which means round-trip), the combination of di erent change types applied in the test, and the required update policy to successfully pass the test. The set of possible change types, currently del (deletion), add (addition), move, and attribute (attribute manipulation) can be extended in the future to accommodate more expressive frameworks. Note that a test case can require an update policy that is a mixture of xed, i.e., design time preferences and conventions, and runtime con guration (in the Families to Persons example, only the backward direction needs con guration parameters).
benchmarx test case bx tool architecture direction change type
update policy fwd bwd del add move attribute runtime
fixed
The test cases provided for the Families to Persons case are separated into two di erent categories: (1) batch test cases and (2) incremental test cases. Each category comprises test cases for each transformation direction. Please note that in the following the term forward is used for transformations from the Families model to the Persons model. Accordingly, a backward transformation describes a transformation from the Persons model to the Families model.
While the batch test cases are used to check if a given source model is transformed into a target model correctly, incremental ones also take updates of the corresponding source models into account. This comprises in particular, renaming, deleting and moving family members or persons respectively. Please note that we tried to keep the number of test cases manageable. To this end, only the batch category contains separate test cases for each combination of con guration parameters in backward transformations. In the incremental category, the parameters are changed dynamically within a backward test case.
Legend optional mandatory or alternative (xor) concrete feature abstract feature
While the behaviour is clear for almost all test cases, testIncrementalRenamingDynamic needs some clari cation: In the precondition for this test case, several persons are created, who are then transformed into corresponding families and members in the family model. In an edit delta, a person is renamed (full name is changed), such that the surname matches another existing family. The subsequent propagation to the Families model with parameter set to PREFER EXISTING FAMILY TO NEW should then move the existing family member to the corresponding family and change the rst name accordingly. Please note that afterwards an empty family remains in the family register in case the old family only had one member (c.f., test case testIncrementalRenamingDynamic in class IncrementalBackward). 5 5.1
Since we strive to provide a generic framework for benchmarking BX tools, and not all of the tools provide access to internal data structures like correspondence models or output deltas of a synchronization run, we decided to design each test case as a synchronisation dialogue, always starting from the same agreed upon consistent state, from which a sequence of deltas are propagated. Only the resulting output models are to be directly asserted by comparing them with expected versions.
A benchmarx test case is depicted schematically to the left of Fig. 5, with a concrete test case for our example to the right, following the proposed structure and representing an instantiation with JavaDoc, Java, and JUnit.
Documentation: What is being tested? Expected behaviour? Classification: List of features of the test case TEST CASE NAME/ID:
Initiate Synchronisation
Dialogue Synchronisation Dialogue to Establish Precondition
Assert Precondition Synchronisation Dialogue to Establish Postcondition
Runtime Configuration
Update Propagation /** Test for changing a person's full name (where another person with * the same name exists). * Expected: first name of the corresponding member and their family * name must be changed. As no fitting family exists, a new family must be * created and the member moved to this new family (as the father of this family). * Features: bwd, attribute, corr-based, structural, runtime */ @Test public void testFullNameChangeOfNonUniquePerson() { tool.initiateSynchronisationDialogue(); util.configure().makeDecision(Decisions.PREFER_CREATING_PARENT_TO_CHILD, true)
.makeDecision(Decisions.PREFER_EXISTING_FAMILY_TO_NEW, true); tool.performAndPropagateTargetEdit(helperPerson::createSimpsonsWithTwoBarts); ... util.assertPrecondition("Pre_MemberNameChangeOther", "Pre_PersonNameChangeOther"); //---------------util.configure().makeDecision(Decisions.PREFER_CREATING_PARENT_TO_CHILD, true); tool.performAndPropagateTargetEdit(helperPerson::fullNameChangeOfOtherBart); //---------------11
Every test case should be documented (cf. Label 1 in Figure 5) stating (i) what is being tested, (ii) expected behaviour, and (iii) a list of the concrete features of the test case taken from Fig. 4 to clarify at a glance if a given bx tool can pass the test or not.
Every test starts with an initialisation command invoked on the bx tool under test (Label 2), giving it the chance to establish the agreed upon starting point (e.g., for the Families to Persons benchmarx this comprises a single empty family register and corresponding single empty person register), and create all necessary internal auxiliary data structures.
The next part of a test case (Label 3) is a series of propagation steps, used to establish the precondition of the test. Although this creates a dependency to other tests (asserting exactly this precondition), this simpli es handling the required correspondence model, as the bx tool can build up the necessary internal state that must go along with the precondition. This means that the old consistent corr is \passed" implicitly to the bx tool via a series of preparatory propagation steps. The precondition is nally asserted (Label 4), representing a well-de ned starting point. If the test requires a runtime update policy, this is con gured just before propagating the actual input delta (Label 5). The last part of a test case (Label 6) is an assertion of the postcondition, checking if the nal source and target models are as expected.
In the concrete exemplary test case, a number of persons are created in the person register and then backward propagated to establish a consistent family register that is asserted as a precondition. As part of the actual test, a person named Simpson, Bart is now renamed in the person register; this change is backward propagated with the update policy to prefer creating parents (if possible) to creating children. The interested reader is referred to the actual test cases3 for all further details. 5.2
In its current state, the Benchmarx Framework provides in total 34 pre-de ned test cases (see documentation on Github) for the Families to Persons example. They cover both batch and incremental scenarios for each transformation direction. As stated above, incremental backward tests change the parameter con guration at runtime during test case execution to avoid specifying a xed test scenario for each possible parameter combination. In the batch cases, empty and non-empty families, families with duplicate names and families with duplicate member names are transformed in forward direction. The backward direction for the batch mode checks for each parameter combination the transformation of male or female persons and how duplicate names are handled. The incremental test cases take deletions, insertions, attribute changes (e.g., renaming), moving and a combination of deleting and inserting into account. However, if some cases are missing they may be supplied following the schema of the sample test case as discussed in the previous section. 5.3
In order to integrate a speci c BX tool into our Benchmarx Framework, the interface BXTool needs to be
implemented. Please note, although the Benchmarx Framework itself is written in Java, it is possible to also
use it with non-JVM-based BX tools (c.f., our reference implementation for the tool BiGUL [
Reference implementations for eMo on [
Please note, that we use parameterized test cases, where the BXTools are the parameters. In order to run the test cases for a single tool, the respective test runner has to be instantiated in the corresponding collection in class FamiliesToPersonsTestCase. 6
The goal of our benchmark is a qualitative evaluation of bx tools and the design of the test cases aims at revealing weaknesses or limitations of the chosen approaches. Due to the heterogeneity of bx tools, it is important to be able to easily and quickly distinguish between:
A test that fails because it requires features that the tool does not support, is referred to as a limitation.
A test that fails even though the tool should pass it (based on its classi cation), is referred to as a failure. 3Available from https://github.com/eMoflon/benchmarx 4http://projects.ikv.de/qvt
Initiate Synchronisation Dialogue perform edit on the person model and propagate changes perform edit on the family model and propagate changes perform edit on the person model perform edit on the family model set configurator with parameters compare actual models with expected ones compare actual models with expected ones save current states of source and target models the name of the BXTool A test that the tool should (based on its classi cation) pass and passes, is referred to as an expected pass.
A test that the tool passes even though this is unexpected in general, is referred to as an unexpected pass.
Expected passes con rm that the tool is correctly classi ed and behaves as expected, while unexpected passes indicate that the test case can either be improved, as it is unable to reveal the missing features of the tool, or that the bx tool has not been classi ed correctly.
Scalability of the transformation approaches is evaluated in simple transformations carried out on models while the number of model elements is being raised constantly. Our benchmark measures the time needed for both batch and incremental transformations in forward and backward direction.
Furthermore, we are interested in the metrics [Lines of Code, Number of Words, Number of Characters] of the transformation speci cations. Those numbers should also be supplied in the spreadsheet5.
Submissions to the Families to Persons case for the TTC should be made by cloning the provided SHARE
VM [
An implementation of the transformation including the test runner (c.f., Section 5.3).
A spreadsheet containing the numbers retrieved from running the supplied test cases.
The metrics of the transformation speci cation (see above).
A plot similar to the ones shown in [
In this paper, we propose a well-known example for bidirectional transformations as a case for the transformation
tool contest. We propose to use the benchmarx [