[1..*] authors
AuthoredEntry
BibTeXEntry id : EString TitledEntry title : EString
DatedEntry year : EString BibTeXFile
Still, a more interesting problem is the case when the target model continues to be worked on after the initial transformation. The DocBook document would be given to an editor, which would reorganise sections, add paragraphs in the middle with further commentary and perhaps extend some of the text itself. The task is to ensure in an efficient manner that these manual editions have not impacted the consistency of the DocBook model with the original BibTeX model.
Ideally, the transformation tool or the editing environment would give some infrastructure to tackle this. However, for many tools, the approach seems to be only feasible through the creation of an external consistency checker. This case is to evaluate the current state of the art in out-of-the-box after-the-fact consistency checking. To do so, the case provides a generator which can produce source models of arbitrary size, a repackaged version of the original ATL transformation, and a mutator which can make a number of changes to the model, some of which may impact consistency. These resources were made available on Github1 before the start of STAF 2019.
The rest of the document is structured as follows: Section 2 describes the structure of the live case. Section 3 describes the proposed tasks for this live case. Section 4 mentions the benchmark framework for those solutions that focus on raw performance. Finally, Section 5 mentions an outline of the initial audience-based evaluation across all solutions, and the approach that will be followed to derive additional prizes depending on the attributes targeted by the solutions. 2
The case is intended to review the different approaches for checking after-the-fact inter-model consistency between a BibTeX model and a DocBook model. The process is roughly as follows: 1. The BibTeX model is generated randomly to a certain size, by the included generator in the models/generator.jar JAR. The generator uses a Java port of the Ruby Faker2 library to produce pseudorandom data given a seed. A number of random models (sizes 10, 100, 1000 and 10000) were generated in advance and included in the case resources. 2. The BibTeX model is transformed automatically to DocBook by the repackaged version of the original ATL transformation in the bibtex2docbook.jar JAR. This transformation deals well with models up to 1000 entries, but struggles with larger models due to recomputation of intermediate results.
ModelChangeSet AttributePropertyChange newValue : EString oldValue : EString CompositionListDeletion index : EInt deletedElement : EObject
CompositionListInsertion index : EInt addedElement : EObject
[0..*] changes [0..*] nestedChanges [1..1] sourceChange AttributeChange CompositionChange
ModelChange ChangeTransaction
ElementaryChange affectedElement : EObject feature : EStructuralFeature CompositionMoveIntoProperty newValue : EObject oldValue : EObject
CompositionMoveToList index : EInt movedElement : EObject 3. The DocBook model is edited, in this case with the automated mutator in the models/mutator.jar JAR.
The mutator will operate on a DocBook file, creating a set of folders whose path will start with the specified prefix, adding -N from 1 to nMutants. Each folder will contain the mutated DocBook model, as well as a change model explaining what was done to the model: Figure 3 shows that such models have a ModelChangeSet as the root, with a number of ModelChange instances of various types. The DocBook model will have gone through a number of random mutations according to a seed: if unspecified, the seed will be based on the current system time.
The mutator has a number of predefined mutation operators that will modify the model: Swapping paragraphs: this should break consistency in terms of sorting, but it will not result in missing information.
Swapping sections should not break consistency.
Deleting paragraphs/sections should break consistency.
Appending text to a paragraph should not break consistency.
Adding a new paragraph: unless it happens to match one of the authors, titles, or journals in its Sect1, it should not break consistency.
The mutated models were created in advance before the contest. There are three sets of mutated models from the generated 10/100/1000-entry models: one with a single mutation, one with two mutations, and one with three mutations. 4. A consistency checker would take any combination of the previous artifacts (source BibTeX, fresh DocBook, mutated DocBook, change model) and make a judgment about whether the mutated DocBook is still consistent or not.
If issues are found, it should point to the element in the source model which lacks a proper mapping on the other side, or the element in the target model which is not mapped correctly from the source model (e.g. it is not sorted anymore).
The case resources include a set of expected results from the reference EVL consistency checker. However, the concrete definition of the consistency requirements has proven to be trickier to formalize than expected. This was raised by several case authors, and in fact one of the solutions considered creating a DSL for expressing inter-model consistency.
The optional task was to re-implement or improve the original transformation itself, in a way that lent itself better to after-the-fact consistency checking. A transformation tool may have better support for this, or ATL could be made to deal better with larger versions of this model.
The mandatory task was to check for the consistency of the source BibTeX against the mutated DocBook models in the models directory, and report this information as efficiently and clearly as possible to the user. Ideally, this should be possible without a full re-run of the transformation. To be considered for the performance-related awards, solutions had to use the benchmarking framework in Section 4.
A reference solution based on the Epsilon Validation Language was provided. This implementation did not re-run any transformations: instead, it did a two-way consistency validation by checking from the BibTeX BibTeXFile, Author, BibTeXEntry, TitledEntry, and Article types, and from the DocBook Para types. The implementation required 98 lines of EVL code and 113 lines of Java framework integration code, and does not use the change models at all.
Solutions could focus on efficiency, conciseness, or clarity of presentation to the user. Solutions that can operate straight from the definition of the transformation (i.e. without a separate consistency checker) would be preferred. The call for solutions also invited solution authors to consider other desirable attributes, e.g. verifiability. 4
If focusing on performance, the solution authors had to integrate their solution with the provided benchmark
framework. It is based on the framework in the TTC 2017 Smart Grid case [
3. Run, which found the consistency violations in the mutated DocBook model. 4.1
Solution requirements
Each solution had to print to the standard output a line with the following fields, separated by semicolons (“;”): MutantSet: set of mutants used (“single”, “double” or “triple”).
Source: base name of the input BibTeX model (e.g. “random10.bibtex”).
Mutant: integer starting at 1, identifying the mutant model within this set.
RunIndex: index of the run of this combination of tools and inputs.
PhaseName: name of the phase being run.
MetricName: the name of the metric. It may be the used Memory in bytes, the wall clock Time spent in integer nanoseconds, or the number of consistency Problems found in the mutated DocBook model.
To enable automatic execution by the benchmark framework, solutions were in the form of a subfolder within the solutions folder of the main repository3, with a solution.ini file stating how the solution should be built and how it should be run. As an example, the solution.ini file for the reference solution is shown on Listing 1. In the build section, the default option specifies the command to build and test the solution, and
Listing 1: solution.ini file for the reference Epsilon solution 1 [ build ] 2 default=true 3 skipTests=true 4 5 [run] 6 cmd=JAVA_OPTS=" Xms4g" java jar epsilon.jar the skipTests option specifies the command to build the solution while skipping unit tests. In the run section, the cmd option specifies the command to run the solution.
The repetition of executions as defined in the benchmark configuration was done by the benchmark. For 5 runs, the specified command will be called 5 times, passing any required information (e.g. run index, or input model name) through environment variables. Solutions could not save intermediate data between different runs: each run had to be entirely independent.
The name and absolute path of the input model, the run index and the name of the tool were passed using environment variables Tool, MutantSet, SourcePath, Mutant, MutantPath, and RunIndex. 4.2
Running the benchmark The benchmark framework only required Python 3.3 to be installed. Solutions could use any languages or frameworks, as long as they could run without human input. Since all the performance-oriented solutions this year were compatible with GNU/Linux, it was possible for the case authors to create a Dockerfile with all solutions built in, for the sake of reproducibility. The resulting image is available on Docker Hub4, and it is automatically rebuilt on any push to the repository.
If all prerequisites are fulfilled, the benchmark can be run using Python with the command python scripts/run.py. Additional options can be queried using the option --help. The benchmark framework can be configured through the config/config.json file: this includes the input models to be evaluated (some of which have been excluded by default due to their high cost with the sample solution), the names of the tools to be run, the number of runs per tool+model, and the timeout for each command in milliseconds. 5
How efficient was the approach in time and space (memory)? The reference ATL solution struggled with large models, and the reference solution was not been designed with performance in mind. Was consistency checking directly supported by the transformation approach? Many tools lack this capability, though it might be interesting as an additional execution mode if the target model has seen manual changes since it was generated.
How informative and accessible was the feedback that can be provided by the approach? Authors were invited to make submissions targeting other quality attributes.