Model validation in model-driven development gains in importance as the systems grow in size and complexity. In this situation an efficiency of validation execution and an immediate feedback whether a recent manual edit operation broke a validation rule is desirable. To increase efficiency, incremental model validation tries to minimize the proportions of the model that have to be rechecked by reusing previous validation results. As a benchmark for efficiency of validation tools, the Train Benchmark Case at the Transformation Tool Contest 2015 was created. In this paper, we present a solution using NMF Expressions, a tool for incremental evaluation of arbitrary expressions on the .NET platform.
In this paper, we evaluate the efficiency of incremental validation with NMF Expressions. The rest of this paper is structured as follows: Section 2 gives a very short introduction to NMF Expressions and Section 3 explains our solution. Finally Section 5 summarizes the paper. 2
The goal of NMF Expressions is to give developers an automated tool at hand providing them with advantages of incremental evaluation for arbitrary expressions. Unlike many other approaches, our approach works implicitly, so developers only have to specify their expressions and NMF Expressions takes care of how to turn this into an algorithm that will evaluate the expression in an incremental fashion. On the other hand, the traditional batch mode specification is still available so that NMF Expressions yields a choice whether to run a given expression incrementally or in batch mode.
In the incremental mode, the approach creates a dynamic dependency graph from a given expression and observes changes. These changes originate from elementary update notifications and are propagated through the dependency graph. Operating on the .NET platform, NMF Expressions uses the industry standard INotifyPropertyChanged and INotifyCollectionChanged interfaces to record elementary changes. These are also required by a lot of other tools including the modern UI libraries on the .NET platform. As a consequence of the theoretical foundation using monads, the dependency graph contains specialized nodes for optimized incrementalization of queries.
While NMF Expressions works with arbitrary model representations implementing the interfaces for elementary change propagation, we use the model representation of NMF. That is, we transformed the given Ecore metamodel of the railway domain into an NMeta metamodel and generated model representation code. NMF thus offers us a deserialization mechanism to load the resulting models as objects into memory. 3
The intended usage of NMF Expressions in incremental mode is that users would modify the model in some editor through a sequence of change operations, each of which providing elementary change notifications. Then, NMF Expressions would use the elementary change notifications and combine them to provide immediate feedback whether the most recent model manipulation has caused some validation rule to fail for some model elements. Currently, NMF Expressions always minimizes the model elements that it has to look at, even at the cost of high memory usage. However, in the Train Benchmark, the only model manipulations we can see are the repair operations, so for us the benchmark does not really reflect the situation for which we have designed NMF Expressions.
In incremental mode, NMF Expressions creates a cache for the selected expressions and maintains this cache. This maintenance happens automatically as NMF Expressions adds computational effort to the (in-memory) online model manipulation. In this case solution, we created expressions for the validation patterns so NMF Expressions caches the invalid elements continuously. However, this means that the phases drawn from the case description get blurred. In particular, the check phases get meaningless as the updated results are always available and could be used for immediate feedback, while more computational effort is put to the model manipulation such as the modify operations.
Because NMF Expressions allows to use the same specification both in a classic batch manner as also incrementally, our solution can also be configured to run in batch mode without any changes to the Georg Hinkel and Lucia Happe
patterns. When executed in batch mode, NMF Expressions simply forwards the call to the LINQ to objects implementation. Besides a negligible runtime compilation effort, this utilizes the highly optimized platform LINQ implementation.
The patterns are enumerable expressions where developers can choose at runtime whether the pattern should be executed in batch mode or whether NMF Expressions should register for elementary change notifications to keep a cache of the result up to date. To specify patterns, we created a small method Fix that captures them. The easiest implementation for the Fix function repairing any validation error as soon as they occur would be the one presented in Listing 1. In Line 2, we tell NMF Expressions that we want to obtain incremental updates for the given pattern. Line 3 repairs all occurences existing so far and Lines 4-8 handle new pattern matches. For the benchmark, we adopted the Fix function to account for the benchmark phases. In particular, the implemented version takes a third parameter to allow us to sort matches. Since these sort keys offer little insight, we omit them in the pattern presentation.
In the following we will present the solution to the tasks, following the structure of the case description, though with omitted sort keys.
Please note that the parameter names such as pattern or action are optional, we only included them for better understandability.
The solutions to SwitchSet, RouteSensor and SemaphoreNeighbor use the query syntax of C#. This syntax is translated to the method chaining syntax by mapping the query keywords like from or where to method calls of NMF Expressions. Such query expressions are commonality on the .NET platform and thus easy to write and understand by most developers.
Note that the order in which the statements occur does make a difference. In particular, e.g. lines 2 and 3 of the SwitchSet solution could logically be interchanged but cause a slightly different implementation. NMF Expressions currently does not optimize the query for performance.
In the solution for SemaphoreNeighbor we can observe that NMF Expressions is not able to inverse directed references. We argue that such inversion is always limited to a particular scope, which is unclear from the context. If the context was clear, the reference should have been navigable in both directions in the metamodel. As this is not the case, we have to cross join the two respective routes and filter them on the semaphores. 4
To evaluate our solution, we ran it in comparison to the reference implementations in Java and
EMFIncQuery [
In all four presented queries, both versions are up to multiple magnitudes faster than the plain Java solution. In medium-sized models, the incremental version also beats EMF-IncQuery, in the SwitchSet pattern it is even faster on the larger models. 5
In this paper, we presented an NMF solution to the Train Benchmark case at the TTC 2015.
The queries and repair transformations demonstrate why we have sticked to the C# language. We think that it is very hard to get a more concise textual solution for this case. At the same time, developers get the full tool support from e.g. Visual Studio and the query syntax that we use is used by thousands of developers already and widely understood.
The performance figures shows that the incremental version of our solution outperforms the batch mode execution of the same solution in all cases. At the same time, our solution yields a batch mode execution in cases where only a single analysis run is needed. This version outperforms the classic Java solution in several orders of magnitude. On the other hand, for large models our incremental solution ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● T0.00.11 ● 1 10000 )1000 sm100 (e 10 im 1 T PosLength, Function: repair+recheck SwitchSensor, Function: repair+recheck 1 2 4 8 16 32 64 128 256 512 1024
Size Tool ● EMFIncQuery Java NMF(Batch) NMF(Incremental) 4 8 16 32 64 128 256 512 1024
Size
Tool ● EMFIncQuery Java NMF(Batch) NMF(Incremental) SwitchSet, Function: repair+recheck RouteSensor, Function: repair+recheck ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2 4 8 16 32 64 128 256 512 1024
Size Tool ● EMFIncQuery Java NMF(Batch) NMF(Incremental) 1 4 8 16 32 64 128 256 512 1024
Size Tool ● EMFIncQuery Java NMF(Batch) NMF(Incremental) cannot keep up with the EMF-IncQuery solution, except for one pattern where the NMF solution is slightly faster.
The biggest advantage of our solution is that it gives both a batch mode solution and an incremental solution our of the same pattern specifications. Thus, the same analysis code can be used in the case setting where incrementality is a clear advantage, or in a batch mode, e.g. when memory is a sparse resource or the analysis results are only required once.