=Paper=
{{Paper
|id=Vol-3089/ttc21_paper_labflow_Hinkel_case
|storemode=property
|title=Incremental recompilation of laboratory workflows
|pdfUrl=https://ceur-ws.org/Vol-3089/ttc21_paper6_labflow_Hinkel_case.pdf
|volume=Vol-3089
|authors=Georg Hinkel
|dblpUrl=https://dblp.org/rec/conf/ttc/Hinkel21a
}}
==Incremental recompilation of laboratory workflows==
https://ceur-ws.org/Vol-3089/ttc21_paper6_labflow_Hinkel_case.pdf
Incremental recompilation of laboratory workflows
Georg Hinkel1
1
Tecan Software Competence Center GmbH, Peter-Sander-Straße 41a, 55252 Wiesbaden, Germany
Abstract
When executing high-level process models, exceptions sometimes need to be handled by changing the planned process on
the fly, leaving the jobs that have already been executed intact. Therefore, transformations that transform such a high-level
process model to low-level jobs need to be incremental and respect low-level job elements that represent actions that have
already happened. As an example, we consider the automation of laboratory workflows where failures of low-level jobs may
need to get compensated by taking out samples from the automated process. Based on this scenario, we present a benchmark
for transformation tools to deal with such kind of problems.
Keywords
Incremental, Model transformation, Laboratory, Workflows
1. Introduction
When the pandemic situation due to Covid-19 started in
spring 2020, the availability of test capacities was a huge
problem and is still a critical problem when the paper
was written. As one of the reasons, this is due to the
fact that workflows executed in laboratories are usually
automated in an inflexible manner or not automated at
all. In the latter case, this was often not due to the fact
that laboratories were lacking instruments capable of
automating a workflow such as Covid-19 tests, but rather
to the fact that it was hard to repurpose these instruments
for Covid-19 tests, besides to verification and validation
efforts also due to the lack of flexibility of the control
software for such instruments.
These instruments are often robotic liquid handlers
(RLHs) equipped with some additional devices such as Figure 1: A Tecan Fluent instrument
thermocyclers, shakers or readers. These RLHs come
either tied and optimized for a set of specific applica-
tions (especially clinical applications) or as flexible in-
struments that can solve various tasks [1]. An example of abstraction provided to the user, which usually is a
of the latter is the Fluent instrument from Tecan1 shown chemist or biologist but not typically not a computer sci-
in Figure 1. These instruments currently offer the end- entist, is often rather low. This makes it hardly accessible,
user a domain-specific language to create and manage unless users get a substantial amount of training [2]. Fur-
programs that can be executed by the robot, in the case thermore, it is very hard for these low-level descriptions
of Fluent called scripts. This domain-specific language is to cope with multiple scales, such as multiple supported
modular, such that it can be extended by certain elements, amounts of samples processed by the robot at a time.
e.g. to support the integration of third-party devices that To overcome this problem, there are several approaches
can perform subtasks such as shaking or heating a plate. to raise the level of abstraction in which the laboratory
The general problem with scripting is that the level workflow is formulated and how it is transformed to
low-level jobs executable by the robot.
TTC’21: Transformation Tool Contest, Part of the Software When following such an approach, one quickly runs
Technologies: Applications and Foundations (STAF) federated into problems that when exceptions occur, e.g. consum-
conferences, Eds. A. Boronat, A. García-Domínguez, and G. Hinkel, ables or reagents run empty during the process. Excep-
25 June 2021, Bergen, Norway (online). tions usually occur at the lower levels of abstraction such
" georg.hinkel@tecan.com (G. Hinkel)
© 2021 Copyright for this paper by its authors. Use permitted under Creative as e.g., the hardware, and it is up to the automation
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
CEUR Workshop Proceedings (CEUR-WS.org)
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
system to translate this exception into the high-level
1
https://lifesciences.tecan.com/ workflow description by tracing back into the high-level
fluent-laboratory-automation-workstation
description of the process. is important to keep a reference between the high-level
RLHs are expected to recover from errors whenever process that is executed and the low-level jobs, but this
possible, e.g. by asking the user to refill the consumables correspondence is not a strict 1:1 relationship. A high-
and continue. Depending on the process, there may be level protocol step can be traced to multiple low-level
implications on the process, for example because time jobs, but a single low-level job represents the processing
constraints can no longer be met. In these situations, of only a selection of samples. Previous benchmarks for
multiple options exist, depending on both the error and incremental change propagation (cf. Section 6) have con-
its context. If viable, a solution can be to just retry or to centrated either on cases where there is no trace between
flag the sample that time constraints could not be met. input models and output or the correspondence is always
Depending on the length of the pause and the process, strictly 1:1. Hence, it is not clear for a practitioner how
this can already mean that the results for these samples such non-trivial correspondences are supported in exist-
are not usable, e.g. because reagents evaporated while ing incremental model transformation tooling. Further-
the users was refilling consumables. In such cases, one more, it is not clear which incremental transformation
would want to simply skip the processing for the affected approaches are capable of specifying such constraints, i.e.
samples in order to save consumables and reagents or that change propagation of some elements is restricted
one wants to abort processing for the samples not yet to some specific states. Therefore, this paper presents
processed in order to save the sample. Another exam- a transformation challenge and benchmark how such
ple are redundant devices that run into failures and one change propagation systems can be specified and exe-
would like to use a different device instead. cuted in an incremental manner.
Therefore, the automation system needs to do two The remainder of this paper presents a tool challenge
things, one is to identify the required changes to the to solve a minimized version of this problem. At first, we
high-level process and the second one is to propagate describe the challenges that we see in this case in Section
these changes to low-level elements. However, since the 2. Section 3 then describes the case itself. Section 4 in-
error occurred while the workflow was executing, some troduces the benchmark framework and Section 5 briefly
elements of the low-level model cannot be changed any- explains the reference solution and its deficits. Section 6
more since represent actions performed in the past. It is finally sums up the differences to previous benchmarks.
impossible to undo what the instrument has already done,
but it is possible to change what the instrument shall be
doing next. Because time constraints are in operation 2. Challenges
while these changes are propagated, it is also important
In particular, the following aspects of the problem are
that the replanning happens fast.
especially important:
Given the recent advances in incremental model trans-
formation and analysis [3, 4, 5, 6, 7, 8], we believe a viable • Because the transformation from high-level pro-
solution approach would to design the transformation cess models to low-level execution jobs happens
from the high-level process model to low-level execu- at runtime, the performance of this transforma-
tion jobs as an incremental transformation. Ideally, the tion is important. However, it is sufficient when
incrementalization is done implicitly, i.e. without the the jobs are clear when they have to be executed,
transformation developer having to adapt the transfor- i.e. it does not matter if the planning is not fin-
mation, to support a wide range of changes in the process ished for the entire jobs as long as it is clear what
model. When the transformation would create jobs, it the instrument shall be doing next. How long
would create them with an initial execution state and does it take for the transformation to return the
then restrict the change propagation to those elements first low-level element?
that are still in initial state.
• If a job fails, one needs to identify how the process
Another way to look at the problem is to understand it
model needs to be adapted. How to specify this in
as a consistency problem: For each sample and for each
an understandable way?
part of the input process, there must be a corresponding
• Given a change of the process model, one needs
low-level action element. When such a low-level element
to adjust the jobs that need to be performed as
fails, the corresponding sample would fail (if there is no
quick as possible. How long does it take?
alternative recovery). Setting the sample to a failed state
would let the consistency mechanism remove all the low- • While propagating changes to the low-level jobs,
level elements that have not been executed yet, ideally it is tremendously important that the jobs that
reporting those that have been executed as inconsisten- have been executed already stay in place. How
cies (because they would not have been necessary given to make sure that the change propagation does not
that the processing of the corresponding sample failed). affect elements representing actions that have hap-
However, in this domain the maintenance of a trace pened in the past?
Figure 2: Minimized metamodel of laboratory workflows.
• The support for change propagation should not the specification of what an arm should do and we also
degrade the understandability of the transforma- removed the scheduling problem. Therefore, the trans-
tion from the abstract process model to the low- formation is limited to calculate what the RLH has to do,
level. What changes are necessary to enable an but not when or where.
incremental change propagation? In the remainder of this section, we present these two
metamodels in Sections 3.1 and 3.2. Then, we discuss
We ask all authors of solutions to comment on how the transformation rules in Section 3.3 and the change
their tool copes with the challenges described above. propagation rules in Section 3.4.
Similar challenges occur in domains such as smart
production where one needs to plan steps according to
3.1. The high-level laboratory process
a general production protocol. Therefore, we aim to
address the following general research questions with metamodel
this benchmark: The minimized metamodel of laboratory workflows that
we use for the benchmark is shown in Figure 2. The cen-
• Are non-trivial (i.e. not 1:1) correspondences sup- tral element is a JobRequest that represents to process a
ported by recent incremental model transforma- range of samples using a given Assay. This assay element
tion approaches? represents the actual process. Meanwhile true laboratory
• Is it (still) possible to implicitly derive an incre- workflows often consist of a vast variety of operations,
mental change propagation in the presence of the benchmark only considers four possible process steps,
non-trivial correspondences? namely distributing sample, adding a reagent, washing
• If not possible implicitly, what is the required or incubating. Each protocol step has an id for the sake
development effort to implement change propa- of identifying it in the remainder. Further, protocol steps
gation in the presence of non-trivial correspon- carry links to their previous and next steps in order to
dences? ease analysis. Meanwhile most workflows require the
• How does the presence of non-trivial correspon- replication of samples and also need to include standards
dences affect performance of incremental model such as positive and negative controls, this is also omitted
transformations? in this simplified model.
Samples are identified by a sample id, which is usually
the barcode on the sample. Assays have a name assigned
to them. In Figure 2, they are contained in a JobRequest
3. Case description to make it easier to work with them, though in practice
The benchmark uses two metamodels. The first is a high- they usually stand for themselves.
level description of laboratory workflows, the second
one is a low-level description of liquid handling jobs. In
the scope of the benchmark, we omit all layers below
Figure 3: Minimized metamodel of liquid handling jobs
3.2. The low-level job metamodel is, it is allowed to pipette cavity indices 0 and 1 inside the
same element, but not 0 and 8. Liquid can be transferred
The low-level metamodel has a viewpoint that is more
from and to any kind of labware.
concerned with the actual execution of the workflow on
We assume here that a separate device is used for wash-
a RLH. Here, the execution of the entire process is repre-
ing that only works with microplates, but washes cavities
sented by a JobCollection that consist of a series of Job
separately. Therefore, a Wash element carries a list of all
elements and a list of labware that is needed. Labware
cavity indices that should be washed and a reference to
means in this context the consumables needed to execute
the plate that should be washed.
a workflow. The jobs could be LiquidTransfer, Wash
Similarly, an Incubate steps works only for mi-
or Incubate elements. For laboratory automation, one
croplates, but it always incubates an entire plate, because
often uses microplates for processing, essentially plates
it is physically difficult to heat only parts of it.
with a large number of wells (‘holes’) that are used as
small tubes (see Figure 4). Microplates have a standard-
ized size, but multiple formats with different amounts 3.3. Transformation rules
of wells exist. The probably most common format is a The transformation from the high-level process model
microplate with 96 wells. Tube runners are racks that to the low-level job model needs to follow these require-
can host multiple tubes and allow easy loading of an in- ments:
strument with new tubes. Troughs are simple containers
with a relatively large volume (usually 100ml or more) • For each JobRequest, a JobCollection should
that are used to host reagents. be created.
Each LiquidTransfer element represents the execu- • Samples are processed on microplates but come
tion of a liquid transfer command in the RLH. Because in tubes stored on racks with 16 tubes called tube
RLHs are usually equipped with multiple pipettes, the runners. Therefore, for each Sample, a cavity of
RLH can pipette multiple cavities of a microplate or tube a TubeRunner should be assigned. Further, the
rack at once. Therefore, a LiquidTransfer element has transformation needs to store a mapping between
up to eight TipLiquidTransfer elements that specify a Sample and a combination of a Microplate and
which wells the individual tips of the transfer operation a cavity index. The combination of microplate
should target. It is important that the pipettes of a pipet- and cavity needs to be unique. At most 96 cavi-
tor usually share the X axis, which means that all source ties of a microplate can be used. For tracing, the
cavity indices of TipLiquidTransfer elements inside a microplates should be simply numbered with a
LiquidTransfer elements may only differ by their re- common prefix. The first microplate should be
mainder in the division by 8. Similarly, all target cavity named Microplate1 and so on and so forth.
indices must be the same except for the modulo 8. That
• We assume that reagents are stored in troughs. 3.4. Change propagation rules
Therefore, for each Reagent element, there
As denoted earlier, there are two change propagations in
should be a Trough element with the name set
the benchmark. One is to change the high-level model
to the name of the reagent.
as reaction to low-level changes and the other change
• For each combination of ProtocolStep and Sam-
propagation is in the direction vice versa.
ple that is not in the Error state, a Job needs to
The first change propagation will be very simple in
be added to the JobCollection corresponding
the scope of this benchmark. Whenever a Job is failed,
to the parent JobRequest of the parent Assay as
all corresponding samples have to change their state to
defined per the following rules:
failed as well. If a LiquidTransferJob fails, only those
– For each DistributeSample element, samples using the failed TipLiquidTransfer elements
there must be a TipLiquidTransfer that have to be set to failed.
represents the transfer of the sample from The more interesting change propagation is the change
cavity 0 of the tube containing the sam- propagation in the opposite direction. In the scope of
ple to the cavity of the assigned mi- the benchmark, we only consider changes to the state of
croplate. Multiple TipLiquidTransfer el- samples. If a sample changes its state to failed (e.g. as
ements may share the same parent Liquid- a consequence of a change propagation from the lower
Transfer element. level), all jobs created for this sample need to be removed,
– For each AddReagent element, there provided that they are still only planned and no other
must be a TipLiquidTransfer that repre- samples are affected. That is, a WashJob or IncubateJob
sents the transfer of reagent to the cavity may only be removed if all samples on the corresponding
with the sample, i.e. from cavity 0 of the plate are failed.
trough created for the reagent to the cav-
ity and microplate assigned to the sample.
Again, multiple TipLiquidTransfer ele- 4. Benchmark
ments may share a parent LiquidTrans-
We provide a benchmark framework that can automati-
fer element if the conditions regarding the
cally compile, run and check solutions and generate dia-
cavity indices are met.
grams to analyze the results. The benchmark framework
– For each Wash, a WashJob should be cre- is based on the Train Benchmark [9]. The framework as
ated that washes the microplate assigned well as metamodels, input models and the reference so-
to the sample and at least the cavity in- lution is publicly available online at https://github.com/
dex of the sample. However, the same tecan/ttc21incrementalLabWorkflows.
WashJob may be reused for multiple sam- In the remainder of this section, we first describe the
ples provided that the transformation as- phases of the benchmark in Section 4.1, then explain
signs the samples to the same microplate. how to run it in Section 4.2. Next, we introduce the input
– For each Incubate, an IncubateJob with models in Section 4.3. Section 4.4 explains the correctness
the same temperature and duration should checks and Section 4.5 introduces the evaluation criteria
be created that incubates the microplate for solutions. Section 6 then explains the procedure to
assigned to the sample. The same Incu- add a solution to the benchmark.
bateJob must be reused for the incubation
of the same microplate, provided that the
transformation assigns them to the same 4.1. Phases
microplate. The benchmark is divided into the following phases:
– Each created job must have the name of the
1. Initialization: Loading the transformation and
protocol step from which it was created.
metamodels
– Each created job must reference jobs cre-
2. Load: Loading the input models
ated for the previous protocol step for the
3. Initial: Creating the initial JobCollection
same samples such that the job is sched-
uled afterwards. 4. Update: Reading changes to job executions and
propagate
• The transformation should ideally produce the
minimum amount of elements, i.e. the Liquid- The last step is performed repeatedly 20 times. The
TransferJob, WashJob and IncubateJob ele- benchmark seeks to compare execution times for all of
ments should be shared where this is possible. the phases. For the last step, any efforts necessary for
parsing and loading the changes can be excluded from
the time measurements.
4.2. Running the benchmark
The benchmark framework only requires Python 2.7 or
above and R to be installed. R is required to create dia-
grams for the benchmark results. Furthermore, the so-
lutions may imply additional frameworks. We would
ask solution authors to explicitly note dependencies to
additional frameworks necessary to run their solutions.
If all prerequisites are fulfilled, the benchmark can
be run using Python with the command python
scripts/run.py. Additional command-line options
can be queried using the option –help.
1 {
2 "Tools": ["Reference"],
3 "Scenarios": [ Figure 4: Example result of an Elisa process
4 {
5 "Name": "test",
6 "Models": ["minimal"]
7 }],
8 "Sequences": 20,
9 "Runs": 5,
In the scope of the benchmark, we always use the
10 "Timeout": 6000 same assay, but vary the number of samples or work
11 }
with duplicates of the assay. That is, the benchmark has
Listing 1: A minimal benchmark configuration a sequence of models with increasing number of samples
but the same assay or an increasing number of steps in
The benchmark framework can be configured using the assay (by replicating the steps) and a constant number
JSON configuration files. A minimal test configuration is of samples. There are multiple change sequences that
depicted in Listing 1. When creating a new solution, we scale the assay or the number of samples, with or without
highly recommend to overwrite the contents of this con- new samples on the fly.
figuration file locally. In the configuration from Listing To specify changes in the low-level job model, we use
1, only the test scenario with just one minimal model is text files that specify these jobs in terms of the high-level
executed, using only the solution in Reference 5 times. model. Each line of these text files corresponds to one
state change of one job. The lines are in the format
__ where
4.3. Input Models state is either S for success or F for a failure (we skip
As inputs, we use models of a very abstract Elisa (Enzyme- the executing state). These status lines have been
linked Immunosorbent Assay) workflows. These work- generated independently from an actual solution and
flows are generally used to detect antibodies, for example thus, they include also status changes of jobs for samples
to proof that a patient has experienced a Covid-19 infec- that would have to be set to a failed state due to a
tion or has been vaccinated. For this, we assume that previous failure. In case of a liquid transfer, there are
we have access to microplates that are already prepared 96 states, indicating the success state for each cavity,
with Covid antigens at the bottom of the wells. Then, we independently of whether this cavity is used or not.
add the sample. After an incubation period, antibodies However, changes may also affect the samples, i.e. new
against Covid-19 in the sample, if any, will bind to the samples may be added dynamically. This is indicated by
antigens at the bottom of the well. Then, the plate is a line in the format NewSample_.
washed (in order to get rid of other antibodies) and a As stated above, the time needed to calculate the job el-
conjugate is added. In a second incubation step, special ements that are actually affected by these changes should
marked antibodies bind to the Covid-19 antibodies, if any. be excluded from the time measurements, but the actual
Afterwards, the plate is washed again and a substrate is change propagation should be included.
added that reacts with the marked antibodies and, after The high-level process models are available in EMF
another incubation period, results in a color reaction. format, but can be made available in other formats as
The result is a plate where some cavities are colored well, upon request.
while others are not, as illustrated in Figure 4. Such
a plate can be read by an absorbance reader, which the 4.4. Correctness checks
limited process model does not support. In such an image,
the color correlates with the amount of antibodies present The benchmark framework performs the following cor-
in the sample. rectness checks after the initial transformation and after
each update:
• It checks that only the step names of input model 1 from sample in samples
2 let location = locationRepository.LocateSampleProcessing(
elements are used. sample )
• It checks that no liquid transfer is made into the 3 group (sample, location) by (location.Plate, location.
same cavity of the same labware and the same 4 select Cavity / 8) into transferChunk
TraceAll( AddTips( new LiquidTransferJob
protocol name. 5 {
ProtocolStepName = Id,
• It checks that no cavity index greater or equal to 67 Source = locationRepository.LocateReagent( Reagent ),
96 is used. 8 Target = transferChunk.Key.Plate
9 }, transferChunk.Select(l => l.location.Cavity) ),
• For the initial execution, the requirement that 10 trace, transferChunk.Select(g => g.sample));
only the minimal amount of jobs are used is a
must and the benchmark framework checks the Listing 2: Generating the liquid transfers for an
number of elements. AddReagent in the reference solution.
4.5. Evaluation criteria
The solutions are evaluated along the following criteria: 5. Reference solution
• Correctness The repository also contains a reference implementation
• Understandability that mimics how this kind of functionality would be im-
• Conciseness plemented using standard object-oriented code. That is, it
• Number of successful samples uses NMF [10] for the model representation but processes
• Number of elements in the low-level model the models using the generated model API using only
• Execution time functionality offered by the .NET Framework. For this,
we extended the model classes with a separate interface
Because the solutions have some degrees of freedom as needed for the transformation and created separated
with regard to the low-level jobs they produce, the result dedicated interfaces for tracing, which is done manually.
models cannot be tested for equality directly. Instead, the The following trace links are created:
benchmark framework performs the following checks:
• For each sample, we store the tube runner and
• The protocol step names must be the names of
the tube index where the sample comes into the
actual protocol steps.
system and the microplate and cavity where it is
• The target wells of TipLiquidTransfer elements
processed. These trace links are created in both di-
are between 0 and including 95.
rections, such that we can also trace a microplate
• The same well is not pipetted into multiple times.
cavity back to a sample.
The understandability of the solutions will be evalu- • For each job, we trace which samples it processes,
ated by a poll during the TTC event. To evaluate the both forwards and backwards, such that we can
conciseness, we ask every solution to note on the lines of identify the jobs for a sample and the samples
code of their solution. This shall include the model trans- processed by a job.
formation and glue code to actually run the benchmark. • For each reagent, we trace the trough in which
Code to convert the change sequence or code generated the reagent is placed.
from the metamodels can be excluded. For any graphical
part of the specification, we ask to count the lines of code Essentially, the reference solution calculates the re-
in a HUTN2 notation of the underlying model. quired jobs for a number of input samples separately
The number of elements is collected from the check through queries. As an example, the query for creat-
application and does not have to be calculated by the ing the liquid transfers for a AddReagent element is
solutions. Similarly, the number of successful samples is depicted in Listing 2.
calculated by the benchmark framework. For that, the In this listing, we first obtain the processing location
benchmark framework calculates all cavities where no for each sample, group them by plate and row and create
liquid transfers into the cavity failed and no incubation a liquid transfer job for each group, adding the tip indices
or wash of the corresponding plate failed. and trace links in separate methods not shown in the
For the execution time, the benchmark framework listing. Similar queries exist also for the other high-level
calculates the means for each phase per change sequence, elements.
tool and phase. We are particularly interested in the When changes of the high-level process model need to
incremental change propagation phase, i.e., the Update be propagated, the solution calculates the samples that
phase. are affected by the failures following the trace links. For
2
LiquidTransferJobs, this is depicted in Listing 3.
https://www.omg.org/spec/HUTN/
1 Tips.Where( t => t.Status == JobStatus.Failed ) of the input changes. For example, in the scope
2 .Select( tip => locationRepository.IdentifySample( Target
, tip.TargetCavityIndex ) )
of the benchmark, we considered the assay steps
3 .Where( s => s != null ); constant, but these may also change when some
parts of the analysis could still be performed even
Listing 3: Calculating failed samples for a if other parts are no longer possible (e.g. because
LiquidTransferJob reagents have run empty).
• The change propagation rules are repetitive and
duplicated while ideally, they could all be summa-
1 foreach(var sample in failedSamples.Distinct()) { rized in the fact that Job elements should only be
2 sample.State = SampleState.Error;
3 foreach(var job in _affectedJobsPerSample[sample]) { present either if they have already started or as
4 if (job.State == JobStatus.Planned) { long as any of the samples they process is not
5 job.GetProcessedSamples().Remove( sample );
6 if(job.GetProcessedSamples().Count == 0) { failed. In the reference solution, this abstract
7 job.Delete(); problem is encoded for each element separately
8 }
9 else if(job is LiquidTransferJob liquidTransfer) { and both for the high-level and the low-level
10 var processingLocation = _locationRepository. model elements separately, leading to a threat
LocateSampleProcessing( sample );
11 var tip = liquidTransfer.Tips.FirstOrDefault( t => t. of inconsistency problems.
TargetCavityIndex == processingLocation.Cavity )
; Given the advances in model transformation, espe-
12 tip?.Delete();
13 } cially in incremental change propagation, we think that
14 } the problem could be solved in a better way.
15 }
16 }
Listing 4: Change propagation implementation of the 6. Related work
reference solution
There have been many benchmarks of incremental or
bidirectional model processing tools, often originating
at the Transformation Tool Contest (TTC). These bench-
Once the failed samples are identified, the reference marks differ from the incremental workflows benchmark
solution removes the trace link to jobs. If a job or tip presented in this paper mainly by the relationship be-
transfer does not have a referenced sample any more, it tween input models and output models.
gets deleted. This is depicted in Listing 4. The Train Benchmark [9] benchmarks the incremental
We believe that the usage of the query syntax makes update performance of solutions for queries on a railway
the reference solution actually not too bad from a read- network that detected semantic errors. The benchmark
ability perspective and given the fact that it is imple- included queries of different complexity targeted at eval-
mented in plain C#, we assume it also has a good per- uating the incremental pattern matching performance.
formance. Still, the reference solution has multiple prob- The change sequences applied were plainly the model
lems: update operations to fix the semantic errors detected.
Similarly, the Social Media Benchmark [11] also aims to
• The high-level process model elements have an benchmark incremental query performance, but on a
explicit knowledge about their transformation to different domain, with a query that integrated graph al-
low-level job elements and in the other direction, gorithms in order to evaluate the integration of custom
the low-level job elements have an explicit knowl- dynamic algorithms and with change sequences indepen-
edge about which samples they are processing. dent from the queries. Both of these benchmarks target
This is good enough for a quick solution, but nor- plain queries where the structure of the result is com-
mally undesirable as the high-level model is also pletely different than the structure of the input models
used in other contexts such as an editor. or not present at all (the result of the social media bench-
• Because the tracing is done manually, it is only mark is just a set of three pointers to model elements).
done on selected points. If the transformation In contrast, the Families to Persons Benchmark [12]
becomes more complex, this leads to additional considers a simple bidirectional transformation scenario
overhead as more trace links will become neces- where input model and output model represent the same
sary. information, though according to different metamodels.
• The change propagation is done manually, which There is always a clear 1:1 relationship between model
means that only selected types of changes are elements of either side. The benchmark is thus rather
actually supported. However, it is very difficult targeted at evaluating and comparing the approaches to
to exclude certain types of changes because there support bidirectional transformations. The benchmark
is usually still some scenario in which every part does consider incremental updates of the models, but
due to the 1:1 mapping, the required efforts to propagate analyses, Softw. Syst. Model. 18 (2019) 3151–3187.
these changes are rather simple and clear. URL: https://doi.org/10.1007/s10270-019-00719-y.
The Smart Grid Benchmark [13] considers queries that doi:10.1007/s10270-019-00719-y.
merge elements from two models into a view that com- [7] G. Hinkel, E. Burger, Change propagation and
bines information from both input models. Still, there is bidirectionality in internal transformation dsls,
a clear 1:1 mapping between tuples of elements of both Softw. Syst. Model. 18 (2019) 249–278. URL: https://
input models and the output model. doi.org/10.1007/s10270-017-0617-6. doi:10.1007/
In the Java Refactoring Benchmark [14], the task was s10270-017-0617-6.
to extract a simplified refactoring model from a Java code [8] T. L. Calvar, F. Jouault, F. Chhel, M. Clavreul, Ef-
model, then apply refactorings at the simplified model ficient ATL incremental transformations, J. Ob-
and put back the resulting changes to the Java model. ject Technol. 18 (2019) 2:1–17. URL: https://doi.org/
Because the refactoring model featured model elements 10.5381/jot.2019.18.3.a2. doi:10.5381/jot.2019.
representing all methods with the same name regardless 18.3.a2.
of declaring class, this transformation also has a more [9] G. Szárnyas, B. Izsó, I. Ráth, D. Varró, The train
complex mapping than 1:1 relationships. However, the benchmark: cross-technology performance eval-
correspondences are not bounded. Furthermore, this uation of continuous model queries, Software &
benchmark is not equipped with a framework targeting Systems Modeling 17 (2018) 1365–1393.
incremental update performance. [10] G. Hinkel, Nmf: A multi-platform modeling frame-
work, in: A. Rensink, J. Sánchez Cuadrado (Eds.),
Theory and Practice of Model Transformation,
References Springer International Publishing, Cham, 2018, pp.
184–194.
[1] F. Kong, L. Yuan, Y. F. Zheng, W. Chen, Automatic
[11] G. Hinkel, The TTC 2018 social media case, in:
liquid handling for life science: a critical review of
Proceedings of the 11th Transformation Tool Con-
the current state of the art, Journal of laboratory
test, co-located with the 2018 Software Technolo-
automation 17 (2012) 169–185.
gies: Applications and Foundations, TTCSTAF
[2] M. Li, Wanted: special-purpose robots automating
2018, Toulouse, France, June 29, 2018, volume
life science wet lab workflows, Future science OA
2310 of CEUR Workshop Proceedings, CEUR-WS.org,
1 (2015).
2018, pp. 39–43. URL: http://ceur-ws.org/Vol-2310/
[3] D. Hearnden, M. Lawley, K. Raymond, Incre-
paper5.pdf.
mental model transformation for the evolution of
[12] A. Anjorin, T. Buchmann, B. Westfechtel, Z. Diskin,
model-driven systems, in: Model Driven Engi-
H. Ko, R. Eramo, G. Hinkel, L. Samimi-Dehkordi,
neering Languages and Systems, 9th International
A. Zündorf, Benchmarking bidirectional transfor-
Conference, MoDELS 2006, Genova, Italy, Octo-
mations: theory, implementation, application, and
ber 1-6, 2006, Proceedings, volume 4199 of Lec-
assessment, Softw. Syst. Model. 19 (2020) 647–691.
ture Notes in Computer Science, Springer, 2006, pp.
URL: https://doi.org/10.1007/s10270-019-00752-x.
321–335. URL: https://doi.org/10.1007/11880240_23.
doi:10.1007/s10270-019-00752-x.
doi:10.1007/11880240\_23.
[13] G. Hinkel, The TTC 2017 outage system case
[4] D. Varró, G. Bergmann, Á. Hegedüs, Á. Horváth,
for incremental model views, in: Proceedings of
I. Ráth, Z. Ujhelyi, Road to a reactive and in-
the 10th Transformation Tool Contest (TTC 2017),
cremental model transformation platform: three
co-located with the 2017 Software Technologies:
generations of the VIATRA framework, Softw.
Applications and Foundations (STAF 2017), Mar-
Syst. Model. 15 (2016) 609–629. URL: https://
burg, Germany, July 21, 2017, volume 2026 of CEUR
doi.org/10.1007/s10270-016-0530-4. doi:10.1007/
Workshop Proceedings, CEUR-WS.org, 2017, pp. 3–
s10270-016-0530-4.
12. URL: http://ceur-ws.org/Vol-2026/paper1.pdf.
[5] A. Anjorin, E. Yigitbas, E. Leblebici, A. Schürr,
[14] G. Kulcsár, S. Peldszus, M. Lochau, Object-oriented
M. Lauder, M. Witte, Description languages for
refactoring of java programs using graph transfor-
consistency management scenarios based on
mation, in: Proceedings of the 8th Transformation
examples from the industry automation domain,
Tool Contest, a part of the Software Technologies:
Art Sci. Eng. Program. 2 (2018) 7. URL: https://doi.
Applications and Foundations (STAF 2015) feder-
org/10.22152/programming-journal.org/2018/2/7.
ation of conferences, L’Aquila, Italy, July 24, 2015,
doi:10.22152/programming-journal.org/
volume 1524 of CEUR Workshop Proceedings, CEUR-
2018/2/7.
WS.org, 2015, pp. 53–82. URL: http://ceur-ws.org/
[6] G. Hinkel, R. Heinrich, R. H. Reussner, An ex-
Vol-1524/paper3.pdf.
tensible approach to implicit incremental model
Appendix: Solution Requirements not save intermediate data between different runs. Mean-
while, all iterations of the Update phase are executed in
The solutions are required to perform the steps of the the same process and solutions are allowed (and encour-
benchmark in the order depicted above. Solutions must aged) to save any intermediate computation results they
report the following metrics between these steps, in case like, as long as the results are correct after each change
of the update phase after every change sequence. The sequence.
reporting is done by printing the following separated by
; to the standard output:
• Tool: The name of the tool.
• Scenario: The scenario of the models (i.e. scaling
samples or assay steps)
• Model: The name of the input model set that is
currently run
• RunIndex: The run index in case the benchmark
is repeated
• Iteration: The iteration (only required for the
Update phase)
• PhaseName: The phase of the benchmark
• MetricName: The name of the reported metric
• MetricValue: The value of the reported metric
Tool, Scenario, Model and RunIndex are provided to
the solution using environment variables with the same
name. Further, the benchmark framework passes the root
directory of the models using the variable ModelPath and
the number of update iterations using Sequences.
Solutions should report on the runtime of the respec-
tive phase in integer nanoseconds (Time) and the work-
ing set in bytes (Memory). The memory measurement
is optional. If it is done, it should report on the used
memory after the given phase (or iteration of the update
phase) is completed. Solutions are allowed to perform
a garbage collection before memory measurement that
does not have to be taken into account into the times. In
the update phase, we are not interested in the time to
parse and identify the changes, but only the pure change
propagation.
To enable automatic execution by the benchmark
framework, solutions should add a subdirectory to the
solutions folder of the benchmark with a solution.ini file
stating how the solution should be built and how it should
be run. Because the solution contains the already com-
piled reference solution, no action is required for build.
However, other solutions may want to run build tools
like maven in this case to ensure the benchmark runs
with the latest version.
The repetition of executions as defined in the bench-
mark configuration is done by the benchmark. This
means, for 5 runs, the specified command-line will be
called 5 times, passing any required information such as
the model that should be computed, the run index, etc.
in separate environment variables. All runs should all
have the same prerequisites. In particular, solutions must