When executing high-level process models, exceptions sometimes need to be handled by changing the planned process on the fly, leaving the jobs that have already been executed intact. Therefore, transformations that transform such a high-level process model to low-level jobs need to be incremental and respect low-level job elements that represent actions that have already happened. As an example, we consider the automation of laboratory workflows where failures of low-level jobs may need to get compensated by taking out samples from the automated process. Based on this scenario, we present a benchmark for transformation tools to deal with such kind of problems.
When the pandemic situation due to Covid-19 started in spring 2020, the availability of test capacities was a huge problem and is still a critical problem when the paper was written. As one of the reasons, this is due to the fact that workflows executed in laboratories are usually automated in an inflexible manner or not automated at all. In the latter case, this was often not due to the fact that laboratories were lacking instruments capable of automating a workflow such as Covid-19 tests, but rather to the fact that it was hard to repurpose these instruments for Covid-19 tests, besides to verification and validation eforts also due to the lack of flexibility of the control software for such instruments.
These instruments are often robotic liquid handlers (RLHs) equipped with some additional devices such as Figure 1: A Tecan Fluent instrument thermocyclers, shakers or readers. These RLHs come either tied and optimized for a set of specific applications (especially clinical applications) or as flexible instruments that can solve various tasks [1]. An example of abstraction provided to the user, which usually is a of the latter is the Fluent instrument from Tecan1 shown chemist or biologist but not typically not a computer sciin Figure 1. These instruments currently ofer the end- entist, is often rather low. This makes it hardly accessible, user a domain-specific language to create and manage unless users get a substantial amount of training [2]. Furprograms that can be executed by the robot, in the case thermore, it is very hard for these low-level descriptions of Fluent called scripts. This domain-specific language is to cope with multiple scales, such as multiple supported modular, such that it can be extended by certain elements, amounts of samples processed by the robot at a time. e.g. to support the integration of third-party devices that To overcome this problem, there are several approaches can perform subtasks such as shaking or heating a plate. to raise the level of abstraction in which the laboratory The general problem with scripting is that the level workflow is formulated and how it is transformed to low-level jobs executable by the robot.
When following such an approach, one quickly runs into problems that when exceptions occur, e.g. consumables or reagents run empty during the process. Exceptions usually occur at the lower levels of abstraction such as e.g., the hardware, and it is up to the automation system to translate this exception into the high-level workflow description by tracing back into the high-level TTC’21: Transformation Tool Contest, Part of the Software Technologies: Applications and Foundations (STAF) federated conferences, Eds. A. Boronat, A. García-Domínguez, and G. Hinkel, 25 June 2021, Bergen, Norway (online). " georg.hinkel@tecan.com (G. Hinkel)
© 2021 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
1https://lifesciences.tecan.com/ lfuent-laboratory-automation-workstation description of the process. is important to keep a reference between the high-level
RLHs are expected to recover from errors whenever process that is executed and the low-level jobs, but this possible, e.g. by asking the user to refill the consumables correspondence is not a strict 1:1 relationship. A highand continue. Depending on the process, there may be level protocol step can be traced to multiple low-level implications on the process, for example because time jobs, but a single low-level job represents the processing constraints can no longer be met. In these situations, of only a selection of samples. Previous benchmarks for multiple options exist, depending on both the error and incremental change propagation (cf. Section 6) have conits context. If viable, a solution can be to just retry or to centrated either on cases where there is no trace between lfag the sample that time constraints could not be met. input models and output or the correspondence is always Depending on the length of the pause and the process, strictly 1:1. Hence, it is not clear for a practitioner how this can already mean that the results for these samples such non-trivial correspondences are supported in existare not usable, e.g. because reagents evaporated while ing incremental model transformation tooling. Furtherthe users was refilling consumables. In such cases, one more, it is not clear which incremental transformation would want to simply skip the processing for the afected approaches are capable of specifying such constraints, i.e. samples in order to save consumables and reagents or that change propagation of some elements is restricted one wants to abort processing for the samples not yet to some specific states. Therefore, this paper presents processed in order to save the sample. Another exam- a transformation challenge and benchmark how such ple are redundant devices that run into failures and one change propagation systems can be specified and exewould like to use a diferent device instead. cuted in an incremental manner.
Therefore, the automation system needs to do two The remainder of this paper presents a tool challenge things, one is to identify the required changes to the to solve a minimized version of this problem. At first, we high-level process and the second one is to propagate describe the challenges that we see in this case in Section these changes to low-level elements. However, since the 2. Section 3 then describes the case itself. Section 4 inerror occurred while the workflow was executing, some troduces the benchmark framework and Section 5 briefly elements of the low-level model cannot be changed any- explains the reference solution and its deficits. Section 6 more since represent actions performed in the past. It is ifnally sums up the diferences to previous benchmarks. impossible to undo what the instrument has already done, but it is possible to change what the instrument shall be doing next. Because time constraints are in operation 2. Challenges while these changes are propagated, it is also important that the replanning happens fast. In particular, the following aspects of the problem are
Given the recent advances in incremental model trans- especially important: formation and analysis [3, 4, 5, 6, 7, 8], we believe a viable • Because the transformation from high-level prosolution approach would to design the transformation cess models to low-level execution jobs happens from the high-level process model to low-level execu- at runtime, the performance of this transformation jobs as an incremental transformation. Ideally, the tion is important. However, it is suficient when incrementalization is done implicitly, i.e. without the the jobs are clear when they have to be executed, transformation developer having to adapt the transfor- i.e. it does not matter if the planning is not finmation, to support a wide range of changes in the process ished for the entire jobs as long as it is clear what model. When the transformation would create jobs, it the instrument shall be doing next. How long would create them with an initial execution state and does it take for the transformation to return the then restrict the change propagation to those elements ifrst low-level element? that are still in initial state. • If a job fails, one needs to identify how the process
Another way to look at the problem is to understand it model needs to be adapted. How to specify this in as a consistency problem: For each sample and for each part of the input process, there must be a corresponding an understandable way? low-level action element. When such a low-level element • Given a change of the process model, one needs fails, the corresponding sample would fail (if there is no to adjust the jobs that need to be performed as alternative recovery). Setting the sample to a failed state quick as possible. How long does it take? would let the consistency mechanism remove all the low- • While propagating changes to the low-level jobs, level elements that have not been executed yet, ideally it is tremendously important that the jobs that reporting those that have been executed as inconsisten- have been executed already stay in place. How cies (because they would not have been necessary given to make sure that the change propagation does not that the processing of the corresponding sample failed). afect elements representing actions that have hap
However, in this domain the maintenance of a trace pened in the past?
their tool copes with the challenges described above.
Similar challenges occur in domains such as smart production where one needs to plan steps according to a general production protocol. Therefore, we aim to address the following general research questions with this benchmark: • The support for change propagation should not the specification of what an arm should do and we also degrade the understandability of the transforma- removed the scheduling problem. Therefore, the transtion from the abstract process model to the low- formation is limited to calculate what the RLH has to do, level. What changes are necessary to enable an but not when or where. incremental change propagation? In the remainder of this section, we present these two metamodels in Sections 3.1 and 3.2. Then, we discuss the transformation rules in Section 3.3 and the change propagation rules in Section 3.4.
The minimized metamodel of laboratory workflows that we use for the benchmark is shown in Figure 2. The cen• Are non-trivial (i.e. not 1:1) correspondences sup- tral element is a JobRequest that represents to process a ported by recent incremental model transforma- range of samples using a given Assay. This assay element tion approaches? represents the actual process. Meanwhile true laboratory • Is it (still) possible to implicitly derive an incre- workflows often consist of a vast variety of operations, mental change propagation in the presence of the benchmark only considers four possible process steps, non-trivial correspondences? namely distributing sample, adding a reagent, washing • If not possible implicitly, what is the required or incubating. Each protocol step has an id for the sake development efort to implement change propa- of identifying it in the remainder. Further, protocol steps gation in the presence of non-trivial correspon- carry links to their previous and next steps in order to dences? ease analysis. Meanwhile most workflows require the • How does the presence of non-trivial correspon- replication of samples and also need to include standards dences afect performance of incremental model such as positive and negative controls, this is also omitted transformations? in this simplified model.
Samples are identified by a sample id, which is usually the barcode on the sample. Assays have a name assigned to them. In Figure 2, they are contained in a JobRequest to make it easier to work with them, though in practice they usually stand for themselves. 3.2. The low-level job metamodel is, it is allowed to pipette cavity indices 0 and 1 inside the same element, but not 0 and 8. Liquid can be transferred The low-level metamodel has a viewpoint that is more from and to any kind of labware. concerned with the actual execution of the workflow on We assume here that a separate device is used for washa RLH. Here, the execution of the entire process is repre- ing that only works with microplates, but washes cavities sented by a JobCollection that consist of a series of Job separately. Therefore, a Wash element carries a list of all elements and a list of labware that is needed. Labware cavity indices that should be washed and a reference to means in this context the consumables needed to execute the plate that should be washed. a workflow. The jobs could be LiquidTransfer, Wash Similarly, an Incubate steps works only for mior Incubate elements. For laboratory automation, one croplates, but it always incubates an entire plate, because often uses microplates for processing, essentially plates it is physically dificult to heat only parts of it. with a large number of wells (‘holes’) that are used as small tubes (see Figure 4). Microplates have a standardized size, but multiple formats with diferent amounts 3.3. Transformation rules of wells exist. The probably most common format is a The transformation from the high-level process model microplate with 96 wells. Tube runners are racks that to the low-level job model needs to follow these requirecan host multiple tubes and allow easy loading of an in- ments: strument with new tubes. Troughs are simple containers with a relatively large volume (usually 100ml or more) • For each JobRequest, a JobCollection should that are used to host reagents. be created.
Each LiquidTransfer element represents the execu- • Samples are processed on microplates but come tion of a liquid transfer command in the RLH. Because in tubes stored on racks with 16 tubes called tube RLHs are usually equipped with multiple pipettes, the runners. Therefore, for each Sample, a cavity of RLH can pipette multiple cavities of a microplate or tube a TubeRunner should be assigned. Further, the rack at once. Therefore, a LiquidTransfer element has transformation needs to store a mapping between up to eight TipLiquidTransfer elements that specify a Sample and a combination of a Microplate and which wells the individual tips of the transfer operation a cavity index. The combination of microplate should target. It is important that the pipettes of a pipet- and cavity needs to be unique. At most 96 cavitor usually share the X axis, which means that all source ties of a microplate can be used. For tracing, the cavity indices of TipLiquidTransfer elements inside a microplates should be simply numbered with a LiquidTransfer elements may only difer by their re- common prefix. The first microplate should be mainder in the division by 8. Similarly, all target cavity named Microplate1 and so on and so forth. indices must be the same except for the modulo 8. That • We assume that reagents are stored in troughs. 3.4. Change propagation rules Therefore, for each Reagent element, there should be a Trough element with the name set to the name of the reagent. • For each combination of ProtocolStep and Sample that is not in the Error state, a Job needs to be added to the JobCollection corresponding to the parent JobRequest of the parent Assay as defined per the following rules:
the benchmark. One is to change the high-level model as reaction to low-level changes and the other change propagation is in the direction vice versa.
The first change propagation will be very simple in the scope of this benchmark. Whenever a Job is failed, all corresponding samples have to change their state to failed as well. If a LiquidTransferJob fails, only those – For each DistributeSample element, samples using the failed TipLiquidTransfer elements there must be a TipLiquidTransfer that have to be set to failed. represents the transfer of the sample from The more interesting change propagation is the change cavity 0 of the tube containing the sam- propagation in the opposite direction. In the scope of ple to the cavity of the assigned mi- the benchmark, we only consider changes to the state of croplate. Multiple TipLiquidTransfer el- samples. If a sample changes its state to failed (e.g. as ements may share the same parent Liquid- a consequence of a change propagation from the lower Transfer element. level), all jobs created for this sample need to be removed, – For each AddReagent element, there provided that they are still only planned and no other must be a TipLiquidTransfer that repre- samples are afected. That is, a WashJob or IncubateJob sents the transfer of reagent to the cavity may only be removed if all samples on the corresponding with the sample, i.e. from cavity 0 of the plate are failed. trough created for the reagent to the cavity and microplate assigned to the sample.
Again, multiple TipLiquidTransfer ele- 4. Benchmark ments may share a parent LiquidTrans- We provide a benchmark framework that can automatifer element if the conditions regarding the cally compile, run and check solutions and generate diacavity indices are met. grams to analyze the results. The benchmark framework – For each Wash, a WashJob should be cre- is based on the Train Benchmark [9]. The framework as ated that washes the microplate assigned well as metamodels, input models and the reference soto the sample and at least the cavity in- lution is publicly available online at https://github.com/ dex of the sample. However, the same tecan/ttc21incrementalLabWorkflows.
WashJob may be reused for multiple sam- In the remainder of this section, we first describe the ples provided that the transformation as- phases of the benchmark in Section 4.1, then explain signs the samples to the same microplate. how to run it in Section 4.2. Next, we introduce the input – For each Incubate, an IncubateJob with models in Section 4.3. Section 4.4 explains the correctness the same temperature and duration should checks and Section 4.5 introduces the evaluation criteria be created that incubates the microplate for solutions. Section 6 then explains the procedure to assigned to the sample. The same Incu- add a solution to the benchmark. bateJob must be reused for the incubation of the same microplate, provided that the transformation assigns them to the same 4.1. Phases microplate. The benchmark is divided into the following phases: – Each created job must have the name of the
protocol step from which it was created. – Each created job must reference jobs created for the previous protocol step for the same samples such that the job is scheduled afterwards. • The transformation should ideally produce the minimum amount of elements, i.e. the Liquid- The last step is performed repeatedly 20 times. The TransferJob, WashJob and IncubateJob ele- benchmark seeks to compare execution times for all of ments should be shared where this is possible. the phases. For the last step, any eforts necessary for parsing and loading the changes can be excluded from the time measurements.
1. Initialization: Loading the transformation and
metamodels 2. Load: Loading the input models 3. Initial: Creating the initial JobCollection 4. Update: Reading changes to job executions and propagate
above and R to be installed. R is required to create diagrams for the benchmark results. Furthermore, the solutions may imply additional frameworks. We would ask solution authors to explicitly note dependencies to additional frameworks necessary to run their solutions.
If all prerequisites are fulfilled, the benchmark can be run using Python with the command python scripts/run.py. Additional command-line options can be queried using the option –help. "Name": "test", "Models": ["minimal"] }], "Sequences": 20, "Runs": 5, "Timeout": 6000
same assay, but vary the number of samples or work with duplicates of the assay. That is, the benchmark has Listing 1: A minimal benchmark configuration a sequence of models with increasing number of samples but the same assay or an increasing number of steps in
The benchmark framework can be configured using the assay (by replicating the steps) and a constant number JSON configuration files. A minimal test configuration is of samples. There are multiple change sequences that depicted in Listing 1. When creating a new solution, we scale the assay or the number of samples, with or without highly recommend to overwrite the contents of this con- new samples on the fly. ifguration file locally. In the configuration from Listing To specify changes in the low-level job model, we use 1, only the test scenario with just one minimal model is text files that specify these jobs in terms of the high-level executed, using only the solution in Reference 5 times. model. Each line of these text files corresponds to one state change of one job. The lines are in the format 4.3. Input Models <ProtocolStepName>_<Plate>_<State> where state is either S for success or F for a failure (we skip As inputs, we use models of a very abstract Elisa (Enzyme- the executing state). These status lines have been linked Immunosorbent Assay) workflows. These work- generated independently from an actual solution and lfows are generally used to detect antibodies, for example thus, they include also status changes of jobs for samples to proof that a patient has experienced a Covid-19 infec- that would have to be set to a failed state due to a tion or has been vaccinated. For this, we assume that previous failure. In case of a liquid transfer, there are we have access to microplates that are already prepared 96 states, indicating the success state for each cavity, with Covid antigens at the bottom of the wells. Then, we independently of whether this cavity is used or not. add the sample. After an incubation period, antibodies However, changes may also afect the samples, i.e. new against Covid-19 in the sample, if any, will bind to the samples may be added dynamically. This is indicated by antigens at the bottom of the well. Then, the plate is a line in the format NewSample_<SampleId>. washed (in order to get rid of other antibodies) and a As stated above, the time needed to calculate the job elconjugate is added. In a second incubation step, special ements that are actually afected by these changes should marked antibodies bind to the Covid-19 antibodies, if any. be excluded from the time measurements, but the actual Afterwards, the plate is washed again and a substrate is change propagation should be included. added that reacts with the marked antibodies and, after The high-level process models are available in EMF another incubation period, results in a color reaction. format, but can be made available in other formats as
The result is a plate where some cavities are colored well, upon request. while others are not, as illustrated in Figure 4. Such a plate can be read by an absorbance reader, which the 4.4. Correctness checks limited process model does not support. In such an image, the color correlates with the amount of antibodies present in the sample.
rectness checks after the initial transformation and after each update:
The solutions are evaluated along the following criteria: • Correctness • Understandability • Conciseness • Number of successful samples • Number of elements in the low-level model • Execution time
Because the solutions have some degrees of freedom with regard to the low-level jobs they produce, the result models cannot be tested for equality directly. Instead, the benchmark framework performs the following checks: • The protocol step names must be the names of actual protocol steps. • The target wells of TipLiquidTransfer elements are between 0 and including 95.
• The same well is not pipetted into multiple times.
ated by a poll during the TTC event. To evaluate the conciseness, we ask every solution to note on the lines of code of their solution. This shall include the model transformation and glue code to actually run the benchmark. Code to convert the change sequence or code generated from the metamodels can be excluded. For any graphical part of the specification, we ask to count the lines of code in a HUTN2 notation of the underlying model.
The number of elements is collected from the check application and does not have to be calculated by the solutions. Similarly, the number of successful samples is calculated by the benchmark framework. For that, the benchmark framework calculates all cavities where no liquid transfers into the cavity failed and no incubation or wash of the corresponding plate failed.
For the execution time, the benchmark framework calculates the means for each phase per change sequence, tool and phase. We are particularly interested in the incremental change propagation phase, i.e., the Update phase.
2https://www.omg.org/spec/HUTN/
The repository also contains a reference implementation that mimics how this kind of functionality would be implemented using standard object-oriented code. That is, it uses NMF [10] for the model representation but processes the models using the generated model API using only functionality ofered by the .NET Framework. For this, we extended the model classes with a separate interface as needed for the transformation and created separated dedicated interfaces for tracing, which is done manually.
The following trace links are created: • For each sample, we store the tube runner and the tube index where the sample comes into the system and the microplate and cavity where it is processed. These trace links are created in both directions, such that we can also trace a microplate cavity back to a sample. • For each job, we trace which samples it processes, both forwards and backwards, such that we can identify the jobs for a sample and the samples processed by a job. • For each reagent, we trace the trough in which the reagent is placed.
quired jobs for a number of input samples separately through queries. As an example, the query for creating the liquid transfers for a AddReagent element is depicted in Listing 2.
In this listing, we first obtain the processing location for each sample, group them by plate and row and create a liquid transfer job for each group, adding the tip indices and trace links in separate methods not shown in the listing. Similar queries exist also for the other high-level elements.
When changes of the high-level process model need to be propagated, the solution calculates the samples that are afected by the failures following the trace links. For LiquidTransferJobs, this is depicted in Listing 3. 1 Tips.Where( t => t.Status == JobStatus.Failed ) 2 .Select( tip => locationRepository.IdentifySample( Target , tip.TargetCavityIndex ) ) 3 .Where( s => s != null );
LiquidTransferJob samples for a
bidirectional model processing tools, often originating at the Transformation Tool Contest (TTC). These bench
Once the failed samples are identified, the reference marks difer from the incremental workflows benchmark solution removes the trace link to jobs. If a job or tip presented in this paper mainly by the relationship betransfer does not have a referenced sample any more, it tween input models and output models. gets deleted. This is depicted in Listing 4. The Train Benchmark [9] benchmarks the incremental
We believe that the usage of the query syntax makes update performance of solutions for queries on a railway the reference solution actually not too bad from a read- network that detected semantic errors. The benchmark ability perspective and given the fact that it is imple- included queries of diferent complexity targeted at evalmented in plain C#, we assume it also has a good per- uating the incremental pattern matching performance. formance. Still, the reference solution has multiple prob- The change sequences applied were plainly the model lems: update operations to fix the semantic errors detected.
Similarly, the Social Media Benchmark [11] also aims to • The high-level process model elements have an benchmark incremental query performance, but on a explicit knowledge about their transformation to diferent domain, with a query that integrated graph allow-level job elements and in the other direction, gorithms in order to evaluate the integration of custom the low-level job elements have an explicit knowl- dynamic algorithms and with change sequences indepenedge about which samples they are processing. dent from the queries. Both of these benchmarks target This is good enough for a quick solution, but nor- plain queries where the structure of the result is commally undesirable as the high-level model is also pletely diferent than the structure of the input models used in other contexts such as an editor. or not present at all (the result of the social media bench• Because the tracing is done manually, it is only mark is just a set of three pointers to model elements). done on selected points. If the transformation In contrast, the Families to Persons Benchmark [12] becomes more complex, this leads to additional considers a simple bidirectional transformation scenario overhead as more trace links will become neces- where input model and output model represent the same sary. information, though according to diferent metamodels. • The change propagation is done manually, which There is always a clear 1:1 relationship between model means that only selected types of changes are elements of either side. The benchmark is thus rather actually supported. However, it is very dificult targeted at evaluating and comparing the approaches to to exclude certain types of changes because there support bidirectional transformations. The benchmark is usually still some scenario in which every part does consider incremental updates of the models, but due to the 1:1 mapping, the required eforts to propagate these changes are rather simple and clear.
The Smart Grid Benchmark [13] considers queries that merge elements from two models into a view that combines information from both input models. Still, there is a clear 1:1 mapping between tuples of elements of both input models and the output model.
In the Java Refactoring Benchmark [14], the task was to extract a simplified refactoring model from a Java code model, then apply refactorings at the simplified model and put back the resulting changes to the Java model.
Because the refactoring model featured model elements representing all methods with the same name regardless of declaring class, this transformation also has a more complex mapping than 1:1 relationships. However, the correspondences are not bounded. Furthermore, this benchmark is not equipped with a framework targeting incremental update performance. The solutions are required to perform the steps of the benchmark in the order depicted above. Solutions must report the following metrics between these steps, in case of the update phase after every change sequence. The reporting is done by printing the following separated by ; to the standard output: • Tool: The name of the tool. • Scenario: The scenario of the models (i.e. scaling
samples or assay steps) • Model: The name of the input model set that is
currently run • RunIndex: The run index in case the benchmark
is repeated • Iteration: The iteration (only required for the
Update phase) • PhaseName: The phase of the benchmark • MetricName: The name of the reported metric • MetricValue: The value of the reported metric
Tool, Scenario, Model and RunIndex are provided to the solution using environment variables with the same name. Further, the benchmark framework passes the root directory of the models using the variable ModelPath and the number of update iterations using Sequences.
Solutions should report on the runtime of the respective phase in integer nanoseconds (Time) and the working set in bytes (Memory). The memory measurement is optional. If it is done, it should report on the used memory after the given phase (or iteration of the update phase) is completed. Solutions are allowed to perform a garbage collection before memory measurement that does not have to be taken into account into the times. In the update phase, we are not interested in the time to parse and identify the changes, but only the pure change propagation.
To enable automatic execution by the benchmark framework, solutions should add a subdirectory to the solutions folder of the benchmark with a solution.ini file stating how the solution should be built and how it should be run. Because the solution contains the already compiled reference solution, no action is required for build.
However, other solutions may want to run build tools like maven in this case to ensure the benchmark runs with the latest version.
The repetition of executions as defined in the benchmark configuration is done by the benchmark. This means, for 5 runs, the specified command-line will be called 5 times, passing any required information such as the model that should be computed, the run index, etc. in separate environment variables. All runs should all have the same prerequisites. In particular, solutions must not save intermediate data between diferent runs. Meanwhile, all iterations of the Update phase are executed in the same process and solutions are allowed (and encouraged) to save any intermediate computation results they like, as long as the results are correct after each change sequence.