=Paper=
{{Paper
|id=Vol-3089/ttc21_paper_labflow_Copei_solution
|storemode=property
|title=The Fulib solution to the TTC 2021 laboratory workflow case
|pdfUrl=https://ceur-ws.org/Vol-3089/ttc21_paper8_labflow_Copei_solution.pdf
|volume=Vol-3089
|authors=Sebastian Copei,Adrian Kunz,Albert Zuendorf
|dblpUrl=https://dblp.org/rec/conf/ttc/CopeiKZ21
}}
==The Fulib solution to the TTC 2021 laboratory workflow case==
https://ceur-ws.org/Vol-3089/ttc21_paper8_labflow_Copei_solution.pdf
The Fulib solution to the TTC 2021 laboratory workflow
case
Sebastian Copei1 , Adrian Kunz1 and Albert Zuendorf1
1
Kassel University, Germany
Keywords
model transformation, tool presentation, java
1. Introduction
This paper outlines the Fulib [1, 2] solution to the Labora-
tory Workflow Case of the Transformation Tool Contest
2021 [3]. Our analysis of the use case showed that it
provides quite a number of different model elements that
require individual treatment but the different cases are
relatively simple. However, some parts of the predefined Figure 1: Design
EMF metamodels do not work very well with the Fulib
modeling approach. For example, the predefined meta-
model uses index numbers to identify the tips of a liquid very light weight implementation of our model in Java
transfer job and these index numbers need to be mapped code and Fulib generates a number of dedicated Table
to the barcodes of the samples that are transported by a classes that enable efficient OCL [4] like queries. The
tip. Similarly, samples need to be mapped to cavities on actual model transformations are coded in Java against
micro-plates. Thus, we took the liberty to adapt the given the generated model API. Thus, the Fulib solution has no
metamodels by adding explicit associations between sam- initialization phase.
ples and some labware elements, cf. Fig. 1. Note, these The various input models that describe a JobRequest,
adaptions connect elements from the source and from the its Assay and the target Samples are given as EMF/XMI
target metamodel of our model to model transformation. files (*/initial.xmi). We load the initial model with a
For these adaptions, we loaded and combined the two generic XML parser and a DOM tree visitor, that builds
given Ecore metamodels of the use case into the Fulib the model based on our light weight model implementa-
code generator and then did some manual modifications tion.
using Fulibs metamodeling API. Due to a misinterpreta-
tion, we also changed the cardinality of the previous-next
association for Jobs from many-to-many to one-to-one. 3. Creating the initial
We felt this meets the semantics of the use case, too, JobCollection
and it resulted in a somewhat simpler model that can be
processed easier and faster. Once the JobRequest and Assay are loaded, we use
The rest of the paper outlines the implementation of an AssayToJobs visitor [5] to generate the initial
the different model processing steps and we conclude JobCollection, cf. Listing 1.Using the visitor pattern
with some measurements. allows for a nice separation of model queries that look up
elements and of transformation rules that do the actual
operations.
2. Initialization and Loading The initial method of our AssayToJobs visitor
The initialization phase allows to load the metamodels first creates the target JobCollection (cf. line 6 of
and transformations. In our approach, Fulib generates a Listing 1). Then it iterates through the samples, reagents,
and assay steps and calls appropriate assign rules (cf.
TTC’21: Transformation Tool Contest, Part of the Software lines 7 to 14).
Technologies: Applications and Foundations (STAF) federated
The assignToTube rule checks, whether we have a
conferences, Eds. A. Boronat, A. García-Domínguez, and G. Hinkel,
25 June 2021, Bergen, Norway (online). TubeRunner that still has place for the new sample (cf.
" sco@uni-kassel.de (S. Copei); a.kunz@uni-kassel.de (A. Kunz); line 20). If not, a new TubeRunner is created (cf. line
zuendorf@uni-kassel.de (A. Zuendorf) 22 to 26) and added to the JobCollection (cf. line 25).
© 2021 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). Then, the sample’s barcode is added to the TubeRunner
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
1 public class AssayToJobs {
2 private JobCollection jc;
3 private JobRequest jr;
4 public JobCollection initial(JobRequest jr) { 1 public class AssayToJobs {
5 this.jr = jr; 2 ...
6 jc = new JobCollection(); 3 Map>
7 jr.getSamples() 4 stepAssignRules = null;
8 .forEach(this::assignToTube); 5 private void assignJob(ProtocolStep ps) {
9 jr.getSamples() 6 initStepAssignRules();
10 .forEach(this::assignToPlate); 7 Consumer rule =
11 jr.getAssay().getReagents() 8 stepAssignRules.get(ps.getClass());
12 .forEach(this::assignToTrough); 9 rule.accept(ps);
13 jr.getAssay().getSteps() 10 }
14 .forEach(this::assignJob); 11 private void initStepAssignRules() {
15 return jc; 12 if (stepAssignRules == null) {
16 } 13 stepAssignRules = new LinkedHashMap<>();
17 TubeRunner tube = null; 14 stepAssignRules
18 int tn = 1; 15 .put(DistributeSample.class,
19 private void assignToTube(Sample sample) { 16 this::assignLiquidTransferJob4Samples);
20 if (tube == null || 17 stepAssignRules
21 tube.getBarcodes().size() == 16) { 18 .put(Incubate.class,
22 tube = new TubeRunner(); 19 this::assignIncubateJob);
23 tube 20 stepAssignRules
24 .setName(String.format("Tube%02d", tn)) 21 .put(Wash.class, this::assignWashJob);
25 .setJobCollection(jc); 22 stepAssignRules
26 tn++; 23 .put(AddReagent.class,
27 } 24 this::assignAddReagentJob);
28 tube.withBarcodes(sample.getSampleID()); 25 }
29 tube.withSamples(sample); 26 }
30 } 27 private void
31 ... 28 assignLiquidTransferJob4Samples
29 (ProtocolStep protocolStep) {
30 jobRequest.getSamples().forEach(
31 sample -> assignTipLiquidTransfer
32 (protocolStep, sample));
33 }
Listing 1: Initial JobCollection via AssayToJobs Visitor 34 LiquidTransferJob liquidTransferJob = null;
35 private void
36 assignTipLiquidTransfer
37 (ProtocolStep protocolStep, Sample sample)
(cf. line 28) and in addition, we connect the sample 38 {
39 DistributeSample distributeSample =
to the TubeRunner for simple reference (cf. line 29). 40 (DistributeSample) protocolStep;
The rules assignToPlate and assignToTrough work 41 if (liquidTransferJob == null ||
42 liquidTransferJob.getTips().size() == 8) {
quite similarly. 43 liquidTransferJob =
Listing 2 shows the handling of assay 44 new LiquidTransferJob();
45 liquidTransferJob
ProtocolSteps. As there are different types of 46 .setProtocolStepName(
ProtocolSteps we use a map of stepAssignRules 47 protocolStep.getId())
48 .setState("Planned")
that provides a special assign rule for each kind of step 49 .setJobCollection(jobCollection)
(cf. line 3, 7, 13 to 24). As an example, ProtocolSteps 50 .setPrevious(lastJob);
51 lastJob = liquidTransferJob;
of type DistributeSample are handled by rule 52 liquidTransferJob
assignLiquidTransferJob4Samples (cf. line 30 to 53 .setSource(sample.getTube())
54 .setTarget(sample.getPlate());
32). Rule assignLiquidTransferJob4Samples 55 }
just iterates through all samples and calls 56 TipLiquidTransfer tip =
57 new TipLiquidTransfer();
rule assignTipLiquidTransfer. Rule 58 tip.setSourceCavityIndex
assignTipLiquidTransfer ensures that a 59 (sample.getTube()
60 .getSamples().indexOf(sample))
LiquidTransferJob is available (cf. line 41 to 61 .setVolume(distributeSample.getVolume())
54). Then, lines 56 to 66 create the corresponding 62 .setTargetCavityIndex(sample.getPlate()
63 .getSamples().indexOf(sample))
TipLiquidTransfer and initialize the corre- 64 .setStatus("Planned")
sponding attributes. Note, line 66 connects the 65 .setJob(liquidTransferJob)
66 .setSample(sample);
TipLiquidTransfer to its sample for easy reference. 67 }
The remaining stepAssignRuleys work similar. 68 ...
Listing 2: Initial JobCollection via AssayToJobs Visitor
4. Reading Changes to Job
1 public class Update {
Executions and Propagate 2
3
private JobCollection jc;
public void update(JobCollection jc, String updates) {
4 this.jc = jc;
5 String[] split = updates.split("\n");
Updating is done via our Update class, cf. Listing 3. 6 for (String line : split) {
7 updateOne(line.trim());
Updates are described by text lines in predefined files. 8 }
new JobCollectionTable(jc)
Our update method calls method updateOne for each 9
10 .expandJobs("job")
line (cf. line 7 and line 14 to 23). Basically, there 11
12
.filter(j -> j.getState().equals("Planned"))
.forEach(job -> removeObsoleteJob(job));
are two kinds of updates, updates that effect a whole 13
14
}
private void updateOne(String change) {
Microplate and updates that effect individual Samples 15 String[] split = change.split("_");
16 String stepName = split[0];
and TipLiquidTransfers. Microplate related updates 17 String states = split[2];
18 if (states.length() == 1) {
are handled by rule updateJob (cf. line 19 and 24 19 updateJob(states, stepName);
20 } else {
to 32). Rule updateJob uses FulibTable code gen- 21 updateSamplesAndTips(stepName, states);
22 }
erated for model specific queries. Line 27 creates a 23 }
private void updateJob(String states, String stepName) {
JobCollectionTable that has one row and one col- 24
25 String jobState = states.equals("S") ?
umn containing the current JobCollection. Line 28 26
27
"Succeeded" : "Failed";
new JobCollectionTable(jc)
does a natural join with the JobCollection and its 28
29
.expandLabware("plate")
.filterMicroplate()
attached labware, i.e. we get a table with rows for 30
31
.expandJobs("job")
.filter(j -> j.getProtocolStepName().equals(stepName))
each pair of JobCollection and Labware. Line 29 32 .forEach(job -> job.setState(jobState));
33 }
removes all rows that do not refer to a Microplate. 34 private void updateSamplesAndTips
35 (String stepName, String states) {
Then, line 30 expands our table to Jobs attached to the 36 new JobCollectionTable(jc)
37 .expandLabware("plate")
Microplates, i.e. we get rows for all possible triples 38 .filterMicroplate().expandSamples("sample")
.forEach(sample ->
of JobCollection, Microplate, and attached Jobs. 39
40 updateOneSampleAndTip
Line 31 filters for Jobs with the right stepName. For 41
42 }
(sample, states, stepName));
each resulting row, line 32 assigns the new state to the 43
44
private void updateOneSampleAndTip
(Sample sample, String states, String stepName) {
corresponding Job. 45
46
JobRequest jobRequest = sample.getJobRequest();
int index = jobRequest.getSamples().indexOf(sample);
Note, our JobCollectionTable query could also be 47 char state = index >= states.length() ?
48 'F' : states.charAt(index);
expressed e.g. using the Java streams API. While using 49 if (state == 'F') {
50 sample.setState("Error");
the Java stream API is quite comparable, the Java stream 51 }
52 TipLiquidTransfer tip = new SampleTable(sample)
API requires some more steps and some extra operations 53 .expandTips("tip")
.filter(t ->
like flatMap and probably some extra type casts. Thus, 54
55 t.getJob().getProtocolStepName().equals(stepName))
we prefer our FulibTables as we consider FulibTables 56
57
.get(0);
if (state == 'S') {
queries to be more concise. 58
59
tip.setStatus("Succeeded");
LiquidTransferJob job = tip.getJob();
Updates with dedicated new states for each sample 60
61
tip.getJob().setState("Succeeded");
} else {
are handled by rule updateSamplesAndTip (cf. line 62 tip.setStatus("Failed");
63 LiquidTransferJob job = tip.getJob();
21 and 34 to 42). Rule updateSamplesAndTip uses a 64 if (job.getState().equals("Planned")) {
65 job.setState("Failed");
FulibTables query to look up all samples attached to some 66 }
67 }
Microplate attached to our JobCollection. For each 68 }
private void removeObsoleteJob(Job job) {
sample we call rule updateOneSampleAndTip (cf. line 69
70 if (isObsolete(job)) {
40 and line 43 to 65). Rule updateOneSampleAndTip 71
72
job.setJobCollection(null);
if (job.getPrevious() != null) {
first retrieves the result state for the current sample (cf. 73
74
job.getPrevious().setNext(job.getNext());
} else {
lines 45 to 47) and updates the sample on failure (cf. line 75
76 }
job.setNext(null);
50). Then the FulibTables query of lines 52 to 56 retrieves 77 }
78 }
the tip that handles the current sample within the current 79 private boolean isObsolete(Job job) {
80 if (job instanceof LiquidTransferJob) {
stepName. Lines 57 to 65 then update the state of the 81 LiquidTransferJob transferJob =
82 (LiquidTransferJob) job;
tip and its job. 83 for (TipLiquidTransfer tip : transferJob.getTips()) {
if (!tip.getSample().getState().equals("Error")) {
Once the updates are propagated, the FulibTa- 84
85 return false;
bles query of lines 9 to 12 of Listing 3 iter- 86
87 }
}
ates through all jobs that are still Planned and 88
89
return true;
} else {
applies rule removeOsoleteJob to them. Rule 90
91
for
(Sample sample : job.getMicroplate().getSamples()) {
removeObsoleteJob calls isObsolete to check, 92 if (!sample.getState().equals("Error")) {
93 return false;
whether the job can be removed (cf. line 70 and lines 94 }
95 }
79 to 96) and and in that case it does a classical removal 96 return true;
97 }
from a doubly linked list. 98 }
}
To be honest, our removal of obsolete jobs iterates 99
Listing 3: Updating the Jobs
through all jobs and thus it is not really incremental. 6. Conclusions
This could be improved by collecting affected jobs during
state changes and by investigating only affected jobs. Overall, the TTC 2021 Laboratory Workflow Case has
However, due to the low number of jobs in the example reasonably simple queries and rules but it also has quite
cases, we do not believe that such a caching mechanism a number of different cases like different kinds of Jobs
pulls it weight and thus we did go for conciseness. and different kinds of Labware that all need special treat-
ment. The Fulib solution addresses these different cases
using maps of rules where appropriate rules are retrieved
5. Results e.g. by the types of current objects. This allows to it-
erate through all tasks, very conveniently. For queries,
In TTC 2020 the Fulib solution used transformation code our solution uses FulibTables, which are quite similar
working directly, with EMF based models [6]. That solu- to Java Streams or to OCL expressions. For the actual
tion was very slow. This, year we use the Fulib generated transformations, we use plain Java code working directly
model implementation. As Table 1 shows, the Fulib imple- on the Java implementation of our model(s). Altogether,
mentation uses an average of 16 megabytes of memory to we consider our solution as easy to read and as quite
handle a case while e.g. the reference solution requires an concise, the whole update transformation needs roughly
average of 46 megabytes. We believe that this reduction 90 lines of Java code.
of memory consumption is a result of the more space The TTC 2021 Laboratory workflow Case is using a lot
efficient model implementation provided by Fulib. of EMF. EMF is the de-facto standard for model exchange
Similarly, the Fulib solution seems to be quite fast: these days. However, as we have discussed compared to
to run all phases of the test minimal case and of all our implementation, EMF has some serious performance
scale_samples cases and of all scale_assay cases on a problems. Using our own implementation (i.e. the Fulib
laptop with Intel Core i7 CPU 3.10GHz and 16 GB RAM code generator) provides us with a more efficient model
we use a total time of about 2 seconds, the Reference implementation, however, we have to implement EMF
solution uses about 5.2 seconds and the NMF solution readers and writers ourselves and we still fight some
coming from the central GitHub repository uses about compatibility issues. These compatibility issues are an-
76 seconds. To us it seems that EMF is a performance other reason for our integration into the Python based
bottleneck. test framework: we need to write EMF files that are not
Tool total time (millisec) avg. memory (mb) just EMF compatible but that are accepted by the test
framework. We will improve our performance on EMF
Reference 5227,95 46,45 compatibility for next years TTC.
NMF 76795,22 319,62
Fulib 1999,54 16,09
Table 1 References
Measurements
[1] A. Zündorf, S. Copei, I. Diethelm, C. Draude, A. Kunz,
U. Norbisrath, Explaining business process software
Concerning correctness, we have had difficulties to with fulib-scenarios, in: 2019 34th International Con-
get the Python script working that runs the checks and ference on Automated Software Engineering Work-
does the analysis of the measurements. Our solution has shop (ASEW), IEEE Computer, 2019, pp. 33–36.
been developed on Windows and the Python installa- [2] fulib, Fulib web service, https://www.fulib.org/, 2019.
tions we tried did not work. A Docker image with the [3] ttc2021labworkflow, Ttc2021 case: Incremen-
correct Python version and the correct libraries would tal recompilation of laboratory workflows,
have been a great help. Concerning completeness, we https://www.transformation-tool-contest.eu/
did not implement updates that generate new samples 2021_labflows.pdf, 2021. Last viewed 25.05.2021.
on the fly. We just did not understand how these new [4] J. Cabot, M. Gogolla, Object constraint language (ocl):
samples shall be added to a running JobCollection: a definitive guide, in: International school on formal
are you allowed to add new samples to an existing plate? methods for the design of computer, communication
Or does each new sample need a new plate? Or can you and software systems, Springer, 2012, pp. 58–90.
add samples to plates as long as those plates are not yet [5] E. Gamma, Design patterns: elements of reusable
under processing? But when does the processing of a object-oriented software, Pearson Education India,
plate actually start? You find our solution on: 1995.
Github: https://github.com/sekassel/ttc2021fuliblabworkflow
[6] S. Copei, A. Zündorf, The fulib solution to the ttc
2020 migration case, arXiv preprint arXiv:2012.05231
(2020).