2. Approach
Use Case: Distributed Semantic ETL. In our welding use case, large amount of data collected
from diferent factories, customers, software versions are integrated and analysed in parallel
to optimise the following welding production. To enable distributed ETL, we need to find a
strategy that makes the ETL parallelisable. SemCloud achieves this by breaking down the ETL
into pipelines of four steps: retrieve, slice, prepare, and store (Fig. 1), where data is first retrieved
from databases or online streams, and then split into subsets (e.g. each belong to one welding
machine) by slice to achieve parallel processing and storage in the following two steps. Here
the cloud configuration play an important role.
KG Construction for ETL Pipelines. SemCloud provides the users GUI to construct semantic
ETL pipelines and encode them into knowledge graphs, based on a SemCloud ontology
(Figure 1a). The ontology SemCloud [
3
] is written in OWL 2, and consists of 20 classes and 165
axioms. It has three main classes: DataEntity, Task, Requirement. DataEntity refers to any dataset to
be processed; Task has sub-classes that represents the four types of tasks in the data preparation:
retrieve, slice, prepare, and store; and Requirement that describes the requirements for computing,
storage and networking resources. We illustrate the generation of ETL pipelines in KGs with
the example in Figure 1b. For these data, the users construct an ETL pipeline p1 with four layers
(via GUI). Firstly, data are “retrieved” from the welding factories. Thus, the layer l1 is of type
RetrieveLayer, and has the task t1 of type Retrieve. The task t1 has an IO handler io, which has
an output d1 of type DataEntity. Then the data are read in by a task t2 of type Slice, and “sliced”
into smaller pieces d2, d3. These slices are input to diferent computing nodes to do tasks t3
and t4 of type Prepare. Finally, all prepared data entities are stored by t5 of type Store.
Datalog ML Rules. Obtaining an optimised cloud configuration is not a trivial task. Cloud
experts typically try diferent configurations by testing the system with various settings and use
heuristics to manually decide on the configurations. To this end, SemCloud uses adaptive rules
in Datalog with aggregation and calls to external predicates learned by ML (they are adaptive
because the function parameters are learned). In particular, we consider non-recursive rules of
the form ← 1, . . . , , where is a head of rule (the consequence of the rule application)
and 1, . . . , are either predicates that apply join, aggregate function that filters out the
results or the expression of the form Var = @FUNCT(Vars). For the theory of Datalog we refer
to [
4
]. We have six set of independent Datalog rules that are divided into three steps.
Graph Extraction Rules. These rules populate the predicates so that these predicates will be used
for the resource estimation and configuration. The rule0 exemplifies populating the predicate
subgraph1 that is related to the ETL pipeline p. Similarly, rule1 creates subgraph2(p,n,v,ms,mp
,ts,tp,nc,ns,mrs,mrp,mode).
subgraph1(p,n,v,ms,mp,ssl,spr,sst) ← ETLPipeline(p),
hasInputData(p,d), hasVolume(d,v), hasNoRecords(d,n)
hasEstSliceMemory(p,ms), hasEstPrepareMemory(p,mp)
hasEstSliceStorage(p,ssl), hasEstPrepareStorage(p,spr)
hasEstStoreStorage(p,sst)
Resource Estimation Rules. These rules are used to estimated required resource assuming one
computing node. For example, rule2 estimates the required slice memory (ms), prepare memory
(mp), slice storage (ssl), prepare storage (spr), and the store storage (sst). It then stores these
estimation in the predicate estimated_resource.
(0)
(2)
where @func_ms, @func_ssl, @func_sst, etc. are parameterised ML functions whose parameters
are learnt in the rule parameter learning. In the implementation, those are defined as external
functions that are called in the grounding phase of the program and are replaced by concrete
values [
5
]. We also have other estimation rules for other resources, such as CPU consumption.
Resource Configuration Rules. These rules find the optimal cloud configurations based on the
estimated cloud resource. 3 is an example for deciding the slicing strategy and the storage
strategy, and finding the optimal resource configuration such as the chuck size ( nc), slice size
(ns), memory reservation for slice (mrs) and for prepare (mrp). In essence, 3 stipulates that if
the maximum of estimated slice memory (ms) and prepare memory (mp) is greater than a given
threshold (c1*nm), and the maximum of estimated slice storage (ssl), prepare storage (spr),
and store storage (sst) is smaller than (or equal to) another threshold (c2*ns), then the chosen
strategy for the given pipeline is slicing (thus nc and ns are computed), and fast storage (fs),
where the thresholds are calculated from cloud attributes.
configured_resource(p,nc,ns,fs,mrs,mrp) ←
subgraph2(p,n,v,ms,mp,ts,tp,nc,ns,mrs,mrp,mode),
estimated_resource(p,ms,mp,ssl,spr,sst),
CloudAttributes(c,c1,c2,c3,nm,ns,fs,cs),
#max{ms,mp} > (c1 * nm), #max{ssl,spr,sst} <= (c2 * ns),
nc = @func_fs_1(n,v,ts,tp), ns = @func_fs_2(n,v,ts,tp),
mrs = #min{ms, #max{@func_ss(n,v,nc,ns), c3*ms}},
mrp = #min{mp, #max{@func_pn(n,v,nc,ns), c3*mp}}
(3)
Rule Parameter Learning with ML. The functions in the adaptive rules are in the form of ML
models. The resource estimation rules are selected from the best model resulting from training
three ML methods and the pilot running statistics. These three ML methods are Polynomial
Regression (PolyR), Multilayer Percetron (MLP), and K-Nearest Neighbours (KNN). We selected these
three methods because they are representative classic ML methods suitable for the scale of the
pilot running statistics. The resource configuration rules are trained with the three ML methods
and with optimisation techniques, such as Bayesian optimisation or grid search. For example,
the functions @func_fs_1 and @func_fs_2 that find the optimal chuck size ( nc) and slice size (ns)
are trained by finding the arguments of ( nc,ns) for the minimal total computing time (total):
nc, ns = arg min total = arg min (v, n, nc, ns, slice, prepare)
nc,ns nc,ns
3. Implementation and Evaluation
Implementation. We implement the Datalog rules with DLV and external functions as Python
plugins [
6
]. In particular, we first write and compile the external functions written in Python,
then we define the interface for the functions in the Datalog program, and then link the Datalog
program with the compiled Python code when running the DLV. For some complicated rules
with reusable part, we introduce a auxiliary predicate that stands for the reusable to improve
eficiency. A rule in the form of ← 1, ..., becomes two parts: (PartI): ← 1, ..., ,
(PartII): ← , +1, ..., , where PartI is reused also in other rules. For instance,
3 applies for one of the four cases of #max{ms,mp}>(c1*nm), #max{ssl,spr,sst}<=(c2*ns
), while there exist other three cases for the comparison (<=,<=), (<=,>), (>,>). We introduce
the auxiliary predicate configured_resource_aux to replace the first three lines ( subgraph2 to
CloudAttributes), which will be reused in the inference of the three other cases.
Evaluation and Discussion. To verify the time eficiency and accuracy of the rule parameter
learning and inference, we run SemCloud repeatedly 3562 times and gather pilot running
statistics. These statistics are split to 80% for training and 20% for testing and inference. Three
ML models are trained and tested. After a grid search, the selected hyper-parameters are,
PolyR: 4 degree; MLP: 2 hidden layers with neurons 10 and 9; KNN : 2 neighbours. We use these
performance metrics: normalised mean absolute error (nmae), minimal training data amount
(Min. || for yielding satisfactory results, optimisation time (Opt. time), learning time (for
ML training) and inference time (including the inference time of ML and Datalog). The results
(Table 1) show that PolyR has the best prediction accuracy, requires the least training data,
and consumes the least time. Therefore, PolyR generates the best results and is selected for the
use case. We presume the reason is that PolyR works better with small amounts of and not very
complex data (3562 repeated running statistics). The results show that our approach exhibit
promising inference accuracy and time eficiency for automated cloud resource configurations.