Machine learning typically requires training and validation of models with large and heterogeneous datasets. The engineering of these datasets is a critical task for enabling high accuracy and generalization, although in many cases it is done following an ad-hoc approach. Hyperknowledge can enable more structured engineering of datasets, by representing the datasets' symbolic and non-symbolic information, within the same framework, and enabling queries for dataset creation, retrieval, resampling, and combination. In this poster paper, we present how the Hyperknowledge Platform evaluates those queries and analyze its performance quantitatively. The preliminary results indicate that our platform can support data scientists' work while adding negligible time overhead.
As the popularity of Machine Learning (ML) tasks increases, so does the amount
of data used to train and test them. The e ectiveness of such algorithms is related
to the quality and variety of the data applied during the training stage [
Hyperknowledge Platform
The central component of the HKW Platform, depicted in Figure 1, is the
HKBase that stores and maintains instances of HKW models. It has a exible
internal architecture to which components can be coupled through the
implementation of one of its APIs (IObserver, IDataBase, and IReasoner).
The functionalities provided
by HKBase are exposed to
applications through its
RESTful API. The current HKBase
implementation uses two
different data sources, one for
storing data structured
following the HKW model and
another for storing
unstructured content, which can be
used for storing datasets. It
also contains both the query
and reasoning engines that
will be used for processing Fig. 1. HKW Platform architecture [
The HyQL engine's main components are its grammar 2 and its processing engine. The processing steps followed by the HyQL processing engine are summarized in Algorithms 1 and 2. In both algorithms, the variable insertClauses will be used to store the set of pairs (c; E) such that c is the target context and E is the set of entities to be inserted in c. Meanwhile, the variable updateClauses will store the set of tuples (c; E; C; P ) such that c is the target context, E is the set of entities to be inserted in c, C is the set of contexts to be generated by the SPLIT function, and P is the set of proportion values associated with each context in C. Finally, in Algorithm 1, the variable R stores the set of entities/rows to be retrieved by the query.
The execDEClauses function logic is described in Algorithm 2. This function receives the insertClauses and updateClauses variables, passed as parameters to the call performed in line 13 of Algorithm 1, to identify which contexts and reference nodes 3 must be created in HKBase as a result of the query. 2 Accessible at https://ibm.box.com/v/iswc2021-hyql-grammar. For clarity, we have suppressed the handling of spaces, comments, and lower case keywords. 3 Reference node is a special type of HKW node that promotes the reuse of nodes across multiple contexts.
Algorithm 1: HyQL engine main processing steps.
Algorithm 2: Dataset engineering clauses execution.
Input: HyQL Query Q 1 insertClauses = ;; updateClauses = ;; R = ;; 2 subQueriesObjects = P arseQuery(Q); 3 foreach subquery object sQ of 1110654789 subiisssssmQffQQQQQssgueussQQreIURRDpeQQgtdrRDea=eUI=U=aist)aesptua;6=(setQ6=gd(slaOsCt=eaQR;Q)S;ttbl;cReaejFtcw;tChteu;ehesicsesslt=tQetaQunehcnsEushlECRat;dsDa)sd;ve;do(dsa=a(sdQstlQsuaQgsPQR(aeQ,;stItesQUsIesQQtCn)uot;sREoloeitn))nSr;st=setCetr,rlaatsiCuQnslt)ase;u((sssQQes,,; 12 end 13 execDEClauses(insertClauses; updateClauses); 14 return R To evaluate our work quantitatively, we have measured the median execution time the HyQL engine spends executing some of the queries related to DE tasks. The main goal of this evaluation is to answer the following research questions (RQs): (RQ1) Is the time required to answer DE queries within our platform lower or in the same order of magnitude as the time consumed by other data science tasks, such as model training and raw le transfer time? (RQ2) Are there meaningful di erences in response times across HKBase con gurations?
To answer those RQs, we designed an experimental dataset based on the
Pascal VOC2012 [
Based on this dataset we designed 11 experimental queries, presented in Table 1, that include 4 SELECT queries, 4 INSERT queries based on the results of SELECT queries, an UPDATE query that splits data into multiple contexts, a SELECT query that uses spatial operators for ltering results, and an INSERT query based on this SELECT query.
In total, 3 HKBase con gurations, presented in Table 2, were evaluated. Evaluating multiple combinations of HKBase data sources, and HyQL engine implementations, helps us understand which con gurations perform better for each type of query and allows us to nd the optimal setting for a given scenario.
The Generic HyQL engine implementation can work with any data source supported by HKBase as all query processing is performed using the standard data source API (IDB) while the SPARQL-based HyQL engine optimizes the processing of HyQL clauses in Jena by translating them directly into a single SPARQL query with multiple subqueries.
The hardware in which Table 1. HyQL queries designed for experiments.
these experiments were exe- ID HyQL Query
cuted was composed of two QQ21 SSEELLEECCTT tdraatiansientg set
dedicated Dell OptiPlex 7050 Q3 SELECT test set
machines 4. Half the repeti- Q4 SELECT distinct image WHERE image has cat
tions for every con guration Q5 IVNASLEURETSI(NSTEOLEmCyTDdaatatsaestest)
were executed on each machine Q6 IVNASLEURETSI(NSTEOLEmCyTTrtaraininininggSestest)
to guarantee a fair compari- Q7 IVNASLEURETSI(NSTEOLEmCyTTetsetsStestest)
son. The software setup of both INSERT INTO myCatImages
machines was also the same 5. Q8 VALUES (SELECT distinct image
eTxhpeersimcreipnttss wuesreed i mtoplermunenttehde (UWSPPHDLEAIRTTE(Eimimdaaagtgea,ese[h"taismsAacPgaetP)LtrYaining set",
iCnomPpyotsheonto adnedplouysedandDorcekseert Q9 ["G0iRm:6O5a;gU0eP:0v5aB;liY0d:a3itm]i)oanges)et", "image test set"],
the service containers. Both the Q10 SAENLDECimTagimeahgaes WcaHtAERNED ipmearsgoenhcaosnptaerinsosncat
server and the client script ran INSERT INTO myCatAndPersonImages VALUES
on the same host machine. For Q11 (SELECT image WHERE image has person
running the queries, the dataset AND image has cat AND person contains cat)
was loaded once and all the Table 2. HKBase con gurations evaluated.
queries were executed in se- Con guration Description
quence, following their numeri- HKGBe+nerMiconHgyoQDLB + anHdKGBeanseerwicitHhyMQoLnegnogDinBe.
cal order. All repetitions of each HKB + Jena + HKBase with Jena and
query were also executed in se- HGKenBer+icJHenyQaL+ HGKenBearsice HwiytQhLJeennagianned.
quence and after each repeti- HyQLtoSPARQL SPARQL-based HyQL engine.
tion of queries with INSERT
and UPDATE clauses, the execution script removed all the entities generated by
the previous repetition to preserve the IID assumption [
A total of 100 query repetitions were executed on each machine, totaling 200
query repetitions. As highlighted by the 95% con dence intervals (vertical black
lines on the upper edge of each bar) in the bar plots of Figure 2, these were
su cient to allow distinguishing which con gurations were performed better for
each query. These preliminary results indicate that the time overhead required
for using our system is minimal when compared to time spent in typical data
science tasks, such as: (1) tting a model to the data (training), which can take
multiple days or even months; (2) managing datasets manually, which currently
consumes a high proportion of data scientists' work hours [
Therefore, we conclude that the answer to the RQ1 is that the requirements posed by our solution can be up to several orders of magnitude lower than the requirements posed by other data science tasks, introducing an overhead of at maximum a couple of minutes required for the retrieval, creation and/or integration of datasets using our system, while removing the need of performing several tasks manually. Also, with regards to RQ2, the results indicate that HKB + Jena + HyQLtoSPARQL con guration presents lower response times. Q1
Q2
Q3
Q4
Q5
Q6 Query
Q7
Q8
Q9
Q10
Q11
Conclusion In this paper, we presented how the HKW Platform enables semantic queries for dataset engineering. The main contributions of this work are the proposal, implementation, and evaluation of the HyQL engine extensions required for supporting these queries. Data scientists could use this solution to nd new datasets, balance, clean, and resample them, and select speci c features, which saves time and promotes better data exploitation. The quantitative analysis of the HyQL engine has indicated that the queries discussed in this work can be evaluated in a reasonable amount of time. When we compare the results obtained in our preliminary analysis with the time currently spent for DE and ML tasks, we conclude that the overhead introduced by our approach is negligible.