This paper reports on an ongoing Knowledge Graph generation project at the industrial automation company Festo. The project aims at establishing a well de ned and easily traceable extract, transform, load (ETL) process of technical data on automation products into Festo's semantic data platform (FSP) to serve complex con guration tasks. Starting with a brief illustration of the data sources and requirements we describe a staged Knowledge Graph generation process that draws on common data extraction techniques as well as on speci c semantic transformation methods such as the mapping framework R2RML or ontology reasoning. We will report on the project status as well as discuss our design decisions towards a reusable FSP Knowledge Graph data ingestion process.
Festo is a multinational industrial control and automation company based in
Germany. The Festo Semantic Platform (FSP ) is a major initiative at Festo to
create and provide a Knowledge Graph and reasoning-based services about
products and solutions for Festo's core business: factory automation. For instance,
tasks such as the computation of compatibility of product components within
electrical drive trains is provided by the FSP infrastructure using ontology
reasoning and SPARQL querying [
Our current solution is structured as an ETL process. We use this approach as our data undergo a series of sequential processing and re nement steps that include materialisation of reasoning results and addition of expert data before the Copyright ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
nal result is delivered. Ontology based data access (OBDA) [
To better deal with this issues and to establish a well de ned Knowledge Graph ETL process, Festo has recently started an initiative to improve the Knowledge Graph generation. The following list summarizes the requirements for the FSP Knowledge Graph generation pipeline currently in progress: Change detection and propagation: The FSP services rely on data that is typically distributed over various source databases that are managed by processes that vary in their level of reliability and their update cycles. In the general use case of technical product data, the ETL process has to integrate data sources of di erent quality. There is well maintained data from SAP about products that have reached a certain development maturity or are already on sale. In addition there is data about upcoming products from a database maintained by product development. Data in the latter database may change without any traceable history or noti cation of any of its data consumers. Therefore, it's a key requirement for our Knowledge Graph generation pipeline to incorporate a data change detection and tracing mechanisms that can detect data changes as early as possible in the data sources to support Knowledge Graph debugging.
Tracking and logging: The FSP Knowledge Graph generation should follow a step wise processing pipeline determined by its functional demands and with a clear speci cation of its processing blocks and data interfaces. Furthermore, each of the processing blocks has to report about its outcome, i.e., whether it has terminated successfully, and provide statistics on the results. Ideally there is a test and evaluation step after each of the processing blocks that checks for unexpected results. All the reporting should be accessible in a way that supports building a dashboard that allows to operate and monitor the ETL process to discover potential problems as early as possible. Declarative approach: Since the FSP is committed to Semantic Technologies, the ETL logic should be as declarative as possible instead of being buried somewhere in program code. In other words, we prefer to have data classi cation done by ontological rules and reasoning as well as data transformation based on mapping rules rather than by procedures, table functions, etc. Based on standards: Whenever possible the Knowledge Graph generation pipeline should employ standards that have shown to work in industry, are widely adapted in the community and for which tools and engines are available. This refers in particular to the data classi cation and transformation part. Since the FSP already uses OWL 2 RL reasoning and SWRL, the ETL process should adapt to this fragment and syntax formats. For the transformation part, we plan to evaluate mappings and tools from relational database tables to RDF datasets such as R2RML or RML. 2
Our target data processing pipeline consists of several processing stages that are performed on a microservice-oriented architecture as shown in Figure 1. In the following we describe the stages of this process.
SQL
SQL + Java (or the like)
The pipeline is in charge of transforming product data as well as data on complex automation systems into a Knowledge Graph for a given ontology schema. A typical product in the product domain is an electric axis. An axis can move, push, or press a work-piece. An axis furthermore is driven by a motor which in turn is actuated by a controller. The whole chain from axis to controller is called a drive train. It also incorporates a mounting kit that allows to mount the motor to the axis and optionally a gear. The ESBF axis3 family is a such a product tho which we refer in our running example in the following.
Even if the number of products is few in numbers with some thousands the ETL process to generate a Knowledge Graph of high quality is a business critical 3 https://www.festo.com/cat/en-gb_gb/data/doc_ENGB/PDF/EN/ESBF_EN.PDF Change Data Capture
Import Selection
Mapping dev DB SAP
Importing component data from relational
DBs
Import DB
Selection of component data (type and time)
IDgeneration
Assigning attributes
Inserting relations DB FSPIDs ontology schema (R2)RML
Component ontology
Enrichment classifica
tion RDF/OWL element since it produces the source data of subsequent semantic processing steps. As an example, based on the basic products and data about technical compatibility we compute all nearly 100 million technically valid drive trains. 2.1
Due to the fact that one of our main data sources is an un-revisioned database, we have to deal with new, deleted or updated product data during Knowledge Graph ingestion. We address this during the Change Data Capture phase.
Technically, this is delivered by a layer of SQL databases, called \Import DB"
in Fig. 1, that are in our sovereignty. We employ type 2 of the Slowly Changing
Dimensions (SCD) method known from Data Warehousing [
On request of the processing pipeline the Change Data Capture stage starts checking for updates in the source databases by comparing with the current data records in the import database. The latter will receive new rows for any detected change to re ect the historical state of the source database.
ID VALID-FROM 1 2019-01-08 1 2019-02-10 2 2019-01-01 3 2019-01-01
VALID-UNTIL TYPECODE 2019-02-09 ESBF-BS-32-%%-5P 2099-01-01 ESBF-BS-32-%%-5P 2099-01-01 ESBF-LS-32-%%-2.5P 2099-01-01 ESBF-LS-32-%%-2.5P-S1
MAX VELOC MAX LOAD ACCUR 0.55 100.0 0.02 0.55 100.0 0.01 0.05 60.0 0.05 0.05 60.0 0.05 Table 1 depicts an example of our import database for axes. It contains some sample axes of the ESBF family and a fraction of its technical data. For the axis ESBF-BS-32-%%-5P (with ID 1) the import database has two entries which account for a data update. More precisely, the axis accuracy (last column labeled ACCUR) changed on February 10th to the new value \0.01". The corresponding row represents the current valid data record for this axis since it holds the prede ned surrogate high date in VALID-UNTIL.
In addition to capturing data changes by applying the SCD method, the import database allows us to presort the data. As a result, the import database follows the design principle, that every table corresponds to a product category that is also represented by a class in our ontology schema about drive trains. Therefore, every row in the table is designed with the intention to represent an individual with its technical data properties in the Knowledge Graph. Other tables represent information about relationships between individuals.
To this end, the import database serves two main purposes. It allows to track changes in our source data on the one hand, and on the other hand it makes the translation to a Knowledge Graph easier since it's a step towards a graph of individuals and relationships. 2.2
The main task of this stage is to select product data from the import database and to add unique identi ers to this data in preparation for a mapping into a Knowledge Graph format.
When considering an initial full load, this stage selects the most current records from the import database. These records are then aggregated into tables such that these tables roughly correspond to product classes of the ontology schema. In the domain of electric drive trains these are axis, mounting kit, gear, motor, or controller plus accessories etc. Due to the history of data changes in the import database of the previous step, the ETL process can also select previous data versions of particular products. For instance, it allows to select the latest data for all products except for axes for which we can choose to have data from a particular point in time. As mentioned above, this is key to generate reasonable Knowledge Graphs from data with varying level of maturity.
ID TYPECODE A000576 ESBF-BS-32-%%-5P A000628 ESBF-LS-32-%%-2.5P A000629 ESBF-LS-32-%%-2.5P-S1
MAX VELOC MAX LOAD ACCUR 0.55 100.0 0.01 0.05 60.0 0.05 0.05 60.0 0.05
For each of these domain objects a unique identi er is required. Such an identi er is important since Knowledge Graphs need a key to distinguish instances from each other. The management of identi ers is provided by an FSP service. For electric drives the FIDGET (Festo ID Generator) service is the central authority for looking up an existing or generating a new identi er. Typically, an identi er is computed from a set of technical characteristics (basic products) or its components in case of a complex system (e.g. a 3D portal). To facilitate debugging the FIDGET service keeps its identi er constant over time. Therefore, we can easily compare Knowledge Graphs generated at di erent times to check for changes on component level. Table 2 depicts an import selection from the import database with the latest technical data and identi ers obtained from the ID service. The mapping step involves the transformation of relational data into instances of the Knowledge Graph, instance attributes and relations between those instances. The target format for the FSP component ontology is any valid syntax speci ed by W3C for RDF or OWL. The mapping step is deliberately designed to be accomplished by a pure declarative transformation not incorporating any procedural data manipulation parts. This is possible because of the preceding import selection step, that has aggregated and aligned the source data appropriately as well as added unique identi ers suitable to serve as IRIs. As a result, this step can be carried out by a R2RML mapping speci cation executed by an R2RML engine. This allows to purely draw on W3C's R2RML which hopefully provides exibility in the selection of tools and engines to choose the optimal deployment option with respect to performance, data load, etc.
The following exemplary R2RML mapping shows how technical data is transformed into a RDF-based Knowledge Graph: <#TriplesMap1> rr:logicalTable [ rr:tableName "import-selection" ]; rr:subjectMap [ rr:template "http://www.festo.com/edrive#{ID}"; rr:class fsp:Axis;]; rr:predicateObjectMap [ rr:predicate rdfs:label ; rr:objectMap [ rr:column "TYPECODE" ]; ]; rr:predicateObjectMap [ rr:predicate fsp:max-velocity ; rr:objectMap [ rr:column "MAX_VELOC"; rr:datatype xsd:double ]; ]; rr:predicateObjectMap [ rr:predicate fsp:max-loading ; rr:objectMap [ rr:column "MAX_LOAD"; rr:datatype xsd:double ]; ]; rr:predicateObjectMap [ rr:predicate fsp:repetition-accuracy ; rr:objectMap [ rr:column "ACCUR"; rr:datatype xsd:double ]; ]. <#TriplesMap2> rr:logicalTable [ rr:sqlQuery """SELECT ID FROM import-selection
WHERE REGEXP_LIKE(TYPECODE, .*-S1.* ) = 1 """ ]; rr:subjectMap [ rr:template "http://www.festo.com/edrive#{ID}"; ]; rr:predicateObjectMap [ rr:predicate rdf:type; rr:objectMap [ rr:template "http://www.festo.com/edrive#IP65" ]; ].
The rst part of the mapping simply associates a label as well as technical data to a Knowledge Graph instance that is identi ed by an IRI compiled from the given identi er. The second part is an example of a typecode based instance classi cation. The latter mapping checks for a particular typecode fragment (\S1") that indicates a particular protection level (IP65: dust and water protected). The typecode chart for ESBF axes is shown in Fig. 2.
All Festo products are classi ed according to typecodes. Not all of these are relevant within the applications supported by the FSP Knowledge Graph. The R2RML mapping therefore just takes care of those which are required for later reasoning tasks.
In a brief analysis we evaluated the R2RML resp. RML editors KARMA [
As mentioned above we are interested in applying a standard mapping approach not making use of non-standard extensions such as procedures etc. Ideally, we want to specify the business logic fully within the mapping language. Further processing should either in the previous or following steps. For those cases where R2RML does not provide enough mapping expressivity we decided to use SWRL in the post-R2RML step (see next section).
Although none of the evaluated (R2)RML editors failed for our purpose we are not using one of them. Instead we are currently add annotations to your ontology schema that allows us to generate the R2RML mapping above via transformation from these annotations. The huge advantage of this approach is that it incorporates this information into the key place of metadata about the Knowledge Graph, namely the ontology schema.
The engine should then be able to run the mapping speci cation independently from front ends in order to be able to execute it from a work ow engine. 2.4
The goal of the enrichment steps is to enhance the Knowledge Graph by making implicit information explicitly available with the help of OWL reasoning.
A key task in this respect is the categorization of Festo components according to product families or technical characteristics that are relevant for the subsequent services. Some classi cation tasks are already done at the mapping stage based on a typecode decomposition with the help of regular expressions (compare with the protection level IP65 classi cation above). Other categorization tasks that do not directly relate to a typecode feature. They are handled via OWL reasoning either based on OWL axioms or SWRL. For instance, components suitable for food manufacturing have to comply to di erent protection classes depending on their component type (a controller typically requires a lower protection class because it is less close to food in comparison to an axis). Another example relates to information that need to be propagated through the Knowledge Graph. For instance, the size of a drive train originate from the respective size of its axis which is automatically derived by an OWL axiom.
After enrichment the result data of the ETL are ready to be used in the core
part of our semantic platform. For example, instance data about axes, motors,
and controllers are combined with background knowledge that is described in
OWL and SWRL. A reasoning step performed by RDFox [
Our original system was conceived as a set of standalone applications. As system
complexity and availability requirements grow, we need to align our system with
software engineering and continuous delivery [
We use Java as programming language. For micro services there is no xed framework. However, in our rst e orts we use the OpenAPI 3 standard to specify our REST interfaces and use Swagger to generate Spring Boot service stubs. Our codebase is maintained in Git repositories. As build system we use Maven. Builds are triggered and monitored using JetBrains TeamCity. To reproduce the system environment we package the artifacts in Docker containers and use Docker Compose to run multi container applications. Maven artifacts and Docker images are deployed to the company internal Artifactory repository.
To orchestrate the execution of the processing steps and monitor the progress we need a work ow solution.
The work ow system should allow automated execution and scheduling of the execution steps. This includes for example regularly checking the database for changes and trigger the mapping process. Information of the current state of processing and failure states should be easily retrievable. Processing state could for example indicated by means of graph visualization of the work ow. For other metrics such as number of generated instances of a speci c concept a dashboard could be helpful.
We currently consider Apache Air ow4 as a platform. It provides to programmatically author, schedule and monitor work ows. It provides templates for common work ow steps such as SQL or REST calls. Custom processing steps can be implemented in Python. 5
In this work we described our ongoing e ort to re ne the data processing pipeline of the Festo Semantic Platform. The system is currently used productively at Festo.
In our target architecture, the processing pipeline is structured into four processing phases | change data capture, import selection, mapping and enrichment. We are in the process of aligning our processing artifacts to this architecture. For parts of the system we plan to evaluate the technology. We identi ed (R2)RML as a promising technology and started evaluating tools and implementations. However, an upfront evaluation of the overall approach remains a challenge.
The Festo Semantic Platform will evolve over time as new knowledge domains are added and further company systems are integrated. In order to gain more exibility to restructure the processing ow when requirements change we rework our services as microservices and move to container based application management. To achieve transparency of the processing we intend to introduce a work ow control system and monitoring. 4 https://airflow.apache.org/