The purpose of the Semantic Web is to enable worldwide access to humanity's knowledge in a machine-processable way. A major obstacle to this has been that knowledge is often either represented in an incoherent way, or not externalized at all and only present in people's minds. Populating a knowledge graph and manually building an ontology by a domain expert is tedious work, requiring great initial effort until the result can be used. As a consequence, knowledge will often never be made available to the Semantic Web. The aim of this project is to develop a new approach for building ontologies from implicit user knowledge that is already present, but hidden in various artifacts like SQL query logs or application usage patterns.
In the last decades, the World Wide Web indisputably changed human society and economy. Computers, paradoxically, although essentially operating the Web, cannot make use of it on their own. Knowledge on the Web is represented mostly in a way suitable for humans, as pages containing plain text and graphics. Although web pages can be structured hierarchically and linked with each other, their inherent semantics are only accessible by a human being perceiving the content. Querying the Web is usually restricted to simple keyword-based search engines or web services with proprietary APIs. Apart from these technical obstacles, the Web also does not define coherent sets of terms that shall be used to describe concepts and entities of a particular domain, leaving that tasks to human interpretation.
The Semantic Web [
For being able to actually interpret RDF data, an ontology must be defined. This can be done either in RDF Schema or in the Web Ontology Language (OWL). In a nutshell, an ontology is a set of axioms which constrain what statements can or cannot be true and allows to deduce new statements from existing statements. Creating these ontologies manually is tedious work and therefore a blocker for Semantic Web adoption.
Knowledge already exists somewhere, either in people’s minds or in various kinds of artifacts: semi-structured file formats like CSV or JSON, plain text in natural language, applications source code, log files, or SQL queries. Transferring all this knowledge by hand into a graph is time-consuming and expensive, wherefore this can be applied only for limited use cases. Developing an at least partly automated method to perform this task could drastically lower the costs for deploying Semantic Web techniques. 2
This PhD research project will extend the scope of the previous master thesis
project Pharos, which results have been published in a followup research paper
[
For example, joining two tables over an attribute indicates that the data analyst probably identified a foreign-key-relationship, otherwise he would not have made that join. Renaming columns with speaking names or explicit type casts give hints about their meaning. s e l e c t sum( p . s a l a r y ) , dep . id , dep . name from p e r s o n p jo in department dep on p . dep_id = dep . i d where dep . l o c a t i o n= ’DE ’ or dep . l o c a t i o n = ’FR ’ group by dep . id , dep . name order by dep . name ;
Listing 1.1. Example query with partial schema information name
id has_attribute has_attribute
has_attribute joined_with
department has_attribute
has_attribute location equals equals ’FR’ is_type string salary person dep_id
’DE’
We had decided to build a knowledge graph from SQL query logs that could be used to help understanding the mental model behind data sources. A prototype was written that demonstrates the feasibility of the approach. It was implemented in form of a JDBC proxy driver that can capture and analyze all SQL queries a Java application sends to a SQL query engine like Apache Drill, allowing a minimal-invasive deployment of the prototype into existing workflows, as it is compatible with any software using JDBC drivers. The prototype was evaluated using a test database with a known schema and a set of test queries, based on exercises of our introductory database lecture. 3
The resulting knowledge graph describes how data sources have been used by analysts, but does not describe the semantics of the data sources themselves, like value constraints or foreign key relationships. Human interpretation of the results is required to gain insights about the used data sources. Therefore, the next level will be to perform automatic reasoning on top of the knowledge graph. This requires to generate an (incomplete) ontology for describing the semantics of data sources. As knowledge derived from SQL queries may be contradictory when the query log contains queries that are not conforming to the underlying schema - an approximate approach is needed to deal with this ambiguity.
Queries do not reflect the semantics of a data source, but the mental model a data scientist has made of it. Analyzing these mental models can already give valuable insights. For example, if someone uses a “grade” attribute and compares it with values that are not present in the dataset, the explanation could be that the user originated from a country with a different school grading system. There are multiple concepts out there about what a “grade” should be, a query-driven approach could provide more transparency about these concepts as an intermediate step.
When analyzing SQL query logs, there will always be queries that are based on wrong assumptions about the schema, especially if the origin of the query log is from an interactive session where queries with undesired results may be rewritten. With multiple query logs from different sessions and users, finding the similarities in their behavior and their mental model could lead to the intended semantics of the used data sources.
A self-learning system shall be developed that makes suggestions to data scientists about suitable sources or queries they may find helpful for their task. Based on their given feedback and performed queries, the system shall incrementally approximate the true semantics behind the data sources.
Thus far, only SQL query logs were considered as a source for query-driven schema inference. But there are other types of queries to consider, like query strings from search engines, application usage patterns extracted from graphical analysis tools or even source code from programs accessing a data source. Other query languages like XQuery or languages from various NoSQL database systems could be included. The approach does not depend on a specific language. 4
Many approaches for schema inference are data-driven, using data profiling
methods to reconstruct the underlying schema of a given dataset. A significant
example is the Metanome project [
The existing prototype will be extended to use Semantic Web reasoning techniques to deduce the meaning of a data source by the knowledge extracted from query fragments. A framework of rules should be defined to achieve this, possibly with the Rule Interchange Format (RIF). This prototype shall then be tested on real world query logs, so the resulting knowledge graph can be compared with the actual schema the data sources are based on. A supplementary user study will show if the software is able to enhance the workflow of data scientists to understand heterogeneous data sources.