The research plan described in this article is intended to develope a system which can help data curators, data scientists, and other users in the domain of Linked Data to identify important data sources, understand their structure, and their schema. In addition, the system should be easy to use for non-expert users so that they can quickly and easily formulate more complex queries, e.g by using a visual interface. Furthermore, Linked Data Federations will be extended to include Web APIs as knowledge bases, denoted as Hybrid Federations. By using Web APIs it should be made possible to integrate so-called user de ned functions (e.g. similarity search) into SPARQL.
The possibility to link di erent sorts of knowledge bases
(e.g., dblp [
With relational databases, a user can display the schema
to get an overview of the data set. However, since Linked
Open Data is a graph database there is no schema needed.
Linked Data is stored in RDF [
Several systems [
Only few systems extract the schema using the SPARQL
Endpoint of the knowledge base. This approach has the
bene ts that we do not need to process data dumps and that
the information is as up-to-date as possible. However, such
approaches have disadvantages that we need to overcome.
Typical problems for example are the response time of the
SPARQL endpoints or the fact that sometimes no response
(depending on the complexity of the query) is delivered at
all. For this reason a goal of our research is to overcome these
limitations and nd a way to extract the schema even for big
knowledge bases such as WikiData [
Furthermore, we connect Linked Open Data in form of
SPARQL endpoints with Web APIs, e.g. CrossRef [
As described in [
The remaining part of the article is structured as follows:
Section 2 shows some use cases in which a hybrid federation
or the visualization of a schema can be helpful. Afterwards
we present our research plan in Section 4. There we explain
which problems need to be solved and go deeper into the
details of Hybrid Federations. In Section 5 we present our
evaluation plan and some data sets we want to use. The
last section gives a brief overview of related work such as
LODeX [
In this part we present several use cases in detail. The
rst use case tackles the problem of non-experts or
nontech users, i.e. it should be possible for a user (without
knowledge about SPARQL) to understand the structure of
a knowledge base in a fast way. Furthermore, the extraction
of a schema should make it possible to identify important
(new) knowledge bases. In addition, it should be easy for
such a user to connect data from a knowledge base with the
data of a Web API. For this reason we motivate the use of
a visual query interface, like presented by FacetGraphs [
The second use case deals with the integration of data.
In this case, information from a Web API (e.g. Springer
SciGraph [
The last use case, namely the data processing use case, tackles the problem of domain oriented functions, e.g. similarity search, topic analysis and more. For example, until today it is not possible to execute a SPARQL query like "Give me all articles that are similar to the article with the DOI d". SPARQL does not understand the concept of similarity and therefore a user has to implement a program to solve this task. This use case illustrates the need to implement user de ned functions within SPARQL. 2.1
A common use case for data curators is nding new and relevant data sets. The curator has to look at the data in the new data set and nd out whether these data t to his database (federation) at all. If, for example, the underlying database is of bibliographic nature, it makes little sense to search for more information in a sport's database. A curator therefore needs to be able to quickly determine the domain of a data set (SPARQL endpoint).
Unlike relational databases, graph databases like Linked Data knowledge bases do not necessarily require a schema. A curator has to formulate multiple queries to discover the structure of the schema and knowledge base. Depending on the complexity and size of the explored knowledge base, this can require several complex SPARQL queries and can be a time-consuming task. Furthermore, the curator needs to know how to formulate SPARQL queries (expert user).
Most data curators, are non-expert users and even if they were experts, it would still take some time to gure out the structure and domain of the database.For this reason, this use case focuses on extracting and visualizing the structures of a Linked Data knowledge base. Further, we consider the problem that non-expert users may still have to formulate their own queries in order to obtain detailed information. However, since the concepts of a query language must be understood by the curator, an alternative query method is needed in this use case. Here the curator should have the possibility to formulate complex queries with a few clicks via a graphical user interface. 2.2
The dblp computer science bibliography [
The usual process is to download (not up-to-date) data dumps and to integrate the downloaded data into the dblp data repository by using self coded scripts or programs. The main problems in this approach are (1) that the used data is not up-to-date and (2) that data providers often change the structure of the data dumps (new tags, di erent structures, etc.) such that the used programs and crawler needs to be changed.
Especially the last task is very bothersome, because it
is not uncommon that programs and crawler have to be
changed completely in order to work again correctly. For
this reason it is desirable to query a Linked Data repository
and combine it with the data provided via other endpoints
e.g. Web APIs. Because also the schema of endpoints can
change it is important that the algorithms, to combine the
data of endpoints with APIs, can automatically detect
linkage points. Furthermore, the user should not notice that
some data providers do not provide a SPARQL endpoints.
The goal is, that the user has the feeling of a homogeneous
database while querying but in reality using di erent data
formats, data modalities and di erent kinds of endpoints
(denoted as hybrid federation). It should also be possible to
extend the used data sources quickly and use di erent kinds
of endpoints such as SciGraph [
In the previous use case we wanted to integrate data into an existing knowledge base by using multiple heterogeneous data repositories and formats (called a hybrid federation). The next step is to work with this information and process the data, e.g. by using data mining or data analysis techniques. One example is a query that lters all publications similar to a previously speci ed publication: \Select all publications that are similar to the publication with DOI d".
SPARQL provides some basic functions such as lter the minimum, maximum or a count function. But more advanced and domain oriented tasks like a similarity search based on abstracts are not included in the SPARQL Query Language. For this reason it is desirable to add user dened functions to the toolbox of SPARQL which can be de ned/implemented by a developer (expert user) and in addition, can be fast and easy adopted into SPARQL.
Some work has already been done in the area of Schema
Inference. Also, the user de ned functions were already
introduced in [
As already mentioned, Schema Inference can be divided into two groups. The rst group of algorithms works on data dumps and can therefore ignore server problems (o ine approach). However, the problem with this approach is that the data dumps are usually not up-to-date. The second group tries to extract the schema via the SPARQL endpoints (online approach) and can therefore work on current data.
SchemEx [
In contrast, LODeX [
Since not all classes of an endpoint are needed in
order to determine the domain of the knowledge base,
LDVOWL [
Kellou-Menouer and Kedad present in SchemaDecrypt [
Approaches as described in [
The idea of federations has been around for a long time
and several systems like FedX [
Preda et al. introduced with ANGIE [
Web APIs are viewed as functions in ANGIE which are modeled as an RDF graph that contains variables like a query. This means that, similiar as in the System of Koutraki et al., a user needs to con gure the Web API manually. 3.3
Most approaches that are focused on hybrid query processing share the assumption that federation members provide their data in Linked Data formats such as RDF. Domain oriented functions such as similarity search are supported by using special indices and prede ned properties (e.g., full-text search in Virtuoso). This is not a general/su cient solution, because not every knowledge base provides these indices.
Some work in this domain is done by Nikolov et al.
introducing the Ephedra system [
In the following section we describe in more detail what we want to implement and what we want to improve. First, our research aims to develop a scalable, e cient and sound algorithm to derive the schema of a SPARQL endpoint. We then present a basic procedure for linking the data from Web APIs to the data of SPARQL endpoints. Furthermore, we implement (as an intermediate) step a graphical query interface, so that non-expert user can formulate complex queries. In addition, we present a basic idea, how to combine Web APIs and SPARQL service calls to realize user de ned functions. 4.1
This section explains what information needs to be extracted from a knowledge base and what problems are encountered. In a rst step we need to nd all classes and entity types. After that we need to nd all connections between classes and the corresponding properties. Furthermore, it is desirable to nd out how much the classes and properties of the knowledge base are used. Using this information we display the most used classes and properties to the user and make it easier to gain an insight into the focus of the knowledge base.
SELECT DISTINCT ? c WHERE f ? s a ? c . g SELECT DISTINCT ? c WHERE f? s <p> ? o . ? s a ? c . g SELECT DISTINCT ? c WHERE f? s <p> ? o . ? o a ? c . g
SELECT DISTINCT ?p WHERE f ? s ?p ? o . g SELECT DISTINCT ?p WHERE f <c> ?p ? o . g
A logical rst step is to request all used classes or
properties in a knowledge base (using the rst queries in Query
Group 1 and 2). Note that in order to classi y also SPARQL
1.0 endpoints, we did not use the EXISTS lters. A SPARQL
endpoint will possibly not answer to these queries,
depending on the size and complexity of the knowledge base. For
example, if we request BNF [
A light knowledge base is a light data set that can answer all queries from query group 1 and 2. Note that <c> and <p> represent classes or properties, contained in the knowledge base. It is important, that the endpoint can answer for all values for <c> and <p> in order to yield as a light knowledge base. We did not use the EXIST lter because we wanted to include SPARQL 1.0 endpoints in our de nition/classi cation.
The reason that the endpoint is able to answer the queries, may be due to the power of the server, an index optimized for such queries, or a data set with few properties and classes.
A type-heavy knowledge base is a data set that cannot respond to all queries presented in Query Group 1. This may be because the server has too few resources, the index is not optimized for such a query, too many types are used in the data set or simply because the data set is very huge.
Similar to type-heavy, a property-heavy knowledge base cannot respond to the queries shown in Query Group 2.
Knowledge bases that are both, type-heavy and propertyheavy, are denoted as heavy knowledge bases.
Schema of the rst three types can still be derived rather easily, because in the case of type-heavy we can simply query all properties and then query the source and object classes for each property. This reduces the amount of results and the server is not stressed as much. In the case of propertyheavy, the procedure is exactly the other way around. First all classes or types are queried and then the properties for each class in the knowledge base are queried. In a last step we have to test which classes are connected to each other.
In principle, we can use approaches as presented by the
LODeX System [
Therefore, our goal is to nd a way, to infer the schema of
heavy knowledge bases like DBpedia [
If a user wants to use multiple heterogeneous data
repositories with heterogeneous data modalities (e.g. SPARQL
endpoints and Web APIs), it is important to have some
information of these data endpoints. For example, if we want
to integrate the SciGraph Web API [
Therefore, we need to learn the con guration of a data endpoint, i.e. to learn what kind of values the parameters of the Web API expect. Consider the example of Springers SciGraph Web API, it requires a parameter called \doi". It is a big overhead, if the user of a federated system has to test every value of a knowledge base in order to determine the correct con guration for a Web API. For this reason, an automatic interface detection for SPARQL is designed and implemented. This detection algorithm uses di erent techniques to match the parameters with the corresponding data types it can process, e.g. DOIs.
Similar as in ANGIE, Web APIs are modeled as RDF graph and describe which input parameters are required or are optional. The di erence to ANGIE is that in our approach a user do not have to specify which data types belong to what parameters (e.g. ?id takes DOIs as input values). Only the parameters need to be speci ed and afterwards the system determines the appropriate data types itself. Furthermore, the graph stores the linkage points between the Web API and the knowledge base. Using this information, we can later determine what API needs to be requested to ll the knowledge base with missing information. To realize this detection algorithm we perform the following steps:
In a rst step we match the parameter names with the property names of our knowledge bases, e.g. doi and dblp:doi. We do this so that we do not have to test all properties from our knowledge base with the Web API parameters, e.g. dblp:title and doi or dblp:isbn and doi. This is only a small improvement, since most Web API parameters do not have a clear meaning like q or id. We can not simply match parameters like q with the properties in our database, because this labeling is too general.
If we can match the API parameter names with properties
from our knowledge base, we send in the next step some
requests to the Web API in order to check if we get results.
Therefore, we select from the found properties a number of
randomly selected entities and send these values with the
corresponding parameters to the Web API. For example,
in case of SciGraph [
If we can not match the API parameter names with
properties from our knowledge base, we need to do the above
described procedure for all properties in our knowledge bases.
If we consider WikiData[
In the next step we need to check, whether the Web API responses with meaningful data. Some Web APIs have a fuzzy search and, in doubt, return any or the best matching result before they return none. For this reason we de ne a meaningful response in the following way: A meaningful response consists of an amount of data in which we already know a minimum amount of information. This means, that the information returned needs to overlap with the information in our knowledge base, in order to measure that the Web API returned a valid response and not just the best matching result.
But before we can determine whether the responses we receive are meaningful, we need to send requests to the Web API again. This time we only send property data to the Web API for which we got a response. In the previous step, we only sent a small number (denoted by ) of requests to the Web API to see if we get an answer. This time we need more data/responses in order to evaluate correctly if we get meaningful answers. For this reason we send a number of requests per property to the Web API.
To test whether some information in the response matches our data, we need to use record linkage algorithms and metrics. Since most Web APIs send JSON or XML as response, we rst need to transform this response in a Linked Data format. Because both, JSON and XML, represent tree structures, we can in a rst step atten the tree. The URL, in which the query for the Web API is encoded, is used in RDF format as subject. The path of the attened JSON/XML response to the actual values are used as properties and the values are used as objects in the created RDF format.
Afterwards we can evaluate, rather the created RDF response is meaningful. Only when a minimum amount (denoted by ) of information overlaps we can be sure that we have received a meaningful response.
As it should be clear, the choice of the thresholds , and is critical when it comes to the quality of the matching and the run time. In addition, there is a new combination of these three values for each selected pair of knowledge base and Web API. In order to get the best results we need to identify the optimal combination of these three values.
It is also possible to change the focus of the matching by varying the threshold values. For example, you can achieve an exact matching by using high threshold values and also reduce the number of requests by using low threshold values and few requests and thus it is possible to nd matches even for paid Web services.
To present linkage points between the knowledge base and the Web API to the user, we provide a visual representation of the derived schema and its linkage points to the corresponding Web API.
In order to prevent the visualization from becoming cluttered, we combine for large knowledge bases, e.g. WikiData, all classes in their super classes and provide in addition the possibility to only show the most used classes and connections in the knowledge base. This allows a non-expert user to quickly see which information can be added to the knowledge base. Accordingly, a schema of the Web API must also be created on record level. 4.3
In this section, we describe brie y planned work that we have not yet been able to devote ourselves to.
As already explained, we want to extend SPARQL and give an expert user the possibility to implement domain speci c functions such as a similarity search and use it in SPARQL. We use Web APIs again, because they are very exible and easy to program. In addition, developers are not limited to the choice of a single programming language and can make Web APIs easily accessible to a community.
By using the SERVICE and BIND keywords from SPARQL
we want to call this user de ned functions and bind the
output to variables. This approach is also used by the Ephedra
system [
Our goal is that a user can call this custom functions using
the data from our Hybrid knowledge base. This includes the
data from Web APIs, e.g. SciGraph [
In the future, we want to develop a visual query interface (as
already proposed in LODeX [
Our goal is that we can extract the schema of a heavy knowledge base (see Section 4.1) as precisely as possible. Furthermore, we want to link information from Linked Data knowledge bases to the data from Web APIs. In order to show the soundness of our approach, we will describe in the following our evaluation plan. 5.1
To determine whether the extraction of the schema worked correctly, we need to create two types of data sets. The rst type of data set is intended to test and train the schema extraction algorithm. The second type of data set is used to evaluate the correctness of the algorithm. To prevent us from falsifying the evaluation, we decided to use two types of data sets.
The rst step to evaluate the schema inference algorithm is to extract the schema of an endpoint by hand (this will be the used gold standard). This implies that the data is searched manually and the schema of the endpoint is extracted accordingly. The Schema Inference algorithm is then applied to the endpoint. The nal step is to compare the two derived schemas and measure how similar they are. Therefore, we will store both schemas in a triple store, using RDF, and count how many triples are common or missing. In case of heavy endpoints, it is hardly possible to derive the schema manually, which is why this type of evaluation is not suitable here. Instead, we will test the correctness of our procedures on light and property-heavy endpoints and, for heavy endpoints, only test them in random samples. This means that we will check whether the connections derived in the schema exist or not. This will not help to nd missing connection but additional connections that erroneously are derived and actual do not exist in the knowledge base. 5.2
When it comes to connecting Web APIs to Linked Data endpoints and forming a hybrid federation, two di erent parts need to be evaluated.
First, we need to check which data types were recognized for the Web APIs, whether they were the correct ones, and whether all matching data types were found. On the other hand we have to check if the record linkage worked correctly. Here, too, we want to divide Web APIs into two groups, as we did previously with the evaluation of the schema. The rst group of Web APIs is used again to test and adapt the algorithms used. The second group of Web APIs is again used to evaluate and verify the algorithms used. 5.2.1
The rst step in evaluating the data type detection algorithm is to nd all data types from the knowledge base that should be classi ed as correct parameters for a specied Web API. As in the case of the evaluation of the Schema Inference algorithm, this step must be performed manually for the rst time and serves as the gold standard. The data types of the gold standard can then be compared with the found data types of the Data Type Detection Algorithms to evaluate the used algorithm. 5.2.2
As already mentioned, it must also be evaluated how good
the results of the record linkage is. An overview of data
linkage is presented in [
A special thanks goes to my supervisor Ralf Schenkel for his invaluable support.