An ever-growing amount of life science databases are (partially) exposed as RDF graphs (e.g. UniProt, TCGA, DisGeNET, Human Protein Atlas), complementing traditional methods to disseminate biodata. The SPARQL query language provides a powerful tool to rapidly retrieve and integrate this data. However, the inability to incorporate custom data processing methods in SPARQL queries inhibits its application in many life science use cases. It should take far less effort to integrate data processing methods, such as BLAST, with SPARQL. We propose an effective framework for extending SPARQL with custom methods should fulfill four key requirements: generality, reusability, interoperability and scalability. We present SCRY, the SPARQL compatible service layer, which provides custom data processing within SPARQL queries. SCRY is a lightweight SPARQL endpoint that interprets parts of basic graph patterns as input for user defined procedures, generating an RDF graph against which the query is resolved on-demand. SCRY's federationoriented design allows for easy integration with existing endpoints, extending SPARQL's functionality to include custom data processing methods in a decoupled, standards-compliant, tool independent manner. We demonstrate the power of this approach by performing statistical analysis of a benchmark, and by incorporating BLAST in a query which simultaneously finds the tissues expressing Hemoglobin and its homologs.
information graphs in the RDF data based on the templates and filters (constraints
on nodes and edges) expressed in the query. Such patterns match relations that are
explicitly present in the RDF graph, or that are derivable from the graph under a
given entailment regime such as RDFS or OWL-DL [
A straightforward example of such a problem, is selecting outliers from a set of query results. Which results are outliers depends on a scoring function and the rest of the set. It is infeasible to precompute which results are outliers for every putative result set and scoring function, and the math for typical scoring functions is not natively supported by SPARQL or RDF/OWL entailment.
Another example, common in life science, involves finding proteins which are evolutionarily related to a given protein of interest. There is a multitude of methods and parameters through which this can be derived, but which of these should be used strongly depend on the exact context of the question. Precomputing every possible relations between all proteins is not only impractical, but combinatorially impossible. Again, RDF/OWL entailment can not derive such complex relations ad hoc.
These examples illustrate that, even though RDF has shown itself to be a very versatile representation language, there are important relations between entities which can in principle be expressed in RDF, but that are impractical or impossible to materialize persistently in a triplestore. Hence, they are not accessible through SPARQL queries over such a triplestore. At the same time, these relations are critical to resolve queries common to the life sciences or other domains.
We present the SPARQL compatible service layer (SCRY) as a solution to this problem. SCRY is a lightweight SPARQL endpoint that allows users to create their own procedures, assign them to a URI, and embed them in standard SPARQL queries. Through these procedures, analytic methods can be executed at query time, effectively extending SPARQL with API-like functionality. SCRY provides a framework for custom data processing within the context of a Semantic Web query, facilitating on-demand computation of domain-specific relations in a generalized, reusable, interoperable and scalable manner. 2
As introduced above, RDF/OWL entailment is not sufficiently expressive to
incorporate complex domain-specific data processing methods in SPARQL queries. We observe
there is an unmet need for a platform which enables computational biologists to
efficiently implement and integrate such methods. A platform enabling this would ideally
be 1) easy to deploy, 2) easy to extend with new methods, and 3) easy to write queries
for. Concretely, we argue this can be achieved by meeting the following requirements:
Generality A generic platform, which enables the implementation of any method
from any domain, avoids Semantic Web services devolving to a multitude of
mutually incompatible "SPARQL-like" extensions (e.g. [
Reusability Implemented methods should be standardized to a degree that allows them to be reused between queries. Done well, this facilitates sharing services and reusing the work of others, effectively extending the shareable nature of Semantic Web data and schemata to apply to methods as well.
Interoperability Queries incorporating methods which extend SPARQL’s functionality should maintain full compatibility with the SPARQL standard, such that any SPARQL-aware server, service or tool can parse and resolve them. Towards this end, methods must be implemented in an endpoint-independent manner. Scalability Many analytic methods are computationally expensive, making them inhibitively slow or outright impossible to run through remotely hosted services. Simpler methods incur bandwidth restrictions if they are given high numbers of inputs, or produce large volume outputs. Such methods benefit from using local resources for computation, which requires users to control both the software and the hardware resolving their queries. 3
The simplest and most prevalent way of processing RDF data accessed via SPARQL
endpoints is post hoc scripting. This typically involves SPARQL-compatible libraries
in some general purpose programming language [
The SPARQL 1.1 specification [
Web Services are a common way of exposing data processing functionality online.
Significant efforts have been made to describe Web Services semantically to enhance
their interoperability [
X
X X X
X Generality Reusability Interoperability Scalability
X X
query engines (e.g. SHARE [
Linked Data APIs are very similar to Web Services in terms of their goals and
challenges [
Table 1 summarizes which of our requirements from the previous section are fulfilled by the listed approaches. 4
Infrastructure: Our SPARQL compatible service layer (SCRY) acts as a lightweight SPARQL endpoint, granting users access to a personalized set of services during query execution. Through these services, SCRY allows users to incorporate algorithms of arbitrary complexity within standards-compliant SPARQL queries, and to use the generated outputs directly within those same queries. Unlike traditional SPARQL endpoints, the RDF graph against which SCRY resolves its queries is generated at query time, by executing services encoded in the query’s graph patterns.
Every service exposed by SCRY comprises one or more procedures. Registered procedures are associated with a URI, which SCRY will recognize when it receives a query. These procedure-associated URIs, as well as the desired inputs and outputs, can be embedded within the graph patterns of standards-compliant, syntactically pure SPARQL queries (see figure 2). Prior to resolving the SPARQL query, SCRY executes embedded procedures and adds their output to the queried RDF graph. Procedures can comprise simple instructions, like rounding off a decimal number. They can also be arbitrarily complex, for example running an external program, parsing its output, performing some form of analysis and exposing specifically requested results as RDF. The output of one procedure can be used as input for the next, allowing simple (non-circular) workflows to be encoded.
Figure 1 illustrates the two typical dataflows of SPARQL queries resolved through SCRY, accessing it through (A) a federated query from a primary endpoint or (B) a direct query to a live SCRY instance.
Engineering effort: To ensure SCRY can be easily utilized by life scientists, we aimed to minimize the effort needed to 1) deploy SCRY, 2) extend it with novel data processing methods and 3) write queries invoking them.
SCRY can be deployed on any system running Python 2.7. The framework itself, as well as the services required for the use cases we describe below, can be easily installed using the package manager pip.4 It is equally straightforward to extend SCRY with services of your own. Python scripts added to SCRY’s working directory are automatically checked for defined procedures, which can take as little as three lines to define.5 SCRY can be further configured through a commandline interface, which includes options to load services from elsewhere.
Defining a procedure requires 1) a URI and 2) a Python function to execute when SCRY finds this URI in a graph pattern. Note these functions can rely on local or remote resources, including other Python functions, shell commands (e.g. through os.system()), and web requests.
Figure 2 shows how easily procedure calls can be embedded in SPARQL queries using SCRY. We explain the query itself, and how SCRY interprets the embedded procedure calls, in the Statistics use case of the Evaluation section. 5
We evaluate whether or not SCRY meets the requirements described in section 2 based on two use cases, from the domains of statistics and bioinformatics respectively.
Use case 1: Statistics. We use the the Berlin SPARQL Benchmark (BSBM) [
Note that calculating standard deviations requires computing square roots, which SPARQL does not natively support. Using SCRY, however, the query shown in figure 2 yields the answers by invoking two procedures defined in the scry-math package.
The graph pattern {GRAPH ?g1 {math:median in: ?ratings; out: ?median.}} instructs SCRY to 1) execute the procedure associated with the URI math:median, using whatever values are bound to ?ratings as input; and 2) create an RDF (sub)graph matching the entire pattern, with the procedure’s output inserted in the position of ?median. Use case 2: Bioinformatics. In this use case, we evaluate a very typical query from the life sciences. We use a protein sequence similarity search, to identify evolutionarily related (i.e. homologous) proteins. The derived relations are then used to determine which of a query protein’s homologs are coexpressed in the same tissues.
Our scry-blast package defines two procedures: one which runs the most commonly
used sequence similarity search method (BLAST [
Of course, SCRY can be extended with similar procedures to expose different similarity search methods, and queries can combine their results with other types of data. Generality: SCRY enables users to integrate data processing procedures of arbitrary complexity in SPARQL queries, by embedding procedure calls in a query’s graph patterns. This design is not restricted to procedures from or applications within any specific domain. Our use cases involved procedures which 1) integrate a service exposing simple mathematics and statistics functions to analyse BSBM data as shown in figure 2; and which 2) integrate a popular bioinformatics tool (BLAST) to derive evolutionary relationships between proteins at query time, as shown in figure 3.
These examples illustrate the simplicity and flexibility of using custom data processing methods within a SPARQL query, and that SCRY can extend queries with custom data processing methods of arbitrary complexity through intuitive, standardscompliant SPARQL syntax.
Reusability: The modular nature of services exposed through SCRY makes them easy to share, modify and reuse. For example, the standard deviation procedure used in the statistics use case can just as easily be applied to the e-values or molecular masses of proteins homologous to hemoglobin identified in the bioinformatics use case. Once implemented, a SPARQL extension exposed through SCRY can be used in any query.
Interoperability: The manner in which SCRY embeds procedure calls in query graph
patterns is completely compatible with the SPARQL standard. Consequently, any
standards-compliant SPARQL engine will accept queries that federate parts of the
query to SCRY, e.g. with the SERVICE statement. To demonstrate, we loaded the
benchmark data from the statistics use case into four popular triplestores: Apache Jena [
Being able to resolve a single query against different endpoints, calling the same (reusable) procedure, sharply contrasts with the alternative: extending every triplestore individually, or using their built-in SPARQL extensions, and writing triplestore-specific queries for each. Making use of triplestore-specific built-in functions requires manual search through heterogeneous and cryptic documentation, if the desired built-in exists at all. Extending triplestores with custom functionality is similarly difficult: extending Sesame7 with a simple function to calculate square roots requires 55 lines of Java8 and deploying a properly documented JAR in a specific directory.
Note that in many cases, SPARQL queries can combine simple functions to express a more complex one. A standard deviation, for example, can be calculated by taking the square root of an array’s variance, divided by its length. As complexity grows, though, such queries quickly become difficult to write and harder still to read9. In this context, SCRY provides an essential layer of abstraction, without which incorporating programs like BLAST would be difficult, tedious or outright impossible. Scalability: As a SPARQL endpoint, SCRY has the same issues as every other; the time it takes to execute a query scales rather poorly with its complexity. Where service execution is concerned, however, SCRY offers two unique advantages over typical (Semantic) Web service platforms. First and foremost, SCRY is designed to be deployed with a personalized set of services and procedures, granting users full control over both the software and the hardware resolving their services. Secondly, it decouples the execution of those services from resolution of the query. This inherently makes services exposed by SCRY more scalable than equivalent web service implementations.
Running local implementations instead of addressing an equivalent web service can significantly improve the execution time of computationally expensive procedures. For example, running BLAST locally instead of through the NCBI’s web service is several orders of magnitude faster (data not shown), in addition to giving users more control over critical parameters. Computationally expensive data processing methods are prominent in the life sciences, and using SCRY instead of online services to include their results in your queries can significantly speed up runtimes.
Note that resolving queries entirely locally has another significant advantage: privacy. Neither your query nor its (partial) results have to be exposed to external endpoints or (web) services to resolve, which is especially valuable when working with 7See http://tinyurl.com/sesame-cookbook 8See http://tinyurl.com/sqrt-extensions#file-sesame-sqrt-java 9See http://tinyurl.com/sqrt-extensions for triplestore-specific queries equivalent to that in figure 2 sensitive (e.g. clinical) data, from behind a firewall, or in an otherwise restricted network. 6
SCRY leverages SPARQL’s query federation mechanism to make services general, reusable, interoperable and scalable, which comes at a cost. The SPARQL protocol requires communication between endpoints to occur through HTTP, which requires inputs and outputs for procedures exposed by SCRY to be serialized and deserialized to flat text. Additionally, SCRY needs time to parse embedded procedure calls, execute them, populate an RDF graph, and resolve a SPARQL query against it. For our statistics use case, which deals with a large number of inputs and outputs, these steps take up approximatetely 10% of the total query time.
In terms of reusability and standardization, SCRY provides optional description of the inputs and outputs of a procedure, but does not enforce their use. Consequently, it is not compatible with automated discovery of relevant services, unlike SADI for example, which registers services using formal semantics. This is a natural consequence of SCRY’s focus on local computation, to enhance scalability, and our aim to be interoperable with extant triplestores and SPARQL tools. 7
We argue SCRY meets each of our requirements: generality, reusability, interoperability and scalability. Thus, computational biologists familiar with just the basics of SPARQL and Python can easily deploy SCRY, extend it with procedures of arbitrary complexity, and incorporate them in their SPARQL queries. SCRY overcomes issues common to the application of semantic web technology in the life sciences, such as the inability to incorporate computationally expensive data processing methods, or working with privacy sensitive data. It provides an essential layer of abstraction, without which incorporating programs like BLAST in SPARQL queries would be difficult, tedious or outright impossible.
Utilizing SCRY incurs some computational overhead, and the services it exposes are not standardized with sufficient rigidity to automate their discovery. Even so, we argue this is a price worth paying for a framework that offers such significant extension to SPARQL’s functionality.