For the last year, the BioPortal team has been working on a new iteration that will incorporate major modi cations to the existing services and architecture. As part of this work, we transitioned BioPortal to an architecture where RDF is the main data model and where triple stores are the main database systems. We have a component (called \Goo") that interacts with RDF data using SPARQL, and provides a clean API to perform CRUD operations on RDF stores. Using RDF and SPARQL for a real-world large-scale application creates challenges in terms of both scalability and technology adoption. In BioPortal, Goo helped us overcome that barrier using the technology that developers were familiar with, an ORM-alike API.
an abstraction over the physical structure of the data and the raw, underlying SQL queries. They also help to map data between relational models and object-oriented programming languages. Popular ORMs include Hibernate, ActiveRecord and SQLAlchemy for Java, Ruby and Python respectively. Goo is therefore an ORM speci cally designed to work with SPARQL backends. The Goo library frees developers from thinking about the intricacies of SPARQL, while still exposing the power of RDF's ability to interconnect data.
The driving requirements for our design are the following:
A number of libraries for di erent platforms o er ORM-like capabilities for
RDF and SPARQL. Jenabean uses Jena's exible RDF/OWL API to persist
Java Beans [
This section brie y introduces the Goo API. In this section, and the rest of the paper, we describe the API using a subset of BioPortal models that are complex enough to help us outline Goo's capabilities. 1 Model definition class Person < Goo: Base: Resource model :person, namespace: :foaf, name_with: :name attribute :name, enforce: [:unique] attribute :birth_date, enforce: [:date_time], property: :birthday attribute :accounts, inverse: [ on: :user_account, property: :person] end
Class definition that maps the model to an RDF vocabulary.
The following set of models is mentioned in the remainder of the paper: User (describes a user pro le), Role (describes di erent roles of users in the application, such as an administrator), Note (describes comments on ontologies provided by users), Ontology (describes the object that represents an ontology entry in our repository) and OWLClass. We do not provide a fully detailed schema for these objects; each example is self-contained and the relations between objects should be clear to the reader. The full documentation for the Goo API is available at http://ncbo.github.io/goo/.
Goo models are regular Ruby classes. To enable RDF support each model needs to extend the Goo::Base::Resource class. Figure 1 gives a brief description of how models get de ned. In the same gure it is shown how Ruby developers can save and validate the object without having to deal with RDF and/or SPARQL.
Once objects are de ned, Goo provides a framework for creating, saving, updating and deleting object instances. Goo assures uniqueness of RDF IDs in collections and tracks modi ed attributes in persisted objects. The DSL allows us to provide both validation rules and de ne how objects are interlinked.
Ruby is a typeless language and thus developers can assign arbitrary value types to variables and attributes (i.e: the language does not enforce the assignment of Date values to a property that should only accept Date objects). We rely on Goo to perform these operations automatically and transparently, notifying when a validation fails. Goo incorporates multiple built-in validations for data types like email, URI, integer, oat, date, etc. Moreover, the framework can be extended by using Ruby lambdas which can be needed to perform custom validations. For example, a custom ISBN format validation can be included in the set of validations for a model with the following attribute de nition: attribute :isbn, enforce: [ lambda { |self| isbn_valid?(self.isbn) } ] 2.1
Querying: From Graphs of Triples to Graphs of Objects Goo's most important feature is its exible query API. The API allows retrieving individual objects, their attributes, collections, and so on.
Retrieving individual objects One can retrieve single resource instances using Resource.find. This call is useful when querying by unique attributes or when the URI that identi es a resource is known.
Retrieving object attributes By default, none of the query API calls attach any attribute values to the instance object that they return. If we try to access an attribute that has not been included (i.e., retrieved from the triplestore), Goo throws an AttributeNotLoaded exception. Our design always defaults to strategies that imply minimum data movements. This strategy improves e ciency by retrieving only the attributes that the application cares about. Data attributes are loaded into objects by using the include command (Figure 2). Incremental Object Retrieval We have encountered situations in which one might not know exactly what attributes need to be loaded in an object. Goo allows incremental, in-place retrieval of attributes. An array of already loaded objects can be populated with more attributes. This operation is in-place because Goo will not create a new array of objects but will populate the objects that are passed into the query via the models call.
Users.where.models(users).include(notes: [:content]) Pagination Our REST API outputs large collections of data and in some cases we have to implement pagination over the responses. Pagination in SPARQL, with LIMIT and OFFSET, works at the triple level but it is not trivial for non SPARQL experts to develop the queries that retrieve a paginated collection of items. Goo provides capabilities that abstract the intricacies of triple level pagination and leverages this capability to the Ruby objects. Every query in Goo can be paginated, the underlying SPARQL query uses SPARQL LIMIT and OFFSET to assure low data transfers and minimun object instantiation. A Goo API paginated call is shown in Figure 2.4. 1 john = User.find("john") notes = Note.where(owner: john) First get John and use that user
object to match the Note graph.
.include(:content) 2 notes = Note.where(owner: [ :username “john”]) .include(:content)
Query directly the Note graph to roewtrnieerveatetrviebruytenothteatthpaotinhtasstoana user with username John 3 john = User.find("john")
.include(notes: [:content]) notes = john.notes
Access the Note graph through User.
Find John and include his notes with their content. 4 notes = Note.where(owner: [ :username “john”]) Same as (2) but using pagination .include(notes: [:content]) .page(1,100) notes.each do |note|
puts note.content end
Just a plain Ruby loop over John’s notes printing the content. Creating complex queries The API also allows for more complex query de nitions. One can combine calls with or and join, and with these we internally construct SPARQL UNION blocks that can be combined with SPARQL joins. Range queries can be also de ned using the Filter object and the filter method. All of these operations can be combined to build complex queries. filter_on_created = (Goo::Filter.new(:created) > DateTime.parse('2011-01-01')) .and(Goo::Filter.new(:created) < DateTime.parse('2011-12-31')) Users.where(notes: [ ontology: [ acronym: "SNOMEDCT"] ]) .or(notes: [ ontology: [ acronym: "NCIT"] ]) .join(notes: [ :visibility [ code: "public"]]) .filter(filter_on_created) .include(:username, :affiliation)
The Ruby code above represents a Goo query that retrieves the list of users, with their :username and :affiliation, that submitted notes to the ontologies "SNOMEDCT" or "NCIT" and these notes have visibility code "public". In addition, a lter is created to lter users to just the ones that were created in the system between a range of dates. This query has an extra complexity, the notes attribute in User is de ned as an inverse attribute. Goo is able to reverse the SPARQL query patterns to match the graph with the correct pattern directionality. The ltering implementation also allows for retrieval of nonexistent graph patterns. To retrieve the list of users that never submitted a note we simply use the unbound call in Filter.1 1 See usage of Filter.unbound at http://ncbo.github.io/goo/
Di erent query strategies can lead to starkly di erent performance in SPARQL. A triplestore might have an e cient query implementation, but if our application, for example, moves data around a lot, our queries will not perform well. Indeed, one often hears complaints about the performance of SPARQL engines whereas the real issue is the client who is not using SPARQL e ciently. Our key rationale with implementing query strategies in Goo is to provide a layer that ensures e cient access and query decomposition for SPARQL without the developer having to worry about it.
Goo's strategy navigates the graph of included patterns recursively. The rst query focusses on constraining the graph and retrieving attributes that are adjacent to the resource type. To retrieve data attributes located more than one hop away, Goo runs additional queries|as many of these queries are types of resources that are involved in the retrieval request. As a result, when a developer uses Goo to request attributes of dependent resources, Goo will decompose the request into multiple queries.
Goo traverses the graph patterns recursively using a Depth First Search (DFS); it focuses on individual resource types in each step. Figure 3 (right side) shows how we chain these queries together with SPARQL FILTERs that join sequences of OR operations. These lters help each subsequent query to retrieve only attributes for the dependent models and not the entire collection.2
Consider the following example. In BioPortal, a user can attach notes to ontologies. A note description links to a user (author of the note) and the ontology that the note refers to. Our testing dataset contains 400 ontologies, 10K notes, 900 users and 10 roles. We have intentionally skewed our testing dataset so that the top 10% ontologies account for 55% of the notes|the distribution that re ects the actual state of a airs in BioPortal. Figure 3 highlights the performance gain when retrieving notes for each ontology in BioPortal using Goo. In the Naive approach performance degrades when we start projecting, into tabular form, n-ary relations that are hidden in the graph. We have seen that it is often the case that hand-written SPARQL queries retrieve too much data at once, and this can cause combinatorial data explosions due to the hidden n-ary relations.
As it can be seen in Figure 3, we can demonstrate the di erence in the performance of di erent query strategies even on this small dataset. Goo's strategy will recursively navigate di erent types of objects in the retrieval, avoiding combinatorial explosions. 4
Traditional software development and database development has a long history of reliable frameworks to access backend systems. The majority of Semantic Web 2 For this experiment we used 4store 1.1.5 in a cluster setup with 8 backend nodes.
Naive SELECT * FROM :User FROM :Note FROM :Role WHERE { ?id a :User .
?id :username ?username . ?id :email ?email . ?note :owner ?id . ?note :content ?content . ?note :ontology ?ontology . ?ontology :acronym "$ONT_ACRONYM" . ?ontology :name ?name . ?id :roles ?roles . ?roles :code ?description } 3 SELECT ?role ?content
FROM :Role WHERE { ?role a :Role . ?role :code ?code . } FILTER (?role = <> | ?role = <> ..) 4 SELECT ?note ?user ?content
FROM : Ontology WHERE { ?ont a :Ontology . ?ont :name ?name . } FILTER (?ont = <> | ?ont = <> ..) development today often includes writing SPARQL queries. In many cases, software developers are unaware of what queries the SPARQL server is optimized for and what queries they should use. Furthermore, many software-development teams do not yet have SPARQL experts, which makes relying on triplestores as components in large software systems problematic. Indeed, even in our team we have experienced this problem as we were redesigning the BioPortal backend to use a triplestore and not all of our developers were pro cient in writing e cient SPARQL queries. The development of Goo enabled our team members to overcome that barrier using the technology that they were familiar with (Ruby). Goo is a completely general ORM for SPARQL and therefore other developers can use it in their projects de ning their models that Goo will validate and relying on the SPARQL query optimization in Goo to access their triplestores.
In this paper we show one example of how using naive SPARQL can lead to unexpected bad performance (see Figure 3). Our preliminary study shows that the combination of n-ary relations in hand-written queries result in query time distributions that may not perform well. Figure 3 shows that a simple partioning strategy can help to alleviate this issue.
We are proponents of semantic technologies, and thus we were drawn to the idea of using ontologies to de ne our schemas, including the domains and the allowed values for attributes. However, we see two major advantages to a DSL like Goo. First, one of our goals was to make schema de nitions easy to use for developers who are not familiar with ontologies or OWL, and using ontologies counteracts that goal. Second, ontologies traditionally entail new information and are not designed to validate constraints.
Goo enables software developers without signi cant experience with semantic technologies, to use SPARQL and RDF naturally and e ciently. The BioPortal developers have found it easy to start working with the Goo API; as a result, the transition to RDF and triple store technology within BioPortal was much faster and smoother than it would have been otherwise. Basic query de nitions in Goo are intuitive because the combination of hashes and arrays to construct Goo query patterns resembles JSON structures and developers are very familiar with them. Knowing that following a few simple restrictions, such as only querying for attributes that are needed, freed them from having to worry about the performance of the data storage layer and allowed them just to focus on the business logic, which sped development time signi cantly.
Acknowledgments This work was supported by the National Center for Biomedical Ontology, under grant U54 HG004028 from the National Institutes of Health.