We describe recent ontology and annotation editing capabilities to a specialized data management system for observational data. The system supports observations and measurements explicitly, allowing users to upload observational data sets as well as semantically describe and query data sets using formal OWLDL ontologies. Recent extensions allow users to extend observational ontologies with domain-specific terms as well as provide detailed semantic annotations using a “markdown”-based approach. In addition, we describe a new implementation of the system using standard semantic web technologies for managing OWL-DL ontologies and RDF triples. Our approach supports a wide variety of observational data, and is especially targeted at helping scientists manage heterogeneous biodiversity and ecological data by allowing access to data through a common and generic observations and measurements data model.
Performing an ecological analysis to study phenomena across geographic, temporal, or
biological scales typically requires access to a variety of existing (already collected)
observational data sets. A major challenge when performing such an analysis is
understanding and reconciling the structural and semantic differences among data sets. In
particular, data sets often differ in the number of attributes, the names of similar
attributes, the relationships implied between attributes, and the coding conventions used
for representing information within data sets. These differences not only make
discovering relevant data difficult, but also requires researchers to spend considerable time
interpreting and integrating potential data sets for use within any particular analysis.
We aim to help address these challenges by providing a suite of lightweight,
ontologybased tools that allow researchers to semantically describe, access, and analyze
heterogeneous data sets (either their own, or those collected for use in research studies). In this
paper, we describe tools that have recently been developed within the ObsDB system
[
evaluate and store query results
(Jena TDB) query validation
HermiT Reasoner The OWL API store
(Materialization)
OWL-DL Ontologies
R System
The tools we have developed leverage a “markdown”-based approach for defining ontologies, for specifying data-level semantic annotations (from which data is “shredded” into the observational model), and for expressing data discovery queries. The “markdown” is then converted automatically within ObsDB (see Figure 1) into corresponding descriptions in OWL-DL (for representing ontologies), RDF (for storing observation and measurement instances), and SPARQL (for executing discovery queries and filtering collections). ObsDB also allows observations and measurements from multiple data sets to be merged into “virtual” observation collections (RDF graphs), which can be saved and further accessed and queried. Ontologies within ObsDB allow observations and measurements to carry rich semantic descriptions, including the types of entities observed, the characteristics of entities measured, the context in which observations were made, and detailed measurement standards (units) for facilitating unit conversion. ObsDB also supports an expressive query language for selecting data sets and filtering data by observation and measurement types. For instance, users can specify queries to find all data sets that contain specific measurements of entities (e.g., diameter and height measurements of trees), relationships and constraints among entities (e.g., the length of branches on trees of a minimum height and at specific elevations), and the use of desired measurement standards (e.g., in meters). Similar queries can also be expressed to obtain all observations (either within or across collections) matching such criteria, where unit conversions are automatically applied as needed. Through integration with the R system 1, analytical scripts can also be called from within ObsDB to perform a variety of exploratory analyses over observation collections. 1 http://www.r-project.org
In this paper, we extend our prior work on ObsDB [
Figure 1 shows the main architectural components of ObsDB. The ObsDB system is implemented in Java and can be used from within other (external) applications (via API calls) or by using the ObsDB interpreter. ObsDB manages user loaded data set files, semantic annotation files, and ontology files. Ontologies are converted to OWLDL files by ObsDB and are stored and managed using the OWL API2. Annotation files can be applied to data sets to produce a “materialized” set of RDF triples (an RDF Graph). All RDF data is stored within ObsDB using the Jena triple store3 technology. Users can query RDF Graphs using the ObsDB query processor, which converts highlevel queries expressed in ObsQL (the query language of ObsDB) into corresponding SPARQL queries. As part of the query evaluation process, ObsDB uses the HermiT OWL-DL reasoner for query expansion (which is also used to verify semantic annotations are semantically consistent). Finally, R scripts can be defined and registered with ObsDB to perform statistical and analytical operations over RDF Graphs stored within ObsDB.
The rest of this paper describes these features in more detail. Section 2 describes the underlying observations ontology employed by ObsDB and its newly supported ontology “markdown” approach. Section 3 describes the new semantic annotation approach employed by ObsDB. Section 4 briefly describes ObsQL and its new implementation in ObsDB. Finally, Section 5 concludes by describing related work and our future directions for ObsDB. 2
The ObsDB system is built on a recent version of the Extensible Observation
Ontology (OBOE) [
Observation ofEntity (1:1)
Entity hasMeasurement (0:*)
ofCharacteristic (1:1) Measurement
Characteristic hasValue (1:1) usesStandard (1:1) usesProtocol (1:1) Value
Standard
Protocol
methods, precision, and accuracy (not all of which are shown in Figure 2). An
observation (event) can occur within the context of zero or more other observations, e.g.,
an observation of a tree specimen may have been made within a specific geographic
location, and the geographic location provides important information for interpreting
and comparing tree measurements. In this case, by establishing a context relationship
between the tree and location observations, the measured values of the location are
assumed to be constant with respect to the measurements of the tree. Context forms a
transitive relationship among observations. A key feature of the model is its ability for
users to assert properties of entities (as measurement characteristics or contextual
relationships) without requiring these properties to be interpreted as inherently (or always)
true of the entity. Depending on the context an entity was observed in, its properties
may take on different values. For instance, the diameter of a tree changes over time, and
the diameter value often depends on the protocol used to obtain the measurement. The
observation and measurement structure of Figure 2 allows RDF-style assertions about
entities while allowing for properties to be contextualized (i.e., the same entity can have
different values for a characteristic under different contexts), which is a crucial feature
for modeling scientific data [
Figure 3 shows the main classes and properties defined in OBOE for representing measurement standards, including units of measure. Every measurement unit is associated with a measurement characteristic (e.g., length, mass, time, area, volume, etc.) and the set of units are divided into four subclasses. A base unit represents a unit that cannot be naturally divided into smaller units. Examples include meter, gram, second, kelvin, and so on. A prefixed unit applies a prefix (represented as a literal value) to a
Standard Categorical Standard
Unit
Index CompositeUnit hasPower
DerivedUnit !"#$ hasUnit (>1) hasPrefixMultiplier %&'!()*$ hasUnit (=1)
hasUnit (=1) PrefixedUnit
BaseUnit base unit. Examples include kilogram (with base unit gram and multiplier 1000) and centimeter (with base unit meter and multiplier 0.01). A derived unit assigns a power (other than 1) to either a prefixed or base unit. Examples include meter squared (m2), hertz (s−1), and microliter (mm3). A composite unit combines 2 or more derived, prefixed, or base units. Examples include meter per second (which is composed of meter and a derived per second unit, i.e., m × s−1) as well as “dimensionless” units like gram per gram (g/g) in which retaining the original units is often needed for interpreting and integrating data.
Users of ObsDB can create their own ontologies that extend OBOE with domainspecific terms when annotating data. ObsDB supports a lightweight “markdown” syntax for describing terms, which was developed specifically to support OBOE modeling constructs and common modeling patterns. The advantage of having a lightweight syntax is that it allows non-expert users in OWL-DL to create and edit terms as needed, without needing to learn DL syntax (required, e.g., in Protege5 or if editing OWL-DL/RDF syntax directly). Ontologies expressed in the lightweight syntax are automatically converted to the corresponding OWL-DL representation by ObsDB (see Figure 1). The lightweight syntax is based on YAML6 and provides support for the following tasks: (i). Importing domain-specific OBOE ontologies for extension; 5 http://protege.stanford.edu/ 6 http://www.yaml.org/ (ii). Creating entity, characteristic, and protocol hierarchies; (iii). Defining characteristic qualifiers (e.g., to specify an “average” length characteristic where “average” denotes the qualifier); (iv). Creating base, prefixed, derived, and composite units; (v). Specifying unit conversions; and (vi). Defining categorical measurement standards.
For instance, the following example establishes a simple class hierarchy using the ObsDB lightweight ontology syntax: !Entity name: "Organism" childClass: !Entity name: "Tree" # basic types of trees childClass: !Entity name: "DominantTree" comment: "A tree that extends above surrounding ... childClass: !Entity name: "OvertoppedTree" comment: "A tree that cannot sufficiently extend its crown ...
equivalentClass: "SuppressedTree" ... # different species of trees childClass: !Entity name: "DouglasFir" equivalentClass: "Pseudotsuga_menziesii" comment: "See Garrison et al., 1972" tree class is also defined with three subclasses (dominant, overtopped, and Douglas fir). Each class has a name and an optional comment. In addition, equivalent classes (i.e., synonyms) can be specified as in the case of an overtopped tree (defined as being equivalent to a suppressed tree) and with Douglas fir (where in addition to the common name the taxonomic name is also given). The following defines an example physical characteristic.
import char: "http://code.ecoinformatics.org/.../oboe-characteristics.owl" !PhysicalCharacteristic name: "DiameterAtBreastHeight" parentClass: "char:Diameter" Here diameter at breast height (DBH) is defined as a subclass of the diameter class, which is imported from another ontology (as given by the import statement). The following example defines a simple base unit, composite unit, and unit conversion. !BaseUnit name: "Meter" characteristic: "oboe:Length" !CompositeUnit name: "MeterPerSecond" characteristic: "oboe:Speed" allUnits: - "Meter" - !Derived baseUnit: "Second" power: -1 !UnitConversion name: "FootToMeter" source: "Foot" target: "Meter" multiplier: 0.3048 offset: 0 In this example, the composite unit is defined over the base unit meter and a derived unit defined “on the fly” (i.e., without providing a specific name to the unit). Finally, the following illustrates a simple categorical standard definition.
!CategoricalStandard name: "TreeGrowthVigorStandard" comment: "Standard values for good, fair, and poor tree growth vigor" values: "TreeGrowthVigorValue" {"good_tree_growth_vigor",
"fair_tree_growth_vigor", "poor_tree_growth_vigor"}
In this case, we are defining a value partition (as in [
After starting the ObsDB interpreter, users can load ontology files using the load onto command. When loading an ontology file, a namespace prefix and URI is also assigned to the ontology for use within ObsDB. For instance, the following shows the result of starting ObsDB and loading the “ont1.yml” ontology file:
ObsDB v1.0 Type ’help’ for a list of commands. Type ’quit’ to quit ObsDB. > import onto ’ont1.yml’ as ’ont1’ using ’http://obsdb.org/ont1’ Ontology created
Ontology loaded In this case, we are assigning the ontology the namespace prefix “ont1” and the URI “http://obsdb.org/ont1”. Once loaded, the ontology can be accessed via the namespace. Ontologies can also be updated using the ObsDB update onto command, which allows ontologies to be modified without having to remove (or drop) an ontology and then load the updated version.
In general, we have found that using a lightweight approach such as this has a number of benefits for specifying OBOE extensions and for annotating data. In particular, the approach allows new ontology terms and entire ontologies to be quickly and easily created by simply opening and editing a text file, and the high-level syntax supports otherwise complex description-logic definitions without requiring users to be experts in description logic (which is often the case in Protege’s OWL-DL editor). The latter is especially an issue in ontologies like OBOE that leverage description logic constraints that must be maintained, e.g., via modeling patterns such as value partitions and various class and property restrictions. Once a user loads an ontology into ObsDB, the system automatically performs syntax and semantic validation (e.g., checking for inconsistencies). Together with the lightweight text-based syntax, this allows for rapid editing, loading, and validation of OBOE ontology extensions. 3
Semantic annotations in ObsDB define how to translate a tabular data set into a corresponding collection of semantically relevant observations and measurements. Semantic
STAND PLOT TAG SPP YEAR DBH CANCLASS VIGOR
B388 1 3319 PSME 1999 22.5 C 1
B388 1 3320 PSME 1999 16 I 1
B388 2 3336 PSME 1999 33 D 1
B388 2 3339 CACH 1999 5.8 S 1
B646 1 1817 PSME 1999 22 C 1
B646 1 1815 CACH 1999 5.7 I 1
B684 2 2207 ALRU 1999 19.9 C 1
... ... ... ... ... ... ... ...
annotations are defined using semantic templates [
observation ’PlotObs’: entity: ’ont1:Plot’ measurement: characteristic: ’obs:Name’ value: ’$PLOT’ context: ’StandObs’q entityKey: ’$PLOT’ within ’StandObs’ 7 Based on one of the many data sets available on the H.J. Andrews Experimental Forest LTER site (http://andrewsforest.oregonstate.edu/). Here the plot is nested within the stand, and so each plot observation has as context the corresponding stand observation. In this data set, the names of plots across stands are not unique (e.g., stand B288 contains a plot 1 as does stand B646). This information is denoted using the “within” keyword. Although not included here, additional measurements can also be added to the template, e.g., the area of the plot (which in this case could be specified as a constant value assuming all plots are of the same area). The year attribute would be annotated similarly to the stand as follows.
observation ’YearObs’: entity: ’ont1:TimePeriod’ measurement: characteristic: ’ont1:Year’ value: ’$YEAR’ entityKey: ’$YEAR’ The remaining attributes would be annotated via a single tree observation template: observation ’TreeObs’: entity: match ’$CANCLASS’ with ’D’ => ’ont1:DominantTree’ ’S’ => ’ont1:SuppressedTree’ ... entity: match ’$SPP’ with ’PSME’ => ’ont1:Pseudotsuga_menziesii’ ’CACH’ => ’ont1:Castanopsis_chrysophylla’ ’ALRU’ => ’ont1:Alnus_rubra’ ... measurement: characteristic: ’obs:Name’ value: ’$TAG’ measurement: characteristic: ’ont1:DiameterAtBreastHeight’ standard: ’ont1:Meter’ value: ’$DBH’ measurement: characteristic: ’ont1:Vigor’ standard: ’ont1:TreeGrowthVigorStandard’ value: match ’$VIGOR’ with ’1’ => individual: ’ont1:good_tree_growth_vigor’ ’2’ => individual: ’ont1:fair_tree_growth_vigor’ ’3’ => individual: ’ont1:poor_tree_growth_vigor’ context: ’PlotObs’, ’YearObs’ entityKey: ’$TAG’
In this example, the entity is defined by two separate attributes: the CANCLASS attribute specifies the canopy class the tree belongs to; and the SPP attribute specifies the tree species. In both cases, attribute values are mapped to ontology classes (i.e., a value such as “PSME” denotes a specific species type). Each tree observation also has three associated measurements: the (tag) name of the tree, the diameter at breast height (measured in meters); and the growth vigor. For the tree’s vigor, each value in the dataset is mapped to a specific OWL-DL individual value (as opposed to a class) that is defined in the corresponding vigor standard of the ontology example of Section 2. Finally, each tree observation is made within the context of its corresponding plot and the year in which the observation was made.
Data sets and annotations are loaded independently in ObsDB. The import table command is used to register a CSV file denoting a data set with ObsDB. For example, the following command can be used to load the example table of Figure 4. > import table ’table1.csv’ as ’table1’ File copied to /data/tables/ directory.
File loaded. > import annotation ’annot1.txt’ as ’annot1’ File copied to /data/annotations/ directory.
File loaded.
Annotations are loaded into ObsDB using the import annotation command: Once loaded, annotations can be applied to tables to generate an RDF graph of corresponding observation and measurement instances using the ObsDB apply command: > apply ’annot1’ to ’table1’ as ’coll1’ using ’http://obsdb.org/coll1’ ***Generating Data From Files*** Annotation: data/annotations/annot1.oal Table: data/tables/table1.csv
Output Triples: data/graphs/coll1.ttl
Here “coll1” is used to name the resulting named RDF graph, which is given the
corresponding URI. ObsDB performs the following steps when applying an annotation to
a table: (1) it verifies the annotation and data file are both syntactically correct; (2) it
verifies the imported ontologies exist and that they have been loaded into ObsDB; (3)
it generates the corresponding observations and measurements (i.e., by materializing
the semantic annotation templates); and (4) it uses the HermiT OWL-DL reasoner to
add inferred axioms (based on ontology definitions and constraints) to the RDF Graph,
and ensures the resulting graph is consistent. Adding inferred axioms is performed to
support query expansion within ObsDB. The resulting RDF Graph is stored within
ObsDB and can then be further accessed and queried. [
Once data sets are semantically annotated and converted into their corresponding RDF graphs, they can be accessed directly from within ObsDB. There are three main ways to access data sets: (1) using the find command to issue data discovery queries to locate observation collections (RDF graphs of observations and measurements); (2) using the query command to select observations within or across data sets, the results of which can be viewed or used to create new observation collections that subset existing collections or combine multiple existing collections; or (3) using the exec command to apply statistical and analytical functions to query results (using built-in aggregation operators or by calling external R scripts).
Each of the above ways to access observation collections are expressed in ObsDB
using the high-level query language ObsQL [
PREFIX obs: <https://code.ecoinformatics.org/.../oboe-core.owl#> PREFIX char: <https://code.ecoinformatics.org/.../oboe-characteristics.owl#> PREFIX ont1: <http://obsdb.org/ont1#> PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> ...
ASK {?temporaryObservationVariable0 obs:ofEntity ?temporaryVariable0 . ?temporaryVariable0 rdf:type ont1:Tree .} As another example, we can refine the search above to look for only those collections that have a specific type of tree with a specific type of measurement: > find ont1:DouglasFir [ont1:DiameterAtBreastHeight] Matching Graphs --------------coll1 ObsQL also supports context constraints using the -> operator. For example, the following expressions locates collections that contain observations of Douglas Fir trees made within named plots: > find ont1:DouglasFir [ont1:DiameterAtBreastHeight] -> ont1:Plot [obs:Name] Matching Graphs --------------coll1 While the above expression is relatively straightforward to express in ObsQL, the corresponding SPARQL query is considerably more verbose: We note that while ObsQL expressions are generally more concise and easier to specify than their corresponding SPARQL queries (in part, because ObsQL is tailored specifically to supporting queries over OBOE models), only a subset of SPARQL can be expressed in ObsQL. In addition to the above example, it is also possible to specify multiple contexts for an observation, for example: > find ont1:IntermediateTree [] -> (ont1:Stand [], ont1:TimePeriod []) Matching Graphs --------------coll1 finds all collections with an observation of the given tree type within the context of both a stand and a time period. Note that here the stand is an indirect context for the corresponding tree since the tree has a plot as context, and the plot has the stand as context. It is also possible to query for observations and measurements within a collection. For example, the following query returns the diameter values of all codominant trees within > query ont1:CodominantTree [ont1:DiameterAtBreastHeight $d] in coll1 -------------------------------| temporaryVariable0 | d | ================================ | :ID1004 | "10.3" | | :ID1017 | "13.1" | | :ID10286 | "48.9" | | :ID10299 | "46" | ...
Here, $d is a “place holder” variable for specifying output values. Note that although not shown here, multiple place holder variables can be given per query. Also, removing the “in” clause above will result in ObsDB querying all collections for matching observations.
Basic computations can also be performed on data when using ObsQL. For instance, when querying it is possible to select a specific unit from which ObsDB will apply appropriate unit conversions. For example, in this query:
ObsDB uses the foot-to-meter unit conversion of Section 2 to convert the diameters in coll1 from meters to feet (since conversions are invertible). As another example, ObsDB can also perform statistical summaries of query results. For instance, the following query computes for each plot the average Douglas Fir tree diameter (in meters): > exec avg $d by $p in coll1 where ont1:DouglasFir [ont1:DiameterAtBreastHeight $d ont1:Meter] -> ont1:Plot $p ----------------------| ?p | mean | ======================= | :ID908 | 13.4 | | :ID10203 | 59.51429 | | :ID7 | 24.6 | | :ID10837 | 155.4143 | ... ObsDB supports the standard aggregate operations supported by SPARQL including average, mean, count, max, min, median, range, and standard deviation. Finally, custom R scripts can be used from within ObsDB. Each R script must contain a comment header denoting the name of the operation (for use within an exec command) and the variables that will be passed into the script (the script inputs). As a simple example, the following commented R script can be used to draw a basic histogram from within ObsDB. #name: hist #argument: $x A vector of x-axis variables x=$x hist(x) Once defined, this script can be called from within ObsDB as follows. > exec hist $d in coll1 where
ont1:CodominantTree [ont1:DiameterAtBreastHeight $d > 75 ont1:Meter] The result of this command generates the histogram shown in Figure 5, showing the distribution of codominant tree diameters greater than 75 m in the underlying data set. This example also demonstrates an ObsQL query that performs a logical comparison on measurement values.
In general, ObsQL is designed to provide scientists with the ability to search for relevant data sets based on domain-specific ontology classes as well as perform basic exploratory analyses through the subselection of relevant observations and measurements of data sets, by applying aggregate operations, and by applying simple R analysis scripts. Although not shown in the examples above, ObsDB also allows the results of queries to be stored in new observation collections using the as keyword. This provides a basic form of data integration, in which observations from multiple data sets can be combined into a single collection, without having to perform similar operations directly on structurally heterogeneous tabular data sets. 5
This paper described extensions to our prior work on semantic annotation and providing
access to observational data through the OBOE model [
Histogram of x 5 2 0 2 5 0 80 100 120 140 160
180 x a new ObsDB implementation over a popular RDF triple store system (Jena TDB); automated unit conversion; and extended query capabilities (e.g., value comparisons, grouping operations, and a wide range of aggregate operations).
A number of systems have recently been developed that leverage observation-based
conceptual and ontology models. Some examples include [
Our ongoing and future work on ObsDB is focused on further extending support for exploratory analysis of observational data via the R system. We are also interested in developing tools within ObsDB to support comparing data sets based on their annotations. For instance, given two observation collections, we would like to determine how closely they “match” in terms of observation and measurement types and to automatically create mappings and transformations to unify the collections into a single integrated annotation template (for further analysis). We are also interested in developing support in R to access observations stored in ObsDB, e.g., to be able to programmatically load one or more observation collections through R calls to perform more sophisticated analyses (within an R script).