Earth Science informatics involves collaboration between multiple groups of people with diverse specializations and goals, often using variations in terminology to refer to common resources. The uniformity of the resource identifiers often does not cross organizational boundaries. Because of this, permanent, widely used, unambiguous identifiers for resources are elusive. We examine real world cases of changing and inconsistent identifiers which inherently work against persistence and uniformity. We also present a solution which mediates factors in these situations; namely the creation of lexicons: mappings of sets of terms to URIs which are curated within the Global Change Information System (GCIS).
We discuss aspects of the GCIS which facilitate the use of lexicons: an information model which disambiguates resources, a RESTful API which provides metadata through content-negotiation, and a strategy for long term curation of URIs, including mechanisms for handling changes to URIs and variations in terms used by different communities while providing persistent URIs and preserving relationships between resources.
We provide working definitions of terms, context s, and lexicons, and relate them to the practical challenges of disambiguation and curation. We also discuss the mechanisms employed and architecture of the GCIS, and how these choices facilitate representation of persistent identifiers and mappings of them to identifiers used colloquially within various earth science communities of practice. 1. 1.1
The U.S. Global Change Research Program (USGCRP) was
established in 1989 by Presidential Initiative and mandated
by the U.S. Congress in the Global Change Research Act
(GCRA) of 1990 to “assist the Nation and the world to
understand, assess, predict, and respond to human-induced
and natural processes of global change.”[
In May, 2014, the USGCRP released the Third National Climate Assessment (NCA3). This 800 page document, authored by 300 people, each of which are affiliated with multiple organizations, has 30 chapters, 161 findings, 290 figures and 3,395 references. The references refer to publications, including government reports, peer-reviewed scientific journal articles, and books. The publications are often supported by datasets that could be based on observations, measurements, processed or derived data, or model projections. Model projection datasets are created by runs of models constrained by scenarios. Observations and measurements are taken using instruments on platforms. The information in the NCA3 provided a starting point for the contents of the GCIS. The GCIS information model includes representations of reports, chapters, findings, figures, people, organizations, references, publications, platforms, instruments, models and scenarios. The GCIS was used to support the production of the report and the dissemination of the web version of the report.
A key design goal of the GCIS was to disambiguate and identify distinct resources, and provide references to sources of information. Other goals included: supporting scientific traceability and reproducibility, facilitating the creation of reports such as the NCA3, providing a scalable backend for rich web versions of the NCA3 and other reports, providing an API for other types of applications, providing the ability to run structured queries about disparate types of earth science information, and facilitating the discovery and representation of connections between earth science information managed by independent organizations.
The GCIS has been implemented as a RESTful API, whose endpoints are URIs which are part of a knowledge base formalized by an ontology and distributed using a SPARQL endpoint to a triple store. The API supports content negotiation, and the HTML representations form a navigable web site.
This paper focuses on the concept of lexicons and the processes and techniques involved in the creation and maintenance of mappings from persistent URIs to pre-existing Earth Science identifiers. In particular, we discuss challenges and techniques for dealing with colloquial identifiers (terms) which are often specific to communities of practice. We also discuss our techniques for maintaining long term persistent identifiers, and working with changing or inconsistent terms.
The general problem of disambiguation of resources has been
known for some time, dating back at least to Leibnitz’s
formulation of the Identity of Indiscernibles in 1686 [
Once there are unambiguous URIs, if these can be mapped
between RDF datasets, the mapping forms a “linkset” [
CrossRef
Journal Article
NOAA Graphic USGCRP Report
NASA/CNES
Joint Mission NASA/JPL
Instrument Various phrases have been used to describe this problem: the co-reference problem, the identity problem, disambiguation, and more. We chose the phrase “normalization” following its usage with relational databases and character encodings.
Figure 2.26 of the Third National Climate Assessment1 depicts Past and Projected Changes to Global Sea Level Rise (see figure 1). A sequence of resolvable URIs within GCIS traces this figure to a journal article which used a dataset which used another dataset, which was captured by an instrument on a platform. Each step in this sequence has a permanent URI within GCIS, but also refers to identifiers outside of GCIS; the journal article has a DOI managed by a publisher and resolvable using CrossRef, the first dataset has a URL managed by scientists, the second dataset has an identifier managed by a NASA Data Archive, the instrument and platform have identifiers created by the Committee on Earth Observing Satellites (CEOS). While all these organizations do provide machine-readable versions of their data, the identifiers are often curated independently. However, within each organization, there are terms which unambiguously identify a particular resource.
In some situations, there may be intentional uses of different names by different organizations. For instance, the “SACD/Aquarius” mission may also be called “Scientific Application Satellite-D”
SAC-D/Aquarius is a cooperative international
mission between CONAE (Comisi´on Nacional de
1http://data.globalchange.gov/report/nca3/chapter/
2/figure/26
Actividades Espaciales), Argentina, and NASA,
USA. NASA uses the term [SAC-D/Aquarius] for
the mission [..] At CONAE, which provides the
spacecraft, the mission is referred to as Scientific
Application Satellite-D [..] [
Organizations in the domain of Earth Science distribute data, metadata, and information using a variety of serializations and interfaces, including:
NASA Earth Observing System (EOS) Clearing House (ECHO) Geographic Information Metadata Federal Geographic Data Committee Content Standard for Digital Geospatial Metadata Directory Interchange Format, NASA’s Global Change Master Directory (GCMD) W3C Data Catalog Vocabulary Open Archives Initiative Protocol for Metadata Harvesting Miscellaneous Serializations
Information conveyed varies by the standard and by the implementation. Despite these differences, the defining characteristics of resources generally remain consistent across representations. This enables a subset of information to be brought into the GCIS for the purpose of identification and disambiguation. In other words, when deciding what to harvest from other sources, a critical question is: does this piece of information help to distinguish this resource from other similar resources?
Science teams working on remote sensing missions produce scientific data and send them to data archives. A primary concern of archives is to maintain the fidelity of the science data as they are received. Because of this division of responsibilities, it’s not clear which community would be better suited to take on the task of harmonizing identifiers across data producers. Instead differences in choices of identifiers may be passed on to end users of the data.
Issues surrounding characters, case folding, encoding, and strings are often omitted when resources are identified in narrative situations. This creates ambiguities in representations and postpones normalization issues. This motivates our definition of the word “term” which is based on the Universal Character Set (UCS).
Definition 1. We define a term to be a sequence of
characters from the Universal Character Set (UCS) which is used
as an identifier for a resource by a group of people.
The linked data glossary [
Within the GCIS, terms are encoded in UTF-8 and
potentially normalized with Normalization Form C (NFC). We
note that W3C recommendations for string matching are
still evolving [
Within communities of practice, terms are created and used as part of the activities and communication between members of the community. Because of this, disambiguation of terms across communities is a secondary consideration. Within a community, a resource can be identified clearly when it is being referenced in a manner which is consistent with other similar resources. With this in mind, we group together terms using the type of resource.
Definition 2. We define a context to be a set of terms used to identify resources of the same type.
In example 3.1, ”mission” and ”instrument” are contexts. We then define a lexicon by putting together the contexts.
Definition 3. We define a lexicon to be a set of contexts used by a particular community.
Returning to example 3.1, the CEOS lexicon uses ”Mission” and ”Instrument” contexts to group together terms.
As previously noted, entities in the GCIS are identified uniquely using a URI. The URI for a particular entity is called a GCIS Identifier, or GCID.
Definition 4. The GCID is the URI for an entity in the Global Change Information System.
Note that any GCID can be used in SPARQL queries, and can also be resolved using the GCIS as an endpoint and using content-negotiation.
One organization’s “instrument” may be another one’s “sensor“. One organization’s “platform” may be another one’s “mission” or a third one’s “source”. We use lexicons to represent the way NASA’s Physical Oceanography Active Data Archive Center (PODAAC), ECHO, GCMD and CEOS all refer to the same resource: Lexicon | Context | Term | GCID (*) 5.2 System Architecture ------------------------------------------------------- The GCIS architecture incorporates elements of relational podaac | Source | JASON-1 | /platform/jason-1 and semantic systems: cascading updates, referential ingccemods || pMriesfsLiaobneIld || J2A8S6ON-1 || //ppllaattffoorrmm//jjaassoonn--11 tegrity, strict type checking and other well-established feaecho | ShortName | JASON-1 | /platform/jason-1 tures of relational databases are all valuable in maintaining podaac | Sensor | POSEIDON-2 | /instrument/poseidon-2 the quality of the URIs and their relationships. ceos | InstrumentId | 182 | /instrument/poseidon-2 (*) under http://data.globalchange.gov See also: http://data.globalchange.gov/lexicon
When a term is an identifier which is part of another triple store, we can use ”owl:sameAs” to connect the two identifiers and form a linkset. We treat dbpedia as a lexicon with a single context (”resource”). An application of this is writing a federated SPARQL query to compare crowdsourced information in dbpedia (or wikidata) to authoritative information from CEOS. This can belp improve the quality of the data in both places.
Creating a lexicon involves several steps:
An example of performing step 3 in an HTTP transaction follows; in this example we are associating “Aqua” a term used by CEOS, with the GCID “/platform/aqua” and the context “Mission”:
PUT /lexicon/ceos/Mission/Aqua Host: data.globalchange.gov Content-Type: application/json { "gcid" : "/platform/aqua" } An alternative interface is available which allows the terms and context to be sent in the payload, rather than the URI.
Looking up a term using a lexicon involves sending a GET request for an IRI containing the context and the term, and receiving a status code of 303 and a Location header, to indicate the corresponding GCIS URI, if one exists. Request:
GET /lexicon/ceos/Mission/Aqua HTTP/1.1
Host: data.globalchange.gov Response: 303 See Other Location: /platform/aqua HTTP requests to the GCIS RESTful interface are handled by querying this relational database. The database contains simple explicit tables for resources that use natural identifiers as primary keys. JSON structures are formed using the names of the columns as names of the keys in the JSON objects. There is also a generic table which may be thought of as a parent table for many of the tables (i.e. using table inheritance).
Relationships between resources are stored in one of two ways: 1. Foreign keys between base tables. 2. Relationships between two entries in the generic table, via a mapping table, which may be annotated with a semantic relationship. Triples are generated data from the tables to fill in text templates which output turtle. The turtle is parsed and used to populate a triple store. The triple store is rebuilt weekly; there are no incremental updates.
When new terms appear, they are captured in the GCIS. Scripts run periodically and pull information from various sources. The mechanism for assimilating new terms is to provide operators with notifications of unmatched terms; operators then manually associate the new terms with GCIS resources.
Example 1. A new satellite achieves orbit. The information in CEOS reflects this new information. It is pulled into GCIS, and a new entry is created. A new default GCIS identifier is created from a descriptive field (but it may be updated, as described below). The relational database is populated with information that reflects that data source. Since no term yet exists for this in the ceos lexicon, a new one is created which maps the term used by CEOS to the newly created identifier in the GCIS.
Example 2. An instrument on a spacecraft begins to produce new data. A science team processes the data and sends them to an archive. The archive makes the data available. In this case, a new term has been created, and it must be matched to a GCID manually. The new data is ingested, no match occurs. An operator notices this and manually associates the new term with an existing GCID. Example 3. Two organizations use different terms for one satellite. Both organizations distribute data collected by an instrument on board the satellite. In this case, there are two lexicons, one for each organization, and each one has a context which has terms which map to the same GCID. 5.4 URIs
URIs are formed using primary keys in the relational database. The values of the columns comprising the primary key are assembled along with the name of the table, into a URI which corresponds to a row of data in a table. The representation of a resource may involve joining to other related tables.
When the value of a primary key column is changed, a cascading foreign key update will change the values in any related tables. Also, triggers will change the columns in the generic (parent) table. Because the templates are based on information in the database, these changes will automatically propagate to the semantic representation of the resource.
Also, for any changes, a note is written to an audit table which contains the old identifier and the new one. When the GCIS receives a request for a resource that does not exist, it first checks the above audit log. If an entry exists for the requested identifier, this entry is used to construct a URL to which a redirect is then returned.
This mechanism allows for changes to identifiers in the GCIS while continuing to provide consistent endpoints in the API.
Changes to the parent table described above also trigger changes to the lexicon tables. So the complete flow for an identifier change is:
API or web form -> primary key of base table -> primary key of generic table (cascading update) -> gcid entry in list of terms (trigger) -> turtle template (which uses database queries) -> Triple store (by ingesting the rendered template) -> SPARQL endpoint
Identifying changes to terms is important in order to effect changes like the ones above. This requires continuous validation. In order to perform this validation, scripts must periodically check to see if the terms are still valid. If there is a mapping from terms to URLs, this can be accomplished through a simple HEAD request. If not, more advanced techniques may be necessary.
Lexicons are a practical way of mapping identifiers within the earth science community to each other, when uniform identifiers for resources do not exist. The GCIS provides resolvable URIs which serve to fill the gap between organizations with varying terms for the same resource. Providing both a semantic and relational mechanism for storage, and both a semantic and RESTful API allows the GCIS to have the advantages of both architectures, namely referential integrity and backwards compatibility, as well as flexibility and adaptability. While we have seen some success in using cross comparisons of data to improve quality of individual data sources, more work can be done in this area. There is also work to be done in scaling up the number and type of data sources. Another future improvement is to provide useful user interfaces for scalable human disambiguation.
Thanks to the various people and organizations who have contributed to the GCIS and the development of its information model, including Andrew Buddenberg and the Technical Support Unit at NOAA’s National Climatic Data Center, Xiaogang Ma and the group at the RPI’s Tetherless World Constellation, and the University Corporation for Atmospheric Research.