In this paper we present CM-Well, a clustered, horizontally scalable data store for linked data. We discuss the key architectural principals of the system and introduce distinguishing features. Thomson Reuters has been running CM-Well in production for over two years, across multiple data centers. The system provides data, via API, to a number of our online products.
CM-Well is based on a clustered architecture (Fig 1). Philosophically, the application is similar to a Linux distribution in that it consists of a well tested
5 https://github.com/thomsonreuters/CM-Well integration of a number of open source packages including Akka6, Cassandra7, ElasticSearch8, Jena9 and Kafka10.
Each node in the cluster has the same hardware con guration and runs a set of processes with no single point of failure. Singleton control roles are moved between nodes on failure, implemented on a \self healing" principle. In a typical deployment, the cluster is fronted by an HTTP load balancer such as haproxy11. The majority of the application is written in the Scala language. Individual triples are grouped by subject to form an \Infoton": the basic unit of storage. These information objects are stored in Cassandra and invert indexed by ElasticSearch. While writes require a full rewrite of the infoton, reads by subject can be served from one node and return all triples for the subject. This grouping of triples forms a fundamental design tradeo between read and write performance but has been found to o er good horizontal scale performance: on our production cluster, API response time for all information on a subject is around 10ms.
7 http://cassandra.apache.org/ 8 https://github.com/elastic/elasticsearch 9 https://jena.apache.org 10 https://kafka.apache.org 11 http://www.haproxy.org
Every Infoton write is treated as immutable; this approach permits eventually consistent replication across distributed data centers with secondary instances subscribing to a change log and applying writes locally. Depending on read/write tradeo , all slaves can replicate from a single master or slaves can form a branching chain. 2.2
While we have found CM-Well capable of balancing read and write tra c, in most applications consumers wrap CM-Well with a data-mart implemented to meet the speci c needs of their application. Depending on use case, this mart might be as simple as a caching reverse HTTP proxy to reduce read load or deployment of a separate search cluster which structures data to meet required performance targets. 3
One of the motivating factors for the development of CM-Well was our perception that key requirements were not met by other tools; either open or closed source. Integrating our own solution permitted us to focus on key di erentiating features. 3.1
The API surface for CM-Well includes a REST based API for boolean querying by predicate. Any query can be used as a (server-side stateless) subscription; returning a time ordered sequence of Infotons matching the query.
Each stream API call returns a continuation token in the HTTP response headers. When passed in a subsequent request, this token is used as a point-intime marker with the stream continuing from the end of the previous response. Additionally, an optional parameter will instruct CM-Well to include tombstones for deleted data, e ectively treating each query as a FIFO queue of content events.
This feature is key to the `information in motion' design of CM-Well permitting downstream stores to subscribe to relevant subsets of data de ned by query. 3.2
Currently, CM-Well supports two types of SPARQL query: sub and full graph. When querying, the sub-graph solution consists of two sub-steps: 1) one or more queries, routed to ElasticSearch, are rst used to select a candidate set of nodes from our data; 2) Such candidate nodes are then loaded to a Jena triple store on a single machine for SPARQL execution. We are also experimenting with a second form which builds an execution plan for input ElasticSearch queries based on data statistics relevant to the SPARQL query. Both solutions are limited to the underlying characteristics of ElasticSearch, such maximum results window of 10,000. Hence, wide ranging queries can miss triples of relevance.
Much of the data we load to CM-Well is highly normalized and often requires simpli cation to ease downstream consumption. Our trigger mechanism permits a sensor to be created, based on a query, which on new data invokes a SPARQL CONSTRUCT statement. This then generates new predicates.
For example, in our organization ontology, company names are modeled as their own type <OrganizationName>. Each name includes date based \from" and \to" predicates to model company name changes over time. While this temporal model is powerful, it complicates the simple retrieval of a company, since one must traverse the set of attached names. A trigger is used to create (or update) a \ attened" name predicate on the organization entity on update of related organization name entity. 4
In this paper we have brie y presented our approach to managing & distributing large volumes of linked data using CM-Well. We believe the application presents a cost e ective way of building and distributing linked datasets and are pleased to be able to share with the wider community via open source: without which CM-Well wouldn't be possible.
A video demo to support this paper can be found on Vimeo12: the demo shows a subset of the data from the permid.org dataset, update via API, immutable history, retrieval by API and our implementation of \full-graph" SPARQL. Acknowledgments. The authors would like to acknowledge the e orts of the entire CM-Well development team, both current and historical, in addition to Thomson Reuters for supporting the open sourcing of CM-Well.