The necessity of manipulating very large amounts of data and the wide availability of computational resources on the Cloud is boosting the popularity of distributed computing in industry. The applicability of model-driven engineering in such scenarios is hampered today by the lack of an efficient model-persistence framework for distributed computing. In this paper we present NeoEMF/HBase, a persistence backend for the Eclipse Modeling Framework (EMF) built on top of the Apache HBase data store. Model distribution is hidden from client applications, that are transparently provided with the model elements they navigate. Access to remote model elements is decentralized, avoiding the bottleneck of a single access point. The persistence model is based on key-value stores that allow for efficient on-demand model persistence.
The availability of large data processing and storage in the Cloud is becoming a
key resource for part of today’s industry, within and outside IT. It offers a
tempting alternative for companies to process, analyze, and discover new data insights,
yet in a cost-efficient manner. Thanks to existing Cloud computing companies,
this facility is extensively available for rent [
One of the principal ingredients behind the success of distributed
processing are distributed storage systems. They are designed to answer to data
processing requirements of distributed and computationally extensive applications,
i.e., wide applicability, scalability, and high performance. Appearing along with
MapReduce [
Another success factor for widespread distributed processing is the appearance of high-level languages for simplifying distribution by a user-friendly syntax (mostly SQL-like). They transparently convert high-level queries into a series of parallelizable jobs that can run in distributed frameworks, such as MapReduce, therefore making distributed application development convenient.
We believe that Model-Driven Engineering (MDE), especially the
query/transformation languages and engines, would be suitable for developing
distributed applications on top of structured data (models). Unfortunately, MDE
misses some fundamental bricks towards building fully distributed
transformation/query engines. In this paper we address one of those components, i.e.
a model-persistence framework for distributed computing. Several distributed
model-persistence frameworks exist today [
In this paper we present NeoEMF/HBase, a persistence backend for EMF
built on top of the Apache HBase data store. NeoEMF/HBase is
transparent w.r.t. model manipulation operations, decentralized, and based on key-value
stores. The tool is open-source and publicly available at the paper’s website2.
This paper is organized as follows: Section 2 presents HBase concepts and
architecture, Section 3 presents the NeoEMF/HBase architecture, data model and
properties; and finally, Section 4 concludes the paper and outlines future work.
1 CDO servers (usually called repositories) are built on top of different data storage
solutions (ranging from relational databases to document-oriented databases).
However, in practice, only relational databases are commonly used, and indeed, only
DB Store [
Apache HBase [
In HBase, data is stored in tables, which are sparse, distributed, persistent multidimensional sorted maps. A map is indexed by a row key, acolumn key, and a timestamp. Each value in the map is an uninterpreted array of bytes.
HBase is built on top of the following concepts [
Fig. 1: Example of a table in HBase/BigTable (extracted from [
Fig. 2 shows how HBase is combined with other Apache technologies to store
and lookup data. Whilst HBase leans on HDFS to store different kind of
configurable size files, ZooKeeper [
Client HMaster
ZooKeeper
DFS Client
HRegionServer g o L H
HRegion
Store ... ...
Store
MemStore
HRegion ... ...
HRegionServer HRegion
HRegionServer
HRegion ...
...
...
...
...
...
DFS Client ...
DFS Client ...
NameNode
DataNode
DataNode
DataNode ...
Fig. 2: HBase architecture the data can be replayed. Stores in a region contain an in-memory data store (MemStore) and a persistent data stores (HFiles, that are persisted into the distributed file system) HFiles are local to each region, and used for actual data storage. The ZooKeeper cluster is responsible of providing the client with the information about both the HRegionServer and the HRegion hosting the row the client is looking up for. This information is cached at the client side, so that a direct communication could be directly setup for the next times without querying the HMaster. When an HRegionServer receives a write request, it sends the request to a specific HRegion. Once the request is processed, data is first written into the MemStore and when certain threshold is met, the MemStore gets flushed into an HFile. 2.3
HDFS is the primary distributed storage used by Hadoop applications as it is designed to optimize distributed processing of multi-structured data. It is well suited for distributed storage and distributed processing using commodity hardware. It is fault tolerant, scalable, and extremely simple to expand. HDFS is optimized for delivering a high throughput of data, and this may be at the expense of latency, which makes it neither suitable nor optimized for atomic model operations. HBase is, on the other hand, a better choice for low-latency access. Moreover, HDFS resources cannot be written concurrently by multiple writers without locking and this results in locking delays. Also writes are always made at the end of the file. Thus, writing in the middle of a file (e.g. changing a value of a feature) involves rewriting the whole file, leading to more significant delays. On the contrary, HBase allows fast random reads and writes. HBase is row-level atomic, i.e. inter-row operations are not atomic, which might lead to a dirty read depending on the data model used. Additionally, HBase only provides five basic data operations (namely, Get, Put, Delete, Scan, and Increment ), meaning that complex operations are delegated to the client appliaction (which, in turn, must implement them as a combination of these simple operations). 3
NeoEMF/HBase Model Manager Persistence Manager Persistence
Backend APIs are consistent between the model-management framework and the persistence driver, keeping the low-level data structures and code accessing the database engine completely decoupled from the modeling framework high level code. Maintaining these uniform APIs between the different levels allows including additional functionality on top of the persistence driver by using the decorator pattern, such as different cache levels.
NeoEMF/HBase offers lightweight on-demand loading and efficient garbage collection. Model changes are automatically reflected in the underlying storage, making changes visible to all the clients. To do so, first we decouple dependencies among objects by assigning a unique identifier to all model objects, and then: – To implement lightweight on-demand loading and saving, for each live model object, we create a lightweight delegate object that is in charge of on-demand loading the element data and keeping track of the element’s state. Delegates load and save data from the persistence backend by using the object’s unique identifier. – For efficient garbage collection in the Java Runtime Environment, we avoid to maintain hard Java references among model objects, so that the garbage collector can deallocate any model object that is not directly referenced by the application. 3.1
We have designed the underlying data model of NeoEMF/HBase to minimize the data interactions of each method of the EMF model access API. The design takes advantage of the unique identifier defined in the previous section to flatten the graph structure into a set of key-value mappings.
Fig. 4a shows a small excerpt of a possible Java metamodel that we will use to exemplify the data model. This metamodel describes Java programs in terms of Packages, ClassDeclarations, BodyDeclarations, and Modifiers. A Package is a named container that groups a set of ClassDeclarations through the ownedElements composition. A ClassDeclaration contains a name and a set of
Package name : String ownedElements ClassDeclaration
* name : String bodyDeclarations *
Modifier modifier BodyDeclaration visibility : VisibilityKind 1 name : String
VisibilityKind none public private protected (a)
p1 : Package name : ’package1’
bodyDeclarations b1 : BodyDeclaration name : ’bodyDecl1’ modifier
m1 : Modifier visibility : public ownedElements c1 : ClassDeclaration
name : ’class1’ bodyDeclarations b2 : BodyDeclaration name : ’bodyDecl2’ modifier
m2 : Modifier visibility : public (b) BodyDeclarations. Finally, a BodyDeclaration contains a name, and its visibility is described by a single Modifier.
Fig. 4b shows a sample instance of the Java metamodel, i.e., a graph of objects conforming with the metamodel structure. The model contains a single Package (package1), containing only one ClassDeclaration (class1). The Class contains the bodyDecl1 and bodyDecl2 BodyDeclarations. Both of them are public.
NeoEMF/HBase uses a single table with three column families to store models’ information: (i) a property column family, that keeps all objects’ data stored together; (ii) a type column family, that tracks how objects interact with the meta-level (such as the instance of relationships); and (iii) a containment column family, that defines the models’ structure in terms of containment references. Table 13 shows how the sample instance in Fig. 4b is represented using this structure.
As Table 1 shows, row keys are the object unique identifier. The property column family stores the objects’ actual data. As it can be seen, not all rows have a value for a given column. How data is stored depends on the property type and cardinality (i.e., upper bound). For example, values for single-valued attributes (like the name, which stored in the name column) are directly saved as a single literal value; while values for many-valued attributes are saved as an array of single literal values (Fig. 4b does not contain an example of this). Values for single-valued references, such as the modifier containment reference from BodyDeclaration to Modifier, are stored as a single value (corresponding to the identifier of the referenced object). Examples of this are the cells for hb1, modifieri and hb2, modifieri which contain the values ’m1’ and ’m2’ respectively. Finally, multi-valued references are stored as an array containing the literal identifiers of the referenced objects. An example of this is the bodyDeclarations containment reference, from ClassDeclaration to BodyDeclaration, that for the case of the c1 object is stored as { ’b1’, ’b2’ } in the hc1, bodyDeclarationsi cell.
Structurally, EMF models are trees (a characteristic inherited from its XMLbased representation). That implies that every non-volatile object (except the 3 Actual rows have been split for improved readability ’package1’ ’class1’ ’bodyDecl1’ ’bodyDecl’ { ’c1’ }
{ ’b1’, ’b2’ } container containment
feature ’ROOT’ ’p1’ ’c1’ ’c1’ ’b1’ ’b2’
’eContents’ ’ownedElements’ ’bodyDeclarations’ ’bodyDeclarations’ ’modifiers’ ’modifiers’
nsURI ’http://java’ ’http://java’ ’http://java’ ’http://java’ ’http://java’ ’http://java’ ’http://java’ root object ) must be contained within another object (i.e., referenced from another object via a containment reference). The containment column family maintains a record of which is the container for every persisted object. The container column records the identifier of the container object, while the feature column records the name of the property that relates the container object with the child object (i.e., the object to which the row corresponds). Table 1 shows that, for example, the container of the Package p1 is ROOT through the eContents property (i.e., it is a root object and is not contained by any other object). In the next row we find the entry that describes that the Class c1 is contained in the Package p1 through the ownedElements property.
The type column family groups the type information by means of the nsURI and EClass columns. For example, the table specifies the element p1 is an instance of the Package class of the Java metamodel (that is identified by the http://java nsURI ). 3.2
NeoEMF/HBase is designed as a simple persistence layer that maintains the
same semantics as the standard EMF. Modifications in models stored using
NeoEMF/HBase are directly propagated to the underlying storage, making
changes visible to all possible readers immediatly. As in standard EMF, no
transactional support is explicitly provided, and as such, ACID properties [
Consistency — Modifications on object’s properties are always consistent using a compare-and-swap mechanism. In the case of modifications involving changes in more than one object, consistency is only guaranteed when the model is modified to grow monotonically (i.e., only new information is added, and no already existing data is deleted nor modified).
Isolation — Reads on a given object always succeeds and always give a view of the object’s latest valid state.
Durability — Modifications on a given object are always reflected in the underlying storage, even in the case of a Data Node failure, thanks to the replication capabilities provided by HBase.
These properties allow the use of NeoEMF/HBase as the persistence backend for distributed and concurrent model transformations, since reads in the source model are consistent and always success; and the creation of the target model is a building process that creates a model that grows monotonically. 4
In this paper we have outlined NeoEMF/HBase, an on-demand, memoryfriendly persistence layer for distributed and decentralized model persistence. Decentralized model persistence is useful in scenarios where multiple clients may access models when performing distributed computing. NeoEMF/HBase is built on top of HBase, a distributed, scalable, versioned and non-relationals big data store, specially designed to run together with Apache Hadoop.
NeoEMF/HBase takes advantage of the HBase properties by using a simple data model that minimizes data dependencies among stored objects. More specifically, NeoEMF/HBase exploits the row-locking mechanisms of HBase to provide limited ACID properties without requiring the use of transactions, which may increase latency in model operations. NeoEMF/HBase provides ACID properties at the object level, and guarantees that: (i) object queries always return the last valid state of an object; (ii) attribute modifications always succeed and produce a consistent model; and (iii) modifications of references which make the model grow monotonically always succeed and produce a consistent model.
Previous work [
Additionally, a more advanced locking mechanism allowing arbitrary object locks will be implemented. Such a mechanism will provide multi-object ACID properties to the framework, allowing client applications to implement the synchronization logic to perform arbitrary, distributed and concurrent modifications. This work is partially supported by the MONDO (EU ICT-611125) project.