Relational database management systems (RDBMS) and NoSQL database systems have been built in order to meet di erent requirements. Organisations are therefore increasingly using hybrid data persistence architectures, that we call hybrid database systems or hybrid polystores. The evolution of such systems is more complex due to potential data duplication, migration across di erent technologies and data heterogeneity. Existing work on NoSQL database evolution exposes the di erent problems regarding data heterogeneity and mainly provide technology-speci c solutions. In this PhD research we propose to apply a schema engineering approach to handle hybrid database system evolution at a conceptual level. Our starting point is an existing uni ed conceptual data model for hybrid databases. Our thesis objectives include (1) to enrich this conceptual data model with generic schema evolution operators, (2) to specify the semantics of those operators depending on the underlying data models, (3) to design a generic evolution framework and (4) to develop its proof-of-concept implementation. Our approach will help to propagate conceptual schema changes to all impacted software artefacts, namely native data structures, database contents, queries and programs.
According to Sadalage et al. [
In the current situation the developer is left alone in the evolution process. Developers have rst to identify which system artefacts are impacted by the considered schema evolution scenario, and how to propagate the schema evolution operations in terms of data migration, native data structure changes and query adaptations. Let us consider an industrial hybrid data-intensive system made up of several application programs that jointly access a document-oriented database, a relational database and a key-value store. Let us assume a simple schema evolution scenario: changing the name of an attribute in an entity type called products. This change would possibly require the adaptation of the native database structures (if the products are stored in a relational database), of the database contents (if the products are stored as a document database collection). It would also involve the rewriting of all queries manipulating the products with an explicit reference to the renamed attribute. Furthermore, some programs may also access the products via mapped data access objects or have some programming logic relying on the renamed attribute. The related code fragments would therefore also require some adaptations. The context of hybrid database system adds another layer of di culty, as in the above scenario we could imagine that the products are stored both in the document-oriented database and in a relational database, i.e., with some (partial) overlapping between both data stores.
We can see, already through a simple evolution scenario, that schema evolution in a hybrid database system may prove complex, time-consuming and error-prone for database administrators and developers, as multiple and heterogeneous software artefacts have to be consistently modi ed.
We argue that this evolution process would be easier to handle by means of an integrated evolution framework, that would de ne generic hybrid schema evolution operations. Those evolution operations should be expressed on top of a uni ed, conceptual data model, covering multiple heterogeneous data modeling paradigms. This model is the entry point of the evolution process as it enables the propagation operations, to be applied automatically to the underlying native database(s) and to the related software artefacts (data, queries, programs).
We identify the following research questions that we will try to answer during this PhD research: RQ 1. How can we specify generic schema evolution operators for hybrid database systems? In this question we identify which conceptual schema evolution operators are needed to evolve a hybrid database system. RQ 2. How can we design an evolution framework propagating such evolution operations to all impacted software artefacts? An evolution operator may have multiple impacts on several artefacts of the system. This question will establish a generic impact matrix that speci es atomic, paradigm-speci c propagating changes to apply depending on the mapping between the conceptual data model and the native backends.
RQ 3. To what extent can this framework actually help users in evolving hybrid database systems? We plan to develop a proof-of concept implementation of our framework and to evaluate, in realistic conditions, its usability, completeness, correctness, and tness-for-use.
NoSQL systems can be grouped into four main di erent data
models, each having its speci c requirements and
advantages. We can cite key value stores, document databases,
column databases or graph databases and reference Hecht
and Jablonski [
NoSQL data models are mostly considered as schema-less
models as they are oriented towards exibility. However
Sadalage et al. [
Several authors have proposed a generic model
combining heterogeneous NoSQL data models. Atzeni et al. [
Our work relies on a TyphonML [
Curino et al. [
Database evolution according to Cleve et al.[
Existing research on NoSQL database evolution is more
recent, yet several related problems have already been
explored. Some approaches are dedicated to how to specify
and formalize NoSQL database evolution, i.e, NotaQL [
The core expected contribution of our PhD thesis is a generic framework for supporting schema evolution in hybrid database systems. We describe this framework below.
Inputs MSource, represents the hybrid database schema at both conceptual and logical levels. fEOg is a set of schema evolution operators to apply to MSource. fDBSourceg is the set of source native data structures and data instances. fQSourceg is the set of existing queries expressed on MSource. Outputs MT arget is the source model adapted according to the evolution operators in fEOg. fDBT argetg represents the set of database structures and instances adapted according to those operators. fQT argetg gives, when possible, equivalent queries as fQSourceg, but expressed on MT arget . fDAOg is a set of database access classes on speci c drivers.
Figure 1 summarizes the di erent components of our framework. The rest of this section will further detail each of those components.
Our approach starts from a data model that allows one to design hybrid polystore systems. The design principles described in this section can be applied to any model and several were listed above in Section 2. In our work we used the Typhon Modeling Language (TyphonML) developed by one of our research partners of the Typhon project (see Section 5). Using such a uni ed model to evolve hybrid database systems reduces the complexity by keeping software artefacts consistent with each other. We implement a top-down evolution approach of evolution from model to the related software artefacts (data, queries, programs). The model is extended to allow users to trigger the execution of evolution operations.
The rst step of our PhD thesis was to de ne and specify the evolution operations that a user can apply to the hybrid polystore data model. Those operations are the entry point of the whole evolution process. An Evolution Operator (EO) is a combination of a domain model object and an evolution operation. The domain model object can be either entity type, attribute, relationship, or identi er. The available evolution operations span from simple changes, manipulating a single domain object add, remove, rename, modify to more complex ones such as horizontal split, vertical split, merge, or migrate. Figure 2 gives an example list of EOs, that can be expressed in our extended data model syntax. evolutionOperators [ add attribute rating : int to Review , add relation responses to Review -> Comment [0..*] , merge entities User CreditCard as User
The impact matrix is the central point of our framework as it de nes the atomic operations that need be applied to actually propagate the polystore schema evolution operators to related software artefacts. The rows of the matrix correspond to the available Evolution Operators. The columns of the matrix then characterize each of the operators according to several dimensions; each in uencing the propagating operations that should be applied. The rst dimension speci es the underlying data models of the object(s) subject to evolution. The possible values of this dimension include relational, document, key-value or graph databases. The second dimension concerns the chosen modeling strategy, as explained in Section 2. The third dimension relates to the chosen data migration strategy. Each cell of the impact matrix speci es the set of atomic, paradigm-speci c operations to apply to the polystore artefacts (native structures, data, queries, etc.), in order to propagate the requested EOs according to the dimensions' values.
The set of atomic operations will be an exhaustive list of function speci cations. Their goal is to enable the actual transformation required to propagate the desired schema evolution. They are classi ed into four categories, depending on which software artefacts are involved.
As we want to keep the polystore conceptual model and the related polystore artefacts consistent with each other, each evolution scenario has to be triggered by means of a list of evolution operators expressed on top of the conceptual polystore schema. Our framework applies changes corresponding to each evolution operator speci ed in the fEOg list. Each time an EO is executed, a new intermediary conceptual model is produced. This intermediary model is then used as the source schema for the following operators. The nal output is a new conceptual model resulting from the application of all evolution operators.
According to the underlying logical mappings some EOs require the adaptation of the native data structures that are mapped with the conceptual schema objects subject to evolution. This adaptation, in the case of a relational database, translates as SQL DDL commands, for example create table, alter table, create index, add constraint,. . . . Equivalent structure manipulation for NoSQL data models include create collection, create index for document data model, create table, create column family for column databases. Key value and graph data models are pure data and do not have equivalent structure objects. Note that in our work we have not planned to take into account the speci c physical aspects of NoSQL databases, meaning that we do not plan to handle the physical storage of data and cloud computing aspects.
The propagation of EOs to data instances aims at transforming/migrating the data in order to make them comply with the target polystore data model/paradigm. As an example, a possible strategy (EAO, see Section 2) to represent a relationship between two conceptual entities in a documentoriented database is to use two speci c collections, one for each entity type. The source entity of the relationship will then have a reference attribute referring to the identi er of the other collection. In this case, renaming an attribute of the target entity type would necessitate to rename the corresponding attribute for each data object in the target collection. Another representation strategy (ETF) of this relationship would consist in using a single document collection, with each data object including the referenced data object as a nested object. The same renaming operation would, instead, require the adaptation of the nested objects.
EO operations may also require the adaptation of existing database queries, that would become invalid due to the applied schema changes. Depending on the semantics-preserving nature of each EO involved in the scenario, it will be possible or not to translate the source database queries into equivalent queries expressed on the target polystore schema, using rule based transformations. Simple Data Access Object classes in programs can also be generated or modi ed according the received EOs.
The schema evolution framework presented in this paper has been partially implemented in the context of the Typhon EU H2020 research project. A total of 18 evolution operators are now fully speci ed and implemented. At this stage of our research, we only consider mappings to relational and document-oriented databases, and we follow an eager data migration strategy. In the next coming months, our framework implementation will be used by four industrial partners, by considering realistic use-case scenarios. Future work in this PhD thesis includes (1) the speci cation of more complex and composite evolution operators, (2) the implementation of other data migration strategies (lazy, predictive or query based) in the impact matrix, and (3) the extension of our framework to key-value and columnoriented data models.
This research was partly carried out in the context of the EU H2020 Typhon research project, the goal of which is to provide an industry-validated methodology and integrated technical o ering for designing, developing, querying, evolving, analyzing and monitoring scalable hybrid data persistence architectures. This PhD thesis research is supported by the Excellence of Science project 30446992 SECOASSIST, funded by the F.R.S.-FNRS and the FWO.