When software requirements change, databases used by an application may evolve, including the database models (conceptual model and schema), which may require data migrations to maintain data integrity. In some databases such as NoSQL column family databases, the tables of the database often store repeated data, as schema denormalization is encouraged to achieve the best performance. This makes data integrity maintenance more complex, as a single conceptual model change may trigger several changes in the schema. In this work, we propose using a model-driven engineering approach named MoDEvo for data migration in NoSQL column family databases to maintain data integrity after the schema evolves. Using a motivating example as a case study from an open source project that requires data migration, we describe MoDEvo and use it to illustrate how MoDEvo determined these data migrations.
The requirements of a project determine how the
tual model during the design phase. This is especially
important for some database types such as NoSQL
column family databases (e.g. Apache Cassandra) where
the schema is usually denormalized [
In NoSQL column family databases, the schema is more lfexible than in a relational database and does not have integrity constraints, which makes mistakes during schema evolution more likely. A single change in one table must Published in the Workshop Proceedings of the EDBT/ICDT 2023 Joint 2.
We use as a motivating example a schema change from the open-source project ‘team’ was added to the primary key of the table “issuer_idp”. As it is not possible to directly alter the pri1 in which the column
mary key of a table, a new table: “issuer_idp_v2” was created, which is illustrated in Figure 1.
must be migrated to “issuer_idp_v2”, adding the
appropriate value for column ‘team’ to complete each row. Values
of column ’team’ must be obtained from table “idp” as it
is displayed in Figure 2.
(Data migration model creation), MoDEvo determines
through model transformations implemented in ATL [
It is important to note, that this migration is still not execution), MoDEvo sends this script to a migration being executed by the developers of the project. They engine (e.g. Apache Spark) to execute the script. We have left the previous table to be queried for the data that focus on the two first phases. was previously stored and a TODO task to migrate the A Data Migration model (Figure 4) contains a Migradata in the future. Maintaining the old table “issuer_idp” tionTable element for each target table that requires data jeopardizes data integrity, as both tables are actually stor- migrations in some of its columns. For each of these taring diferent data, although they are intended to store the get columns, a MigrationCol element is defined, which same relationship between entities “issuer” and “idp”. In determines this target column that receives data and the the next section, we propose MoDEvo to avoid scenarios source column that provides the data. like this one. The target and source columns are specified through references to Target and Source elements, which likewise contain the references ‘Data’ and ‘Key’ to schema column
In this section we propose MoDEvo, an MDE approach that determines the required migrations of data to maintain data integrity when the schema evolves. Figure 3 displays the MoDEvo process divided in three phases (middle row) alongside its inputs (top row) and results of each phase (bottom row), which are also inputs in the following phase.
In the first two phases, MoDEvo receives the information of the evolution of the schema through 3 input models: 1) the conceptual model, 2) the schema before the change and, 3) the schema change. In the first phase • ‘Data’: Indicates the source column from where data will be obtained and the target column where data will be inserted. • ‘Key’: Indicates a functional dependency that the ‘Data’ column has with other columns, which makes it possible to determine the specific data to insert in each row of the ‘Data’ target column.
MoDEvo determines a functional dependency if the following condition is met: Being a column Datac mapped to a non-key attribute Dataa and a set of columns Keyc, mapped to a set of key attributes Keya, if Keya → Dataa then Keyc → Datac.
databases such as MongoDB by determining the required data migrations. Focusing on column family DBs, Störl
When applying MoDEvo to the case study from the mo- et al have addressed other issues, such as optimizing the
tivating example, it first analyzes the conceptual model, performance of these databases [
Using the generated model, MoDEvo creates through We made a first approach for database evolution [
After the migration is performed, the previous table “is- [
Database migrations for column family databases have provides the migrations required to maintain data
inbeen approached in several work by Störl et al. [
MoDEvo also reduces the time that developers need to employ when evolving the database, as it automatically determines what needs to be migrated. Developers will have a script with the database statements that are required to perform all the data migrations needed to maintain the data integrity.
As future work, we plan to complete MoDEvo by
developing its third phase and migrate data by sending the
data migration script to a migration engine. We also
plan to combine our previous work for schema evolution
[
76956-C3-1-R] funded by the Spanish Ministry of Economy and Competitiveness, [PID2019-105455GB-C32] funded by MCIN/ AEI/10.13039/501100011033 and the Severo Ochoa pre-doctoral grant PA-21-PF-BP20-184.