Database Schema Evolution in Microservices

Abstract

Microservices architecture has emerged as a dominant model for designing cloud-based applications. In this architecture, modular and heterogeneous services, independently deployed and sometimes geo-distributed, dynamically scale and interact with each other to respond to user requests. Typically, each service has its own database(s) and shares information through APIs calls. Facilitating software evolution is one of the main motivations for adopting a microservices architecture. Paradoxically, from a database point of view, recent surveys reveal that database schema evolution remains among the most pressing data management challenges for microservices architecture developers. While there are many tools available in the literature for supporting the evolution of microservices architecture, none of them is specifically designed to address the problem of database schema evolution. To tackle this challenge, the primary objective of this PhD thesis is to provide developers with a set of efective and eficient tools for automatically supporting database schema evolution in microservices architecture,

-

thereby reducing their burden in cloud-based applications evolution. © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License pressing data management challenges for microservices microservice, database schema, automated evolution

1. Introduction

Over the past few years, microservices architectures have

gained significant popularity among developers and industry leaders such as Netflix, Google, and Amazon, who leverage this architectural style to design cloud-based applications [ 1, 2, 3 ].

A microservices architecture is defined by several key properties.

Modularity is the most commonly cited.

Within this architectural style, microservices are organized as a collection of small, loosely coupled services, with each service managing its own responsibilities and data, thus respecting separation of concerns [ 4, 5, 6, 3, 7, 1, 8 ]. Another property is the use of heterogeneous technologies. Specifically, each microservice may be developed using a diverse set of polyglot and hybrid technologies, including persistence technologies such as relational and NoSQL databases. Each microservice is technologically independent and manages its own codebase and database(s) [ 5, 8, 3, 6, 7, 9, 2 ]. Furthermore, microservices should respect deployability property. They are often deployed on public or private cloud infrastructures, accompanied by mechanisms such as autoscaling and geographic distribution. Each microservice and its database(s) can be rapidly and independently redeployed [ 10, 6, 7, 11 ]. Due to modularity property, nEvelop-O LGOBE (M. André)

Ops, and CI/CD methodologies, developers can quickly

evolve and maintain microservices and databases by detaching from certain constraints [ 4, 6, 3, 11, 1, 2 ].

Despite evolutivity property promises, microservices

developers continue to face several issues, particularly in regards to database management. Recent surveys reveal that database schema evolution remains one of the most architecture developers [ 5 ].

2. Related work

Indeed, practitioners often lack resources to assist them in microservices evolution [ 8 ], especially in regards to their databases. Developers have a real need To prepare that research, works related to microservices of support tools able to understand, explicit, ensure, vi- and databases evolution have been reviewed. Even if sualize and monitor, over time, databases, schemas, data, they do not focus on both microservices and databases transactions, and constraints [ 6, 10, 9 ]. Particularly, it together, they represent good inspiration for reverse enconcerns data replication, implicit constraints such as gineering, modeling, generation, evolution, visualization, implicit foreign keys, message structure and database and monitoring of microservice databases. schema change propagation distributed across several Reverse engineering. Reverse engineering of mimicroservices [ 5, 6 ]. In that context, dealing with CAP croservices aims to recover a comprehensive holistic view theorem [ 9 ] and ensuring ACID is a true challenge [ 8 ]. of an existing system. Various approaches exist for that This is even more significant considering that modern purpose. Some authors exploit the version history by systems employ hybrid databases composed of heteroge- mining software repositories to identify changes helpneous database technologies, including NoSQL, particu- ing to understand the current version [ 3, 18 ]. Others larly known for their absence of fully explicit schemas use techniques of static and dynamic analysis. In an ofwhere consistency is usually delegated to client programs. fline and white-box approach, static analysis analyzes the This complexifies modeling and evolution of database source code and API contract specifications. In a runtime schemas, thereby further justifying the need for support and black-box approach, dynamic analysis inspects logs, tools [ 14, 15 ]. Moreover, even if DevOps gains in popu- deployment platform metrics, and any other traces. Both larity, less attention is given to automation of database approaches are implemented in tools like MICROLYZE, schema evolution [ 16 ]. While microservices are often MicroART, Zipkin, etc. [ 7, 11, 19, 1, 20, 21 ]. Techniques associated with quick and frequent releases, there is a and tools operate either at software or at infrastructure potential risk of microservices dependencies breaking level [ 13 ]. These approaches primarily examine microserwhen introducing a change [ 6 ]. This is particularly true vices architecture but lack of persistence perspective in when modifications involve substantial database schema a distributed and heterogeneous context. changes leading to an expensive cost and even a total Modeling. Modeling in the context of microservices downtime [ 16, 17 ]. Practitioners are directly afected by can serve for representing reverse engineering output or those problems while spending precious time in manual simply for designing a new architecture. Additionally to operations [ 9 ]. traditional models such as UML, Archimate, and BPMN

In that way, this research aims to answer this main [ 19 ], others are specifically dedicated to microservices. research question: RQ. How to automate database Some propose to combine information from diferent schema evolution in microservices architecture system layers such as instance, architectural and infrasto reduce developers burden? The following sub- tructure layers [ 3 ] or service, infrastructure and interacresearch questions structure the work in more details. tion layers [ 13 ]. Other focuses on distinction between The first four ( ■ ) will be the focus of this thesis. Ac- abstract types of components and their concrete deploycessorily, the last two (□ ), to be further refined, will be ment instance configurations [ 12 ]. Certain models lead addressed collectively through collaborative work by the to the creation of Domain Specific Language (DSL) [ 1 ]. In RAINDROP project researchers. the same way, creating a modeling language specifically for databases schemas is also studied. For instance, a DSL ■ SRQ1. [Reverse engineering]: How to automate exists for specifying hybrid polystores schemas [ 22 ] and rsecvheermsea?engineering of a cross-microservices database another for abstracting database schema changes coupled ■ SmRicQro2s.ervi[cMesoddaetalibnagse]:schHemowa? to model a cross- taoppthroeapchro,gsormame rceosdeear[c2h3i]n.vFeisntaigllayt,esinhaowmteotad-emsiogdnewlinelgl ■ SRQ3. [Generation]: How to automate generation a modeling language in the context of cloud architecture of artefacts according to a cross-microservice database [ 24 ]. Despite interesting directions, no work considers schema? modeling distributed and heterogeneous microservices ■ SRQ4. [Evolution]: How to automate evolution of and databases jointly.

a cross-microservice database schema while propagat- Generation. Accordingly to modeling, DSLs ofer caing changes? pabilities like code generation [ 24 ]. For instance, a mod□ SRQ5. [Visualization]: How to visualize a cross- eling language representing hybrid polystores schemas microservice database schema? plans an extension able to generate data manipulation □ SRQ6. [Monitoring]: How to monitor microservices APIs [ 22 ]. Again, no work is specifically dedicated to architecture databases to prepare potential evolution microservice database artefact generation in a heterogeof its schema? neous and distributed context.

Evolution. Some evolution approaches suggest Microservices architecture Reverse engineering Reversereepnogritneering reusing a model as a starting point for evaluating changes impact between a current and a future version [ 3 ]. Retrospectively and with support of reverse engineering, infor- Sources Rudnattiame mation recovered is used for understanding architectural Microservices data changes. Combining modeling and reverse engineer- meta schema Modeling ing approaches help for decisions making (e.g., splitting or merging microservices). Recovered indicators (e.g., size of a microservice, number of interfaces, coupling index) take part in problem identification and change Generated code impact evaluation [ 13 ]. Knowing that evolving microser- Generation vices also means evolving their database(s), it is clear that those techniques are relevant in the context of distributed and heterogeneous microservice database schema evo- Evolution lution. However, no work currently put together these recommendations Evolution approaches and tools despite the need [ 16, 17 ].

Visualization. Many visualizations exist for representing graphically microservices architecture. The most Evoluted system tphoepuinlaterrtdaekpeetnhdeenfocriems o[1f1g,r2a5p]h. sDheipgehnlidgihntginognstpoeoclisfic,avllayri- Visualization visualization ous visualizations are considered such as metaphor-based visualization, 3D visualization, and virtual or augmented reality visualizations. Visualization is closely linked to Monitoring report modeling, sometimes being its graphical representation. Monitoring Hence, some switch between diferent views and combination of them like architectural, process, or data views among many others depending on the targeted abstract level [ 19, 21 ]. Unfortunately, these views stop at the microservices level and do not go into the details of their Figure 1: Example of an evolution process supported by envidistributed and heterogeneous databases. Visualizations sioned tools. of databases exist but none are specifically applied to microservices databases [ 26 ].

Monitoring. Microservices monitoring aims to col- Figure 1 illustrates an example of evolution process lect real-time data (e.g., runtime metrics) for following up involving all support tools (that could be used indepensystem behaviour. Even if tools and practices appear like dently without any order constraints). reverse engineering, specifically for dynamic analysis, Reverse engineering. The reverse engineering tool the approach is more centred on the instant. Goals are will help developers to analyze a given heterogeneous mainly focused on microservice current state checking and distributed microservices architecture (sources and and changes detection rather than documentation recov- runtime data) under a data-oriented vision (e.g., concepts, ery. Monitoring is essential for preparing any evolution associations, constraints, database accesses, data locawhile making decision based on data collected. Those tions, etc.) to produce a report as a basis for the microsertasks are usually ensured by tools such as OpenTelemetry, vices data meta schema. The scientific result will be an Kiali, Zipkin, Jaeger, Prometheus [ 11, 19, 21 ]. Regrettably, algorithm based on static and dynamic analyses. analyses often ignore the database perspective. There Modeling. The modeling tool will help developers to is a need for tools to monitor microservices messages model a microservices architecture, its components and transit through the entire distributed and heterogeneous its (cross-)constraints, new or existing, to produce or edit architecture, from API endpoints to inside the database. a microservices data meta schema. The scientific output will be a DSL defining microservice data meta schemas and evolution operators. 3. Research direction Generation. The generation tool will help developers to generate, based on the microservices data meta This PhD study aims to address weaknesses evoked in schema, useful generated code for microservices database the related work by designing and developing support (e.g., database creation script, additional code for cross tools for microservices database schema evolution. implicit constraints, etc.). The scientific output will be an algorithm able to generate those artefacts.

Evolution. The evolution tool will help developers to evolve a cross microservices database schema based on the microservices data meta schema. This process will try to generate, based on evolution operators, what-if scenarios and smells detected, recommendations (e.g., implicit constraint materialization, databases split, unused data structure deletion, etc.) responding to developers goals (e.g. consistency, performance, code quality, etc.).

The scientific output will be an algorithm able to produce those recommendations and integration in CI/CD tools.

Visualization. The visualization tool will help developers to provide visualizations of the current or evolved microservices data meta schema and its details (e.g., implicit constraints, metrics, changes impact, etc.). The scientific output will be several graphical models customizable on purpose.

Monitoring. The monitoring tool will help developers in real-time microservice architecture monitoring while producing a runtime data report (e.g., message exchanges, microservices states, critical paths in calls, etc.) specifically related to the microservices data meta schema.

The scientific output will be several interactive graphical models and monitoring algorithms.

As highlighted, the backbone of this contribution is the microservices data meta schema. This representation will model all data-related aspects in a microservice architecture. It will contribute in the whole evolution process of microservices database schema through proposed tools and across intermediary artefacts.

4. Conclusion

In summary, this PhD thesis aims to address challenges related to automated database schema evolution in microservices architecture. The research aims to provide supporting tools reducing developers burden. Specifically, the research will focus on automatic reverse engineering, modeling, generation, evolution, visualization, and monitoring of database schema in microservices. The reverse engineering tool is currently under development with focus on static analysis of JS REST APIs and NoSQL databases.

Acknowledgments

This PhD project is part of the RAINDROP project studying cross-stack adaptations for the edgification of microservices. The project is supported by the subsidy ARC funded by the Wallonia-Brussels Federation (Belgium).

References

[1] G. Granchelli , M. Cardarelli , P. Di Francesco , I. Malavolta , L. Iovino , A. Di Salle , Towards recovering the software architecture of microservice-based systems , in: 2017 IEEE International Conference on Software Architecture Workshops (ICSAW) , 2017 .

[2] L. P. Tizzei , L. Azevedo , E. Soares, R. Thiago , R. Costa , On the maintenance of a scientific application based on microservices: an experience report , in: 2020 IEEE International Conference on Web Services (ICWS) , 2020 , pp. 102 - 109 .

[3] A. R. Sampaio , H. Kadiyala , B. Hu , J. Steinbacher , T. Erwin , N. Rosa , I. Beschastnikh , J. Rubin , Supporting microservice evolution, in: 2017 IEEE international conference on software maintenance and evolution (ICSME) , IEEE, 2017 , pp. 539 - 543 .

[4] J. Lewis , M. Fowler , Microservices: a definition of this new architectural term , MartinFowler. com 25 ( 2014 ) 12 .

[5] R. Laigner , Y. Zhou , M. A. V. Salles , Y. Liu , M. Kalinowski , Data management in microservices: State of the practice, challenges, and research directions , Proc. VLDB Endow . 14 ( 2021 ) 3348 - 3361 .

[6] H. Chawla , H. Kathuria , H. Chawla , H. Kathuria , Evolution of microservices architecture, Building Microservices Applications on Microsoft Azure: Designing , Developing, Deploying, and Monitoring ( 2019 ) 1 - 20 .

[7] A. Bakhtin , A . Al Maruf, T. Cerny , D. Taibi , Survey on tools and techniques detecting microservice api patterns , in: 2022 IEEE International Conference on Services Computing (SCC) , IEEE, 2022 , pp. 31 - 38 .

[8] R. M. Munaf , J. Ahmed , F. Khakwani , T. Rana, Microservices architecture: Challenges and proposed conceptual design , in: 2019 International Conference on Communication Technologies (ComTech) , IEEE, 2019 , pp. 82 - 87 .

[9] X. Zhou , S. Li , L. Cao , H. Zhang , Z. Jia , C. Zhong , Z. Shan , M. A. Babar , Revisiting the practices and pains of microservice architecture in reality: An industrial inquiry , Journal of Systems and Software 195 ( 2023 ) 111521 .

[10] D. Abadi , A. Ailamaki , D. Andersen , P. Bailis , M. Balazinska , P. A. Bernstein , P. Boncz , S. Chaudhuri , A. Cheung , A. Doan , et al., The seattle report on database research , Communications of the ACM 65 ( 2022 ) 72 - 79 .

[11] V. Bushong , A. S. Abdelfattah , A. A. Maruf , D. Das , A. Lehman , E. Jaroszewski, M. Cofey , T. Cerny , K. Frajtak , P. Tisnovsky , et al., On microservice analysis and architecture evolution: A systematic mapping study , Applied Sciences 11 ( 2021 ) 7856 .

[12] Y. Wang , D. Conan , S. Chabridon , K. Bojnourdi , J. Ma, Runtime models and evolution graphs for the version management of microservice architectures , in: 2021 28th Asia-Pacific Software Engineering Conference (APSEC) , IEEE, 2021 , pp. 536 - 541 .

[13] B. Mayer , R. Weinreich , An approach to extract the architecture of microservice-based software systems , in: 2018 IEEE symposium on serviceoriented system engineering (SOSE) , IEEE, 2018 , pp. 21 - 30 .

[14] P. Benats , M. Gobert , L. Meurice , C. Nagy , A. Cleve , An empirical study of (multi-) database models in open-source projects , in: Conceptual Modeling: 40th International Conference, ER 2021, Virtual Event , October 18-21 , 2021 , Proceedings 40, Springer, 2021 , pp. 87 - 101 .

[15] M. Gobert , L. Meurice , A. Cleve , Hydra: A framework for modeling, manipulating and evolving hybrid polystores , in: 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , 2022 , pp. 652 - 656 .

[16] M. De Jong , A. Van Deursen, Continuous deployment and schema evolution in sql databases , in: 2015 IEEE/ACM 3rd International Workshop on Release Engineering, IEEE, 2015 , pp. 16 - 19 .

[17] A. Maule , W. Emmerich , D. S. Rosenblum , Impact analysis of database schema changes , in: Proceedings of the 30th international conference on Software engineering , 2008 , pp. 451 - 460 .

[18] D. A. d'Aragona , L. Pascarella , A. Janes , V. Lenarduzzi , D. Taibi , Microservice logical coupling: A preliminary validation , in: 2023 IEEE 20th International Conference on Software Architecture Companion ( ICSA-C) , IEEE, 2023 , pp. 81 - 85 .

[19] T. Cerny , A. S. Abdelfattah , V. Bushong , A. Al Maruf , D. Taibi , Microservice architecture reconstruction and visualization techniques: A review , in: 2022 IEEE International Conference on Service-Oriented System Engineering (SOSE) , IEEE, 2022 , pp. 39 - 48 .

[20] T. Cerny , D. Taibi , Static analysis tools in the era of cloud-native systems ( 2022 ).

[21] M. E. Gortney , P. E. Harris , T. Cerny , A . Al Maruf, M. Bures , D. Taibi , P. Tisnovsky , Visualizing microservice architecture in the dynamic perspective: A systematic mapping study , IEEE Access ( 2022 ).

[22] M. Gobert , L. Meurice , A. Cleve , Conceptual modeling of hybrid polystores , in: Conceptual Modeling: 40th International Conference, ER 2021, Virtual Event , October 18-21 , 2021 , Proceedings 40, Springer International Publishing, 2021 , pp. 113 - 122 .

[23] S. Scherzinger , W. Mauerer , H. Kondylakis , Debinelle: Semantic patches for coupled database-application evolution , in: 2021 IEEE 37th International Conference on Data Engineering (ICDE) , IEEE, 2021 , pp. 2697 - 2700 .

[24] A. Alidra , H. Bruneliere , T. Ledoux , A feature-based survey of fog modeling languages, Future Generation Computer Systems ( 2022 ).

[25] G. Parker , S. Kim , A . Al Maruf, T. Cerny , K. Frajtak , P. Tisnovsky , D. Taibi , Visualizing anti-patterns in microservices at runtime: A systematic mapping study , IEEE Access ( 2023 ).

[26] L. Meurice , A. Cleve , Dahlia 2.0: A visual analyzer of database usage in dynamic and heterogeneous systems , in: 2016 IEEE Working Conference on Software Visualization (VISSOFT) , 2016 , pp. 76 - 80 .