=Paper=
{{Paper
|id=Vol-3917/paper60
|storemode=property
|title=Automating machine learning: A meta-synthesis of MLOps tools, frameworks and architectures
|pdfUrl=https://ceur-ws.org/Vol-3917/paper60.pdf
|volume=Vol-3917
|authors=Danylo O. Hanchuk,Serhiy O. Semerikov
|dblpUrl=https://dblp.org/rec/conf/cs-se-sw/HanchukS24
}}
==Automating machine learning: A meta-synthesis of MLOps tools, frameworks and architectures==
https://ceur-ws.org/Vol-3917/paper60.pdf
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Automating machine learning: A meta-synthesis of MLOps
tools, frameworks and architectures
Danylo O. Hanchuk1 , Serhiy O. Semerikov1,2,3,4,5
1
Kryvyi Rih State Pedagogical University, 54 Universytetskyi Ave., Kryvyi Rih, 50086, Ukraine
2
Institute for Digitalisation of Education of the NAES of Ukraine, 9 M. Berlynskoho Str., Kyiv, 04060, Ukraine
3
Zhytomyr Polytechnic State University, 103 Chudnivsyka Str., Zhytomyr, 10005, Ukraine
4
Kryvyi Rih National University, 11 Vitalii Matusevych Str., Kryvyi Rih, 50027, Ukraine
5
Academy of Cognitive and Natural Sciences, 54 Universytetskyi Ave., Kryvyi Rih, 50086, Ukraine
Abstract
Automating the end-to-end lifecycle of machine learning models is critical for their effective operationalization.
Various tools, frameworks and architectures have emerged to support Machine Learning Operations (MLOps)
practices. This paper presents a meta-synthesis of existing reviews to provide a comprehensive overview of such
enabling technologies for MLOps. The capabilities and features offered by common commercial and open-source
MLOps platforms are compared. Patterns in the MLOps architecture and design philosophies are identified. The
role of containers, orchestration, configuration management, and infrastructure automation in ML pipelines is
examined. Approaches for model deployment on cloud and edge are also discussed. The synthesis offers insights
for tool selection and usage to automate enterprise-scale machine learning.
Keywords
MLOps, automation, tools, frameworks, architecture, model deployment, ML pipelines, meta-synthesis
1. Introduction
In the modern world, machine learning is becoming an increasingly important technology that finds
application in various fields such as finance, healthcare, industry, retail, etc. [1] However, despite
significant progress in the development of machine learning algorithms and models, their effective
deployment in production environments remains a challenging task [2, 3]. This is due to a number of
factors, such as the need to ensure scalability, reproducibility, security, and reliability of models, as well
as the complexity of integrating development and operation processes.
To solve these problems, the MLOps (Machine Learning Operations) methodology has emerged, which
aims to apply the principles and practices of DevOps to the development and deployment processes
of machine learning models [4, 5]. MLOps covers a wide range of practices, such as automation of
machine learning pipelines, versioning of data and models, monitoring model performance, experiment
management, etc. [6, 7]. Research shows that the application of MLOps practices can significantly
increase the efficiency and reliability of deploying machine learning models in production environments
[8, 9].
At the same time, despite the significant interest in the topic of MLOps from both scientists and
practitioners, there are still certain gaps and unresolved problems in this area. In particular, there are no
generally accepted standards and best practices for implementing MLOps, issues of integrating MLOps
with other approaches (DataOps, ModelOps, AIOps, etc.) are insufficiently researched, and there is a
need to develop new tools and platforms to automate MLOps processes [5, 10, 11].
This work is aimed at solving the current problem of defining and analysing MLOps practices
necessary for the effective deployment of machine learning models. The basis for performing the work
CS&SE@SW 2024: 7th Workshop for Young Scientists in Computer Science & Software Engineering, December 27, 2024, Kryvyi
Rih, Ukraine
" danilhanchuk@gmail.com (D. O. Hanchuk); semerikov@gmail.com (S. O. Semerikov)
~ https://acnsci.org/semerikov (S. O. Semerikov)
0009-0004-6474-3521 (D. O. Hanchuk); 0000-0003-0789-0272 (S. O. Semerikov)
© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
362
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
is the need to systematize and generalize knowledge about MLOps practices, as well as the need to
develop recommendations for their implementation in organizations to increase the efficiency and
reliability of deploying machine learning models in production environments.
According to the aim, the following main objectives of the study are defined:
1. Perform a meta-synthesis of systematic reviews to generalize knowledge about MLOps practices
necessary for the effective deployment of machine learning models.
2. Analyze the relationships between MLOps principles, processes, and practices.
3. Identify the most effective MLOps practices for deploying machine learning models.
2. Meta-synthesis of MLOps practices
2.1. Main concepts of the study
DevOps (Development & Operations) is becoming increasingly widespread, and companies are ap-
plying its methods in various fields [6]. In this context, MLOps (Machine Learning & Operations)
automates ML workflows, such as pipelines, applying DevOps practices (implementing continuous
integration/continuous deployment (CI/CD) for machine learning projects) [12, 6, 8].
According to Calefato et al. [12], key MLOps practices can be realized using GitHub Actions and CML
(Continuous Machine Learning). While some workflows automate ML tasks with GitHub Actions and
CML, production-grade, end-to-end MLOps pipelines appear rare in the analysed open-source GitHub
projects. Practices focus more on reporting and metrics rather than retraining or deployment.
2.2. Research methodology
Systematic reviews can provide syntheses of the state of knowledge in a field, from which future
research priorities can be identified; they can address questions that otherwise could not be answered
by individual studies; they can identify problems in primary research that should be rectified in future
studies; and they can generate or evaluate theories about how or why phenomena occur [13, p. 1]. The
main aim of a systematic review is facilitating evidence-based decision-making [13, p. 6]. The main
difference between systematic and literature reviews is the “place of idea”. The literature review can be
idea-driven; thus, all sources can be selected to confirm some idea. Indeed, the systematic review is the
scientific method; instead of ideas, we operate with research questions and hypotheses. As a result, the
systematic review can produce new evidence-based knowledge.
On 23 February 2024, we proceeded with a search request to the Scopus database by article title:
TITLE ( ( systematic OR review OR survey ) AND mlops )
5 documents were found (table 1), of which 3 [6, 8, 5] relate to systematic reviews.
Table 1: Results of searching for existing systematic reviews in Scopus.
Bibliographic description Review content
G. Recupito, F. Pecorelli, G. Catolino, Recupito et al. [6] conducted a “multivocal” literature review – a kind of
S. Moreschini, D. D. Nucci, F. Palomba, systematic review that uses both “white” (articles, book chapters, etc.)
D. A. Tamburri, A Multivocal Literature and “grey” sources (blog posts, technical documents, videos, etc.). The
Review of MLOps Tools and Features, authors’ aim was to identify tools for creating MLOps pipelines and
in: 2022 48th Euromicro Conference analyze their main characteristics and features. The authors investigated
on Software Engineering and Advanced the functionality of 13 MLOps tools and showed that most MLOps
Applications (SEAA), 2022, pp. 84–91. tools support the same features but apply different approaches that can
doi:10.1109/SEAA56994.2022.00021. provide different advantages depending on user requirements.
Continuation on the next page
363
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Continuation of table 1
Bibliographic description Review content
A. Lima, L. Monteiro, A. P. Furtado, Lima et al. [8] conducted a systematic literature review to identify
MLOps: Practices, Maturity Models, practices, standards, roles, maturity models, challenges, and tools of
Roles, Tools, and Challenges – A System- MLOps. 30 articles were selected for analysis. The study results allowed
atic Literature Review, in: Proceedings of to conclude that MLOps is still at an initial stage.
the 24th International Conference on En-
terprise Information Systems - Volume
1: ICEIS, INSTICC, SciTePress, 2022, pp.
308–320. doi:10.5220/0010997300003179.
C. Haertel, D. Staegemann, C. Daase, M. Haertel et al. [14] provided an overview of MLOps applications in Data
Pohl, A. Nahhas, K. Turowski, MLOps Science projects. The authors showed that when considering contempo-
in Data Science Projects: A Review, in: rary MLOps approaches, the emphasis is placed on model development
2023 IEEE International Conference on and deployment, while organizational aspects (business understanding,
Big Data (BigData), 2023, pp. 2396–2404. evaluation) receive insufficient attention. Since Data Science project
doi:10.1109/BigData59044.2023.10386139. success does not exclusively depend on technical matters, the authors
propose that future research should continue to advance the MLOps
field by bridging the gap between business objectives of the organi-
zation and how these objectives are represented and modelled using
appropriate concepts.
R. Cohen, Digital Strategy, Machine As part of a digital strategy, machine learning (ML) has become a com-
Learning, and Industry Survey of MLOps, mon toolset and capability across many businesses. However, the op-
in: Digital Strategies and Organizational erational aspects of machine learning (MLOps) are often overlooked
Transformation, 2023, pp. 137–150. for ML projects until they are already installed and being executed in
URL: https://tinyurl.com/33z6zpd3. the business environment. This chapter provides a review of MLOps
doi:10.1142/9789811271984_0008. [15] products and vendors to give data scientists the ability to set up the
appropriate ML infrastructure in a proactive manner.
J. Diaz-de Arcaya, A. I. Torre-Bastida, G. Diaz-de Arcaya et al. [5] analyze the challenges, opportunities, and
Zárate, R. Miñón, A. Almeida, A Joint perspectives of implementing MLOps and AIOps. The authors ana-
Study of the Challenges, Opportunities, lyzed the open issues, opportunities, and trends faced by organizations
and Roadmap of MLOps and AIOps: A when implementing MLOps and AIOps, the frameworks and architec-
Systematic Survey, ACM Comput. Surv. tures, as well as the fields of their use. The systematic review of 93
56 (2023) 84. doi:10.1145/3625289. studies provided an opportunity to identify: 1) successful implemen-
tation of artificial intelligence projects requires a collaborative culture
and a combination of software engineering, data science, and DevOps
skills; 2) containerization, data and model versioning, FaaS (Function-
as-a-Service) and serverless architectures are useful for supporting the
MLOps/AIOps lifecycle; 3) monitoring the environment is important
for retraining and redeploying components; 4) AIOps is used predomi-
nantly in complex environments, such as 5G and 6G technologies, while
MLOps is more common in traditional industrial environments.
The papers by Haertel et al. [14] and Cohen [15] are not systematic reviews, but the results obtained in
these works were taken into account when performing the meta-synthesis [16] for combining (appendix
B) and thematic analysis of the results obtained in the systematic reviews.
The meta-synthesis was performed according to Chrastina [17, pp. 123-125]:
1. Defining the subject of research: MLOps practices for the effective deployment of models.
2. Identifying relevant sources: systematic literature reviews [6, 8, 5] and a review of MLOps products
and providers [15].
3. Thorough study to determine the common time period for the analyzed works, similarities and
differences: in the aim, research questions, sources, inclusion, exclusion, and quality criteria,
definitions of MLOps and MLOps stages.
364
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
4. Defining the relationship between works through identifying and grouping key topics.
5. Mutual translation of results of different works through defining common terminology, explaining
contradictions in the results from different works, and generalizing results from different works.
6. Synthesis of results.
7. Publication of the meta-synthesis.
2.3. Thorough study and defining the relationship between works
2.3.1. Distribution of reviews by year
Two works [6, 8] refer to 2022, one [5] – to 2023. At the same time, the sources analyzed in [6] are
limited to 2020, in [8] – to 2021, and in [5] – to 2023. In addition, the work [5] mentions the work [8] as
a previous one.
2.3.2. Review objectives
The aim of the meta-synthesis of the review objectives [6, 8, 5] was to identify common and different
aspects regarding the general focus and tasks of these studies.
Common aspects of the objectives of the considered reviews are:
1. All reviews [6, 8, 5] are aimed at researching and generalizing knowledge about the MLOps
methodology, its practices, tools, and challenges.
2. The reviews [6, 8] aim to identify and analyze MLOps tools used to automate the development
and deployment processes of machine learning models.
3. The reviews [5, 8] seek to provide an understanding of the general state of MLOps practices
implementation in industry and academia.
Distinct aspects of the review objectives are:
1. The review [6] focuses more on identifying and analyzing the functional capabilities of MLOps
tools for creating machine learning pipelines.
2. The review [8] pays attention to a wider range of MLOps aspects, such as practices, roles, maturity
models, challenges, in addition to tools.
3. The review [5], in addition to the MLOps methodology, also considers the related concept of
AIOps. More attention is paid to highlighting the opportunities, challenges, and future trends in
both areas.
Thus, despite some differences in focus and breadth of coverage, all the considered reviews are
united by a common goal – to investigate and generalize knowledge about the MLOps methodology, its
practical application, tools, challenges, and state of implementation to promote further development of
this area.
2.3.3. Review research questions
The aim of the meta-synthesis of the research questions of the reviews [6, 8, 5] was to identify com-
mon and different aspects regarding the main directions of research within the study of the MLOps
methodology.
Common aspects of the research questions of the considered reviews are:
365
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
1. All reviews [6, 8, 5] contain questions about tools and platforms used to implement MLOps
practices, automate development processes, deployment, and monitoring of machine learning
models.
2. The reviews [8, 5] include questions regarding challenges and open problems faced by organiza-
tions when implementing MLOps.
3. The reviews [8, 5] consider questions about opportunities and future trends in the field of MLOps.
Distinct aspects of the research questions of the reviews are:
1. The review [6] contains more specific questions about the functional capabilities and features of
MLOps tools for creating machine learning pipelines.
2. The review [8] includes questions regarding the roles and responsibilities of specialists involved
in MLOps implementation, as well as maturity models for assessing the level of automation of
model deployment processes.
3. The review [5], in addition to MLOps, also considers questions specific to the AIOps methodology
and pays attention to current and future areas of application of these approaches.
Thus, the research questions of the considered reviews cover a wide range of MLOps aspects, from
tools and platforms to challenges, opportunities, and areas of application. Despite some differences in
the focus of the questions, all reviews seek to explore the key components and factors that influence
the implementation and development of MLOps practices in organizations.
2.3.4. Review information sources
The aim of the meta-synthesis of the information sources of the reviews [6, 8, 5] was to identify common
and different aspects regarding the databases, search engines, and types of literature used to search for
relevant studies.
Common aspects of information sources of the considered reviews are:
1. All reviews [6, 8, 5] used electronic databases of scientific publications to search for relevant
studies.
2. The reviews [6, 5] included both academic (peer-reviewed) and non-academic (“grey”) literature
sources such as blogs, websites, videos, code repositories, etc. in the search.
Distinct aspects of information sources of the reviews are:
1. The review [6] used Google Scholar to search for scientific publications and regular Google search
for “grey” literature.
2. The review [8] limited the search to only academic databases such as ACM Digital Library, IEEE
Xplore, ScienceDirect, and SpringerLink.
3. The review [5] used several databases (Scopus, arXiv, Springer, IEEE), but the main source was
the Scopus database from Elsevier.
Thus, the considered reviews demonstrate different approaches to the selection of information
sources. Some studies [6, 5] include both academic and non-academic sources to obtain a more complete
picture of the practical application of MLOps. Others [8] focus exclusively on peer-reviewed scientific
publications. The choice of sources can influence the coverage and type of studies found, and hence the
results and conclusions of the reviews.
366
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
2.3.5. Criteria for including information sources in reviews
The aim of the meta-synthesis of the criteria for including information sources in the reviews [6, 8, 5]
was to identify common and different aspects regarding the requirements that studies must meet for
inclusion in the analysis.
Common aspects of the inclusion criteria in the considered reviews are:
1. All reviews [6, 8, 5] included studies that directly relate to the topic of MLOps, its practices, tools,
and application.
2. The reviews [6, 8] considered studies that describe the experience, practices, architecture, or
implementation of MLOps tools and processes.
Distinct aspects of the inclusion criteria in the reviews are:
1. The review [6] included studies that describe the components of the minimal MLOps lifecycle or
present the experience and opinions of experts regarding MLOps.
2. The review [8] additionally included studies that assess the maturity of MLOps processes, consider
roles and responsibilities in the ML model development lifecycle, and identify challenges in
developing and implementing MLOps solutions.
3. The review [5] had more general inclusion criteria, considering studies published from 2018 to
2023 that contain new ideas and are closely related to the topic of MLOps and AIOps.
Thus, despite some differences, the inclusion criteria in the considered reviews mainly focus on
studies that directly relate to MLOps, describe practical experience, tools, and processes, and consider
various aspects of the ML model development lifecycle. The reviews [8, 5] have somewhat broader
criteria, also including studies related to maturity assessment, roles, and MLOps challenges.
2.3.6. Criteria for excluding information sources from reviews
The aim of the meta-synthesis of the criteria for excluding information sources from the reviews [6, 8, 5]
was to identify common and different aspects regarding the characteristics of studies that lead to their
exclusion from the analysis.
Common aspects of the exclusion criteria in the considered reviews are:
1. The reviews [6, 8] excluded studies that do not provide sufficient details about the architecture,
implementation, or application of MLOps tools and processes.
2. The reviews [8, 5] excluded studies published in languages other than English.
Distinct aspects of the exclusion criteria for information sources from the reviews are:
1. The review [6] excluded studies that promote commercial MLOps platforms without providing
details on their implementation or use.
2. The review [8] excluded studies that relate only to the application of ML models without con-
sidering MLOps aspects, as well as short articles, posters, and studies without access to the full
text.
3. The review [5] excluded studies with an insufficient number of citations (depending on the year
of publication), as well as materials with limited access (by subscription) and articles published in
insufficiently reliable sources.
Thus, the considered reviews apply different exclusion criteria to filter out studies that do not meet
their requirements. The common aspect is the exclusion of studies with insufficient descriptions of
MLOps processes and tools, as well as non-English publications. Differences lie in additional criteria,
such as the exclusion of commercial platforms without technical details [6], short articles and posters
[8], and materials with limited access and a low number of citations [5].
367
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
2.3.7. Quality criteria for information sources in reviews
The aim of the meta-synthesis of the quality criteria for the reviews [6, 8, 5] was to identify common
and different aspects regarding the requirements for the quality and reliability of studies included in
the analysis.
Common aspects of the quality criteria in the considered reviews are:
1. The reviews [8, 5] evaluated the quality of studies based on the completeness of the description
of the methodology, context, and results.
2. The reviews [8, 5] considered the presence of substantiated evidence and arguments to support
the study conclusions.
Distinct aspects of the quality criteria in the reviews are:
1. The review [6] used quantitative indicators of popularity (number of stars on Github, views on
YouTube) to assess the quality and relevance of “grey” literature.
2. The review [8] evaluated whether the study presents empirical results, not just expert opinions,
and whether the results are properly validated.
3. The review [5] used an extended set of criteria, including the presence of a comprehensive
literature review, verification of results on use cases, the number of research questions addressed,
the availability of open access, publication in high-impact journals, and the number of citations.
Thus, the considered reviews apply different approaches to assessing the quality of studies. The
common aspect is the desire to include studies with a complete description of the methodology and
substantiated results. However, the specific quality criteria differ: from the use of popularity indicators
for “grey” literature [6], to the assessment of the empirical nature of the results [8] and consideration of
bibliometric indicators such as journal impact factor and number of citations [5].
2.4. Mutual translation of results from different works and synthesis of results
For the mutual translation of the results from different works, in addition to systematic reviews [6, 8, 5],
a review of MLOps products and providers [15] was involved.
2.4.1. Definition of MLOps
The aim of the meta-synthesis of MLOps definitions in the reviews [6, 8, 5, 15] was to identify common
and different aspects in the understanding and interpretation of this concept.
Common aspects of MLOps definitions in the considered reviews are:
1. All reviews [6, 8, 5, 15] consider MLOps as a set of practices, principles, and processes for
automating and managing the lifecycle of machine learning models.
2. The reviews [6, 5] emphasize the use of approaches and practices from DevOps in MLOps, such
as continuous integration, delivery, and monitoring.
3. The reviews [8, 15] emphasize the role of MLOps in operationalizing machine learning solutions
and transferring them to industrial operation.
Distinct aspects of MLOps definitions in the reviews are:
1. The review [6] focuses more on the technical aspects of MLOps, such as model lifecycle manage-
ment, pipeline automation, and performance monitoring.
2. The review [8] considers MLOps as a set of practices specifically for operationalizing data science
solutions.
368
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
3. The review [5] emphasizes the use of software engineering and machine learning principles in
MLOps to create model-based products.
4. The review [15] considers MLOps as a separate area that focuses on automating the machine
learning model lifecycle as part of companies’ digital strategies.
Thus, despite some differences in emphasis and wording, all the considered reviews define MLOps
as an approach for managing, automating, and operationalizing the processes of developing,
deploying, and supporting machine learning models based on practices from software en-
gineering and DevOps. MLOps is a key component for the successful implementation of machine
learning solutions in an industrial environment.
2.4.2. Stages of the MLOps workflow
The aim of the meta-synthesis of the stages of the MLOps workflow in the reviews [6, 8, 5, 15] was
to identify the most common steps in the lifecycle of developing and implementing machine learning
models.
Common stages of the MLOps workflow in the considered reviews are:
1. All reviews [6, 8, 5, 15] include the stages of data collection and processing, model development
and training, and model deployment in the working environment.
2. The reviews [6, 5, 15] highlight the stage of monitoring the performance and degradation of
deployed models as an important part of the MLOps workflow.
3. The reviews [6, 15] include the stage of retraining models based on new data or on a schedule as
part of the MLOps lifecycle.
Distinct aspects of the MLOps workflow stages in the reviews are:
1. The review [6] provides a detailed breakdown of the workflow stages, including the steps of data
extraction, analysis, cleaning, and transformation, as well as model validation.
2. The review [8] focuses less on detailed stages and more on general MLOps functions, such as
data collection, transformation, model training, and implementation.
3. The review [5] groups stages into broader categories, such as data management, distributed
training, deployment, and monitoring.
4. The review [15] additionally highlights the stages of generating predictions and managing models
and data as part of the MLOps workflow.
Thus, despite different levels of detail and grouping, the considered reviews demonstrate general
consistency regarding the main stages of the MLOps workflow. These stages cover the entire lifecycle
of machine learning models, from data collection and processing to deployment, monitoring, and
retraining of models. Differences in the presentation of stages reflect different approaches to structuring
and describing the MLOps workflow.
2.4.3. Frameworks and architectures that facilitate MLOps implementation
The aim of the meta-synthesis of frameworks and architectures that facilitate MLOps implementation
in the reviews [6, 8, 5, 15] was to identify the most common and effective approaches and technologies
in this area.
Common frameworks and architectures that facilitate MLOps implementation, according to the consid-
ered reviews, are:
369
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
1. The reviews [6, 8, 5] highlight open-source platforms and frameworks such as MLflow, Kubeflow,
and TensorFlow Extended (TFX) as key components of the MLOps ecosystem.
2. The reviews [6, 5, 15] emphasize the importance of using cloud computing platforms and services
such as AWS, Google Cloud, and Azure to deploy and scale MLOps solutions.
3. The reviews [8, 5] note that architectures based on containerization (e.g., using Docker) and
container orchestration (e.g., using Kubernetes) are key to ensuring portability and scalability of
MLOps solutions.
Distinct aspects of the considered MLOps frameworks and architectures in the reviews are:
1. The review [6] additionally highlights MLOps pipeline orchestration platforms such as Apache
Airflow, Jenkins, and Polyaxon.
2. The review [8] additionally mentions tools such as Kubeflow, Polyaxon, Comet.ml, Kafka-ML,
MLModelCI for managing pipelines and deploying models.
3. The review [5] considers a broader range of architectural approaches, including the use of edge
computing, serverless computing, and event-driven architectures.
4. The review [15] focuses primarily on proprietary platforms and solutions from commercial
providers such as Iguazio, Domino Data Lab, Comet, and Valohai.
Thus, there are many frameworks and architectural approaches that facilitate MLOps implementation,
from open platforms and libraries to commercial solutions and cloud services. Key factors are support
for automation, scalability, portability, and integration with existing systems and tools. The choice of
appropriate frameworks and architectures depends on the specific requirements and constraints of the
organization, as well as the level of maturity of its MLOps processes.
2.4.4. MLOps tools for creating machine learning pipelines and operationalizing models
The aim of the meta-synthesis of MLOps tools for creating machine learning pipelines and operational-
izing models in the reviews [6, 8, 5, 15] was to identify the most popular and functional tools in this
area.
Common MLOps tools mentioned in the considered reviews are:
1. The reviews [6, 8] highlight MLflow as a popular open-source platform for managing the lifecycle
of machine learning models, experiments, and deployment.
2. The reviews [6, 15] mention cloud platforms from major providers such as AWS SageMaker,
Google Cloud AI Platform, Azure Machine Learning, as tools for operationalizing models.
3. The reviews [8, 5] note that containerization tools such as Docker and orchestration tools such
as Kubernetes are often used to deploy models.
Distinct aspects of the considered MLOps tools in the reviews are:
1. The review [6] provides a detailed list of tools for different stages of the MLOps pipeline, including
orchestration platforms (Apache Airflow, Jenkins, Kubeflow, Polyaxon, Seldon Core, etc.) and
deployment (TensorFlow Extended).
2. The review [8] additionally mentions tools such as Kubeflow, Polyaxon, Comet.ml, Kafka-ML,
MLModelCI for managing pipelines and deploying models.
3. The review [5] focuses more on general categories of tools, such as experiment management
systems, data and model versioning, and infrastructure automation.
370
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Table 2
Popular MLOps platforms and products, and associated providers (based on [15, p. 141]).
Platform/product Provider URL
MLflow MLflow https://mlflow.org/
Google Cloud AI Google https://cloud.google.com/products/ai
Kaggle Kaggle https://www.kaggle.com/
SageMaker Amazon https://aws.amazon.com/sagemaker/
Cloud-Native Toolkit IBM https://develop.cloudnativetoolkit.dev/
resources/workshop/ai/
Iguazio MLOps Platform Iguazio https://www.iguazio.com/
Azure Machine Learning Microsoft https://azure.microsoft.com/en-us/
products/machine-learning
Huawei Cloud ModelArts Huawei https://www.huaweicloud.com/intl/en-us/
product/modelarts.html
SparkCognition Generative AI Suite SparkCognition https://www.sparkcognition.com/products/
sparkcognition-generative-ai-suite
Comet Comet https://www.comet.com/site/
Grid.AI Grid.AI https://www.grid.ai/
Modzy ModelOps Platform Modzy https://github.com/modzy
Valohai MLOps Platform Valohai https://valohai.com/
HPE Ezmeral ML Ops Hewlett Packard Enterprise https://www.hpe.com/us/en/software/
ezmeral-ml-ops.html
Domino Enterprise MLOps Platform Domino https://domino.ai/
4. The review [15] details the functionality of popular commercial MLOps platforms such as Iguazio,
Domino Data Lab, Comet, Valohai, etc. (table 2)
Thus, there is a wide range of MLOps tools for creating machine learning pipelines and operational-
izing models, from open platforms such as MLflow, to commercial solutions from cloud providers and
specialized companies. The choice of specific tools depends on the needs and scale of the organization,
as well as compatibility with the existing technology stack.
2.4.5. Main features offered by MLOps tools
The aim of the meta-synthesis of the main features offered by MLOps tools in the reviews [6, 8, 5, 15]
was to identify the key capabilities and components of these tools.
Common features of MLOps tools, highlighted in the considered reviews, are:
1. The reviews [6, 8, 5] note that MLOps tools usually provide capabilities for tracking experiments,
versioning models and data.
2. The reviews [6, 8, 15] emphasize the importance of automation and orchestration features of
MLOps workflows, such as model training and deployment pipelines.
3. The reviews [6, 5, 15] indicate the presence in MLOps tools of components for monitoring the
performance and degradation of deployed models.
Distinct aspects of MLOps tool features, considered in the reviews, are:
1. The review [6] provides a detailed classification of features into three categories:
a) general features related to all stages of the MLOps pipeline:
• open source support;
• scalability and elasticity;
• extensibility;
371
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
• cloud-agnostic or cloud environment support;
• metadata management;
• continuous integration and delivery (CI/CD);
• user interfaces: graphical (GUI), command line (CLI), application programming interface
(API);
b) data management features:
• real-time data streaming;
• data storage;
• data analysis, cleaning, and transformation;
• data monitoring;
• metadata management;
• providing data access via API;
c) model management features:
• support for various machine learning libraries and frameworks;
• experiment tracking and model versioning;
• model registry;
• automatic hyperparameter optimization;
• model testing (A/B testing);
• anomaly and model drift detection;
• model performance monitoring;
• model metadata management;
• model deployment via API.
2. The review [8] additionally highlights features such as automatic hyperparameter optimization
of models and mobility support for deployment in different environments.
3. The review [5] notes the importance of integrating MLOps tools with existing systems and
supporting collaborative work of teams.
4. The review [15] details the features of commercial MLOps platforms:
• model development: environment for data analysis, feature development, training, and model
experiments;
• operationalization of model training: creating reproducible pipelines for model training and
testing;
• continuous model training: automatic support for the frequency of model retraining based
on schedule, events, or ad-hoc requests;
• model deployment: packaging, testing, and deploying trained models in the production
environment;
• generating predictions: providing predictions or classifications in real-time or batch process-
ing mode;
• monitoring model performance: tracking the efficiency and degradation of models, warning
about the need for retraining;
• data and feature management: support for storing, processing, and accessing data and
generated features.
372
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Thus, MLOps tools provide a wide range of features to support the lifecycle of machine learning mod-
els, with a focus on automation, experiment tracking, versioning, monitoring, and model deployment.
Some tools offer more specialized features, such as hyperparameter optimization or data management.
The choice of a tool with an appropriate set of features depends on the specific needs and goals of the
organization in the field of MLOps.
2.4.6. Ways of deploying machine learning models in production environments
The aim of the meta-synthesis of ways of deploying machine learning models in production envi-
ronments in the reviews [6, 8, 5, 15] was to identify the most common and critical practices in this
area.
Common ways of deploying machine learning models in production environments, according to the
considered reviews, are:
1. The reviews [6, 5, 15] note that models are often deployed using container technologies such as
Docker, which provides model mobility and isolation.
2. The reviews [6, 8, 15] indicate the prevalence of deploying models in cloud environments using
platforms and services from major providers such as AWS, Google Cloud, and Azure.
3. The reviews [6, 5] note that models are often deployed as web services using REST API or other
protocols to provide access to predictions in real-time.
Distinct aspects of the considered ways of deploying models in the reviews are:
1. The review [6] additionally describes deploying models using orchestration platforms (Apache
Airflow, Jenkins, Kubeflow, MLflow, Polyaxon, Seldon Core, Valohai) to provide automatic scaling
and container management.
2. The review [8] notes that some MLOps tools, such as MLflow, Kubeflow, and Kafka-ML, have
built-in capabilities to facilitate model deployment in different environments.
3. The review [5] considers deploying models not only in the cloud but also on edge devices using
specialized frameworks such as TensorFlow Lite and Core ML.
4. The review [15] provides a detailed description of the main stages and features of the machine
learning model deployment process using CI/CD pipelines and support for different environments
using commercial MLOps platforms:
a) creating a CI/CD pipeline for models: MLOps platforms such as SageMaker, Azure ML, and
Databricks allow creating CI/CD pipelines to automate the model deployment process,
which includes the stages of building, testing, and deploying models, as well as tracking
artifacts and version management;
b) supporting different deployment environments: MLOps platforms typically support multiple
deployment environments, such as development, testing, and production environments;
models can be deployed in different environments using appropriate configurations and
access policies;
c) model deployment process:
• trained models are packaged in a standardized format (e.g., Docker container) along
with necessary dependencies;
• the model goes through testing and validation stages to ensure its correctness and
compliance with requirements;
• after successfully passing the tests, the model is deployed in the target environment
(when deploying in the production environment, additional security and monitoring
measures may be applied);
373
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
d) automation and orchestration of deployment: MLOps platforms use automation tools such
as Jenkins or GitLab CI/CD to ensure continuous integration and deployment of models,
which can be configured to automatically trigger on certain events, such as updating the
model code or the appearance of new data;
e) monitoring and management of deployed models: MLOps platforms provide tools for moni-
toring the performance and metrics of deployed models in real-time: in case of problems
or model degradation, the platform can automatically initiate the process of retraining or
rolling back to the previous version of the model.
Thus, the most common ways to deploy machine learning models in production environments are
the use of container technologies, cloud platforms and services, and the deployment of models as web
services. The choice of a specific approach depends on the requirements for latency, scalability, and
availability of models, as well as the existing infrastructure and ecosystem of tools in the organization.
2.4.7. Maturity models for assessing the level of automation in deploying machine learning
models
Lima et al. [8] refer to several maturity models for assessing the level of improvement in the development
process of machine learning solutions:
1. The maturity model proposed by Amershi et al. [18], mentioned simultaneously in [8] and [5].
This model, based on the Capability Maturity Model (CMM) and Six Sigma methodology, checks
whether the activity: (1) has defined goals, (2) is consistently implemented, (3) is documented, (4)
is automated, (5) is measured and tracked, and (6) is continuously improved.
2. According to Dhanorkar et al. [19], organizations can be classified into three levels of maturity for
developing machine learning solutions: (1) data-oriented, (2) model-oriented, (3) pipeline-oriented.
3. Lwakatare et al. [20] describe five stages of improvement in development practices: (1) manual
process led by data science, (2) standardized process of experimental-operational symmetry,
(3) automated ML workflow process, (4) integrated software development and ML workflow
processes, and (5) automated and fully integrated CD and ML workflow process.
4. Akkiraju et al. [21] proposed an adaptation of the CMM model with the definition of five levels
of maturity for each assessed capability: (1) initial, (2) repeatable, (3) defined, (4) managed, and
(5) optimizing.
All systematic reviews [6, 8, 5] indicate that the level of automation of MLOps processes is one
of the key factors in assessing the maturity of an organization in this area. Despite the fact that the
considered reviews do not provide an exhaustive description of MLOps maturity models, they emphasize
the importance of assessing the level of automation of model development, testing, and deployment
processes as a key factor in the maturity of an organization in this area. The adaptation of existing
software development maturity models to the specifics of MLOps can be an effective approach to
assessing and improving machine learning processes in an organization.
2.4.8. Roles and responsibilities identified in the activities of operationalizing machine
learning models
The aim of the meta-synthesis of roles and responsibilities identified in the activities of operationalizing
machine learning models in the reviews [6, 8, 5, 15] was to identify the key participants in the MLOps
process and their functions.
Common roles and responsibilities identified in the considered reviews:
1. All reviews [6, 8, 5, 15] mention the involvement of data scientists / data science researchers who
are responsible for developing, training, and experimenting with machine learning models.
374
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
2. The reviews [6, 8, 5] highlight the role of data engineers / data providers who are involved in
extracting, processing, transforming, and ensuring the quality of data for model training.
3. The reviews [6, 5, 15] note the importance of DevOps engineers, ML/MLOps engineers, and soft-
ware engineers in operationalizing models, automating deployment processes, creating pipelines,
and managing environments.
4. The reviews [6, 5] emphasize the role of managers, leadership, and business stakeholders in
defining model requirements for deployment, decision-making, and supporting the MLOps
strategy.
5. The reviews [6, 5] emphasize the role of managers, leadership, and business stakeholders in
defining model requirements, decision-making, and supporting the MLOps strategy.
Distinct aspects of roles and responsibilities, considered in the reviews:
1. The review [8] additionally highlights the roles:
• domain specialist has deep knowledge of the subject area, plays an important role in obtaining
data and validating results);
• computational scientist/engineer has high technical skills to prepare the environment for the
operation of machine learning models;
• ML scientist/engineer is responsible for designing new machine learning models, has in-depth
knowledge of statistics and ML algorithms;
• provenance specialist manages the supply of data in the lifecycle of developing machine
learning solutions, has knowledge of both the subject area and machine learning;
• manager assesses models before their publication;
• application developer develops applications in which the created models will operate;
• deployment lead assesses aspects related to infrastructure components when deploying ML
models to production.
2. The review [5] mentions the role of subject matter experts in labeling data in specific domains.
Thus, despite some differences in the detail of roles, the considered reviews recognize the need to
involve specialists from different areas – software development, data engineering, machine learning,
subject matter experts, and management, for the successful operationalization of machine learning
models. Close collaboration and communication between these roles is critical for implementing MLOps
practices in organizations (figure 1).
2.4.9. Challenges encountered in deploying machine learning models in production
environments
The aim of the meta-synthesis of challenges encountered in deploying machine learning models in
production environments in the reviews [6, 8, 5, 15] was to identify the most common and critical
problems in this area. In [6] and [15], specific challenges are not explicitly listed, but they can be
determined indirectly based on the discussion of MLOps and automation of machine learning pipelines
in [6] and the description of the various stages of MLOps and the need for appropriate tools in [15].
Common challenges identified in the considered reviews:
1. The reviews [6, 8, 5, 15] note the complexity of managing the machine learning model lifecycle,
including versioning, tracking, and reproducibility of models and data, as well as the problem of
ensuring scalability and performance of models in real-world conditions with large amounts of
data and requests.
375
production, and monitors both the model and the ML infrastructure is a
[14,17,26,29] [α, β, γ, δ, ε, ζ, η, θ]. data
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
was
feat
DS BE
Aft
Data Scientist Backend Engineer ana
(ML model development)
ML
(ML infrastructure management)
pipe
ML Engineer / the
MLOps Engineer clea
(cross-functional management
of ML environment and assets: scie
ML infrastructure,
ML models, eng
ML workflow pipelines,
data Ingestion, adv
DE
monitoring)
DO rule
Data Engineer DevOps Engineer bas
(data management, (Software engineer with DevOps skills,
data pipeline management) ML workflow pipeline orchestration, eng
CI/CD pipeline management,
monitoring) mod
{…}
SE
Software Engineer requ
(applies design patterns and
coding guidelines) data
buil
init
Figure 3. Roles
Figure 1: Intersection and
of roles andtheir intersections
responsibilities contributing
(according to [2, p. 5]). to the
the
MLOps paradigm
2. The reviews [8, 5, 15] point to challenges associated with monitoring and maintaining models in eng
the production environment, including detecting data drift and model performance degradation. mod
3. The reviews [6, 8] consider challenges related to integrating software development with the the
5 Architecture
machine learning pipeline.and Workflow
and
On 4.the
Thebasis
reviewsof[6,the identified
5] consider challengesprinciples, components,
related to ensuring data securityand roles,when
and privacy we of
deploying machine learning models.
derive a generalized MLOps end-to-end architecture to give ML und
5. The reviews [5, 15] indicate that the quality, availability, preparation, labeling, and integration of
researchers and practitioners
data from different sources is a significantproper guidance.
challenge that requires a lot It is and
of time depicted in
resources, and feat
highlight the problem of interpreting and explaining the results of model operation to end-users
Figure 4. Additionally, we depict the workflows, i.e., the sequence
and business stakeholders. (for
in which the different tasks are executed in the different stages. The
Distinct challenges, considered in the reviews: clou
artifact was designed to be technology-agnostic. Therefore, ML
1. The review [6] emphasizes the need to automate all stages of the MLOps pipeline and integration (10
researchers
with existing and
software practitioners
development systems and can choose the best-fitting
processes.
clea
technologies and
2. The review [8] notesframeworks for their
the problem of selecting needs.
and managing infrastructure for deploying models,
requ
including the choice between cloud and on-premises environments.
As depicted in Figure 4, we illustrate an end-to-end process, mai
from MLOps project initiation to the model serving. It includes (A) tran
the MLOps project initiation steps; 376 (B) the feature engineering feed
pipeline, including the data ingestion to the feature store; (C) the
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
3. The review [5] notes the problems: a) the gap between software engineering and machine
learning skills – data scientists often do not understand the requirements of certain production
environments, and software developers do not have sufficient machine learning skills; b) effective
distribution, parallelization, and orchestration of data and ML tasks; c) the diversity of computing
infrastructure.
The considered reviews show that deploying machine learning models in production environments
is associated with a number of challenges, such as managing the model lifecycle, ensuring scalability
and performance, monitoring and maintaining models in real-world conditions. Addressing these
challenges requires an integrated approach that includes automation of MLOps processes, selection
of appropriate infrastructure, ensuring data security and privacy, and effective communication with
business stakeholders.
2.4.10. Open issues, challenges, and peculiarities of MLOps
The aim of the meta-synthesis of open issues, challenges, and peculiarities of MLOps in the reviews
[6, 8, 5, 15] was to identify the most relevant and promising areas of research and development in this
field. In [6] and [15], open problems, challenges, and peculiarities of MLOps are not directly discussed,
but they can be identified based on the analysis of MLOps tools and their capabilities in [6] and the
description of the components and functions of MLOps platforms [15].
Common open issues and challenges of MLOps, identified in the considered reviews:
1. The reviews [6, 8, 5, 15] indicate the need to develop methods and tools to ensure the inter-
pretability, reproducibility, and responsible use of machine learning models in the context of
MLOps.
2. The reviews [6, 5, 15] emphasize the importance of developing approaches to data management
in MLOps, including ensuring data quality, privacy, and security.
3. The reviews [6, 8] note the need to develop and implement MLOps standards and best practices
to ensure consistency and compatibility between different tools and platforms.
4. The reviews [8, 5] emphasize the importance of the human factor in MLOps, including the need
to ensure effective communication and collaboration between different roles and teams and the
training of qualified personnel with cross-functional skills in programming, data processing, and
operational activities.
Peculiarities of MLOps, identified in the considered reviews:
1. The review [6] notes that MLOps should take into account the specifics of the machine learning
model development process, which differs from traditional software development.
2. The review [15] considers MLOps in the context of the overall digital strategy of the organization
and emphasizes the need to align MLOps practices with business goals and needs.
Thus, the considered reviews identify a number of open issues and challenges in MLOps, such as the
need to develop standards and best practices, ensure interpretability and responsible use of models,
and effectively manage data. Peculiarities of MLOps, such as the difference from traditional software
development, the importance of the human factor, and the need to integrate knowledge from different
fields, require consideration when implementing MLOps practices in organizations.
2.4.11. Opportunities, future trends, and areas of application of MLOps
The aim of the meta-synthesis of opportunities, future trends, and areas of application of MLOps in
the reviews [6, 8, 5] was to identify promising directions of development and potential areas where
MLOps practices can bring significant benefits. In [6], they are not directly discussed, but they can be
377
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
determined indirectly based on the presented MLOps tools and their capabilities, it is possible to outline
some potential directions and trends.
Opportunities and future trends of MLOps, identified in the considered reviews:
1. The reviews [6, 8, 5] note the potential for developing standardized MLOps platforms and tools
that will simplify and accelerate the implementation of machine learning models in production.
2. The reviews [8, 5] note the prospects for integrating MLOps with other approaches, such as
DataOps, ModelOps, and DevSecOps, to provide comprehensive management of the machine
learning model lifecycle.
3. The review [8] points to: a) significant opportunities for further academic research and devel-
opment due to the fact that MLOps is still at an early stage; b) the expectation of increasing
demand for MLOps tools and platforms with the spread of artificial intelligence solutions; c) the
emergence of new roles and competencies related to MLOps as the industry develops.
4. The review [5] points to: a) opportunities to apply MLOps practices in the context of distributed
and federated model training, which will allow efficient use of decentralized data; b) involving
business units and training leadership in MLOps principles; c) using hardware platforms such as
FPGA and IoT to improve performance and privacy.
Current and future areas of MLOps application, identified in the considered reviews:
1. The reviews [6, 8, 5] note that MLOps is already actively used in industries such as finance,
healthcare, commerce, marketing, and manufacturing, where machine learning models are used
to solve real business problems.
2. The review [5] points to the potential of applying MLOps in the field of IoT and edge computing,
where machine learning models can be deployed on resource-constrained devices, 5G and 6G
technologies, educational and scientific activities.
3. The review [8] notes the prospects for using MLOps in transportation and logistics.
Thus, the considered reviews outline a number of opportunities and development trends for MLOps,
such as creating standardized platforms, applying in the context of distributed learning, and integrating
with other approaches to managing the lifecycle of data and models. Current and future areas of MLOps
application include a wide range of industries, from finance and healthcare to IoT and natural language
processing, which indicates a significant potential impact of this approach.
3. Analysis of MLOps practices
3.1. Relationship between MLOps principles, processes, and practices
MLOps is based on a set of principles [2, p. 3] and processes that ensure effective development, deploy-
ment, and support of machine learning models (MLOps practices).
MLOps principles define the fundamental foundations of designing machine learning pipelines:
• automation: maximum automation of all stages of the machine learning model lifecycle to reduce
manual interventions and improve efficiency;
• reproducibility: ensuring the ability to reproduce the results of experiments and model deployment
processes;
• collaboration: establishing effective collaboration and communication between different teams
involved in model development and implementation;
378
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
• continuous learning and improvement: regular updating of models based on new data and feedback,
continuous improvement of MLOps processes;
• data governance: ensuring data quality, security, and confidentiality throughout the model lifecy-
cle.
MLOps processes define the sequence of actions for designing and implementing machine learning
pipelines:
1. Defining business goals and requirements: aligning the goals of developing machine learning
models with the business strategy of the organization.
2. Data collection and preparation: collecting, cleaning, transforming, and enriching data for model
training.
3. Model development and training: selecting algorithms, developing model architecture, training
and validating models.
4. Model evaluation and testing: evaluating model performance on test data, conducting tests for
reliability, security, and compliance with requirements.
5. Model deployment: packaging models with necessary dependencies, deploying in target environ-
ments.
6. Model monitoring and maintenance: tracking model performance, identifying and resolving issues,
updating models as needed.
7. Model lifecycle management: coordinating all stages of model development, deployment, and
support, ensuring compliance with regulatory requirements.
MLOps practices define the most effective methods and technologies for implementing machine
learning pipelines:
• basic MLOps practices include:
– continuous integration and delivery (CI/CD): automation of the processes of building, testing,
and deploying machine learning models;
– model and data versioning: tracking changes in models and datasets, ensuring reproducibility
of results;
– ML pipeline automation: creating automated workflows for data collection, processing,
model training, and evaluation;
– model performance monitoring: tracking model quality metrics in the production environ-
ment, detecting performance degradation;
– experiment management: organizing, tracking, and comparing different experiments with
models and hyperparameters;
– model deployment: packaging models with necessary dependencies, deploying in different
environments (cloud, edge, etc.);
– model lifecycle management: coordinating the processes of model development, testing,
deployment, and monitoring to best ensure compliance with requirements;
• additional MLOps practices include:
– data security and privacy: ensuring the protection of data used for model training, compliance
with regulatory requirements;
379
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
– model explainability and interpretability: using methods and tools to understand and explain
model behavior, especially in regulated industries;
– data quality management: monitoring and ensuring the quality of data used for model
training and evaluation, detecting and handling anomalies;
– configuration management: versioning and managing configurations of environments where
models are deployed, ensuring consistency across different environments;
– model deployment strategies: selecting and implementing appropriate deployment strategies;
– infrastructure automation: using Infrastructure as Code to automate provisioning and man-
agement of infrastructure for model training and deployment;
– collaboration and communication: establishing effective collaboration between data science,
development, operations, and business units;
– risk management and compliance: identifying and mitigating risks associated with the use
of machine learning models, ensuring compliance with regulatory requirements.
Figure 2 illustrates the relationships between key MLOps principles (light blue rectangles), main
processes (green rectangles), and common practices (orange rectangles). Arrows show how principles
influence processes, and processes, in turn, are implemented through specific practices. For example,
the principle of automation influences all MLOps processes, from goal definition to model lifecycle
management. The model development process is associated with practices such as versioning, pipeline
automation, experiment management, and model interpretability.
3.2. CI/CD
CI/CD (Continuous Integration/Continuous Delivery) is a key element/practice/implementation of
DevOps for automatic testing and deployment of code, data and models in a production environment
(figure 3) [7, p. 7]. In MLOps, it is extended to automate the process of developing and deploying ML
models, including the stages of building, testing, delivery, and deployment [2, pp. 3-4].
The CI/CD process in MLOps includes the stages of build, test, delivery, and deploy [2, p. 4]. However,
unlike traditional CI/CD, MLOps may also have additional stages, such as model retraining.
CI/CD in MLOps is part of the MLOps system architecture and provides fast feedback to developers
on the success or failure of certain stages, increasing overall productivity [2, pp. 3-4].
Typical triggers for starting the CI/CD process in MLOps on GitHub are git push and pull_request
events [12, p. 4]. The events issue_comment, release, and schedule (on a schedule) can also be used.
In the article Steidl et al. [7], potential main triggers were also investigated, such as feedback systems
and alerts, scheduled orchestration service, traditional repository updates, and manual triggers. These
triggers start the execution of the pipeline, which consists of four stages: (1) data processing, (2) model
training, (3) software development, and (4) system commissioning. The data processing stage consists of a
repetitive end-to-end lifecycle of data-related tasks, such as preprocessing, quality assurance, versioning,
and documentation. The model training stage uses the results of data processing and illustrates model
development tasks such as model design, training, quality assurance, collecting metadata for model
improvement, version management, and documentation. After the pipeline completes model training,
the software development stage prepares the model for deployment through packaging, quality assurance
at the software level, and system versioning. In the final system commissioning stage, the model is
deployed in a specific environment using different deployment strategies and the system is monitored
[7, p. 21].
A feature of the CI/CD process in MLOps is the need to version not only code, but also data and
models. This allows for reproducibility and the ability to roll back to previous versions [2, p. 4].
Among the popular tools for implementing CI/CD in MLOps are Jenkins [2, p. 3] and GitHub Actions
[2, 12] and tools from cloud providers such as AWS CodePipeline, Azure DevOps Pipelines, etc.
380
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Defining business
Automation CI/CD
goals and requirements
Model and data versioning
ML pipeline automation
Data collection
and preparation
Model performance monitoring
Reproducibility
Experiment management
Model develop-
ment and training
Model deployment
Model lifecycle management
Collaboration Model evalua-
Data security and privacy
tion and testing
Model explainability and interpretability
Data quality management
Model deployment
Configuration management
Continuous learning
Model deployment strategies
Model monitoring
and maintenance
Infrastructure automation
Collaboration and communication
Data gGovernance Model lifecy-
Risk management and compliance
cle management
Figure 2: Diagram of relationships between MLOps principles, processes, and practices.
Figure 4 from the article by Kreuzberger et al. [2] presents an end-to-end MLOps architecture and
workflow with functional components and roles involved at each stage. Let’s consider in more detail
each of the zones and stages depicted in the figure:
1. A. MLOps Project Initiation. At this stage, the business stakeholder (BS) analyzes the problem
and defines the goal. The data scientist (DS) formulates the ML problem based on the business
goal. The necessary data is also determined and an initial analysis is performed by the data
engineer (DE) and data scientist (DS).
2. Data Engineering Zone. This zone includes two sub-stages:
• B1. Requirements for feature engineering pipeline. Rules for transformation, cleaning,
and calculation of new features are defined.
• B2. Feature Engineering Pipeline. A data processing pipeline is implemented that
receives data from various sources, applies transformations, and loads it into a feature store
system.
3. C. Experimentation. At this stage, the data scientist (DS) performs data analysis, preparation,
and validation, as well as model training and validation. The best model is saved in the Model
Registry.
381
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Figure 3: Continuous lifecycle pipeline for AI applications [7, p. 10].
382
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
A MLOps Project Initiation
BS SA DS DE AND DS DE AND DS
Connect to raw data
derive ML problem Understand required
Business problem designs architecture for initial data analysis
from business goal data to solve problem
analysis and technologies to (distribution analysis,
(e.g., classification, (data available?, where
(define goal) be used data quality checks,
regression) is it?, how to get it?)
validation checks)
MLOps Project Initiation Zone
B1 Requirements for
DE DS AND DE Feedback Loop
feature engineering feature requirements (iterative)
pipeline define
define feature
transformation
engineering rules
& cleaning rules
Data Sources
data pipeline code
transformation feature
streaming data
rules engineering rules
B2 Feature Engineering Pipeline {…} orchestration
DE AND SE
component
batch data data feature engineering data Ingestion
Connect to data
transformation (e.g., calc. of new job (batch or artifact store
raw data extraction
& cleaning features) streaming)
cloud storage
data preprocessing
(labeled data) data processing computation infrastructure CI/CD component
Data Engineering Zone
C Experimentation model engineering (best algorithm selection, hyperparameter tuning) model
{…}
DS SE
training code DO OR ML
ML workflow
data ML
model model export pipeline code
data analysis preparation & model
training validation model model
validation
Repository serving code {…}
versioned SE
feature data
ML metadata store CI / CD component
model training computation infrastructure continuous integration
DE / continuous delivery
Feature store ML Experimentation Zone versioned artifacts: model + ML training & workflow code (build, test and push)
system
ML Production Zone continuous deployment
offline DB (build, test and deploy model)
(normal
Scheduler artifact
(trigger when Workflow orchestration component
latency) store Model Registry
new data (e.g., Image prod ready
available, Registry) ML metadata store ML model DO OR ML
ML metadata store
online DB event-based or model status (staging or prod)
(low-latency) periodical) (metadata logging of each ML workflow task)
parameter & perf. metrics
D Automated ML Workflow Pipeline (best algorithm selection, parameter & perf. metric logging)DO
OR ML
versioned
feature data data
data model training model export push to model
preparation & registry
extraction / refinement validation model
validation
model training computation infrastructure ML
Monitoring component DO OR ML Model serving component
Feedback Loop – enables continuous training / retraining & continuous improvement (prediction on new batch or
continuous monitoring of model serving performance
streaming data) {…}
new versioned feature data (batch or streaming data) SE
model serving computation
ML infrastructure
LEGEND {…} General process flow Feedback loop flow
BS SA DS DE SE DO ML
Data Engineering flow Versioned Feature Flow
Business IT Solution Data Data Software DevOps ML Engineer
Stakeholder Architect Scientist Engineer Engineer Engineer Model / Code flow
Figure
Figure 4. End-to-end MLOps
4: End-to-end architecture
MLOps and workflow
architecture with functional
and workflow withcomponents and
functional roles
components and roles [2, p. 6].
(11) The feature engineering task calculates new and more (C) Experimentation. Most tasks in the experimentation stage
advanced features based on other features. The predefined feature are led by the data scientist (R3). The data scientist is supported by
4. MLrules
engineering Production Zone.
serve as input This
for this zone
task. Theseincludes
feature an automated pipeline
the software engineer (R5). (D.
(13) Automated ML
The data scientist (R3)Workflow
connects to
Pipeline),
engineering which ensures
rules are continuously improved data preparation,
based on the feedback. training,
the featurevalidation, andforregistration
store system (C4) of(Alternatively,
the data analysis. the model
(12) Lastly, a data ingestion job loads batch or streaming data into the data scientist (R3) can also connect to the raw data for an initial
in production mode. The Model Serving component
the feature store system (C4). The target can either be the offline or
deploys the model, and the Monitoring
analysis.) In case of any required data adjustments, the data
Component
online database ensures
(or any kind its continuous monitoring. In case of problems, information is transmitted
of data store).
through the Feedback Loop to initiate model retraining.
In addition to functional components, the figure also shows different roles and their areas of respon-
sibility: Business Stakeholder (BS), Data Scientist (DS), Data Engineer (DE), DevOps Engineer (DO), ML
Engineer (ML), Software Engineer (SE), and IT Solution Architect (SA).
3.3. Model and data versioning
Versioning is one of the key MLOps practices that ensures reproducibility and traceability of machine
learning models [2, p. 3]. Versioning covers data, code, and the models themselves [7, p. 9]. Model
383
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
versioning not only captures model artifact versions but also model dependencies for tracking or
reproducing different model versions that rapidly change over time [7, p. 13].
The purpose of data versioning is to guarantee the reproducibility of models and compliance with
regulatory requirements. Data versioning can be implemented either by storing data snapshots or by
referencing the original dataset. Since traditional version control systems cannot handle large amounts
of data, specialized tools such as Data Version Control (DVC) are used [7, p. 11].
In the MLOps workflow, model versioning occurs at the model training stage. Its purpose is to save
different versions of models along with their metadata for the ability to roll back to previous versions
and reproduce results. The conditions for using model versioning are: (1) the constant evolution of
models over time; (2) the need to track dependencies between models, data, and code [7, p. 12].
A feature of model versioning, as opposed to traditional code versioning, is the need to track a larger
number of artifacts and metadata, as well as the need for larger amounts of memory due to the constant
development of models [7, p. 13].
In addition to the data itself, versioning is subject to dependencies, data processing steps, and extracted
features [7, p. 11]. For the latter, specialized feature stores are often used.
Model dependencies capture the relationship with related elements such as the corresponding dataset,
source code, and configuration files. In addition, model versions store associated log files and model
evaluation results. This allows checking whether model versions are constantly improving throughout
the continuous lifecycle. Since versioning of artificial intelligence models is more complex and requires
more memory due to continuous development, standard version control systems such as Git cannot be
used as model repositories. Potential alternatives such as MLFlow, H2O, and DataRobot are container
registries where image versions are stored, or model repositories that store model versions, including
code, metadata, test results, and dependencies [7, p. 13].
3.4. ML pipeline automation
ML pipeline automation is a key MLOps practice that allows simplifying and accelerating the develop-
ment, testing, and deployment of ML models in a working environment. This practice encompasses
the automation of various stages of the ML pipeline, including data collection, preprocessing, model
development, training, testing, validation, and deployment [7, p. 2].
The algorithm of an automated machine learning pipeline (figure 5) consists of the following stages:
1. Process initiation.
2. Extraction of versioned data from the storage.
3. Automated data preparation and validation.
4. Automated model training on new data (iteratively).
5. Model evaluation and hyperparameter tuning.
6. Model export.
7. Storing the model in the model registry.
8. Model deployment.
9. Model serving to obtain predictions.
10. Monitoring model performance.
The feedback loop ensures continuous improvement of the model by returning to the training
stage. If necessary, monitoring can initiate a model retraining trigger, which starts the process from
the beginning. This automated pipeline provides the ability to effectively manage the lifecycle of a
384
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Initiation
Extraction of versioned
data from storage
Automated data preparation
and validation
Iteratively
Automated training
Ret
on new data
rain
ing
trig
ger
Model evaluation,
hyperparameter tuning
Model export
Model
Model registry
deployment
con F
tinu eedbac
ous k
mod loop fo
el im r
prov Model serving
em ent to obtain predictions
Performance monitoring
Figure 5: Algorithm of an automated machine learning pipeline (based on Kreuzberger et al. [2]).
machine learning model, from data preparation to deployment and monitoring, ensuring continuous
improvement of the model. Features of using ML pipeline automation include the need to take into
account the heterogeneity of models, frameworks, and execution environments. Therefore, each step of
the ML pipeline should be as isolated as possible and have clear interfaces, for example, through the
use of containerization [22, p. 6]. It is also critically important to ensure the ability to reproduce results
and track artifacts [23, p. 1706].
Methods of ML pipeline automation include the use of pipeline management systems such as Kubeflow,
385
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
TFX [7, p. 19] or Apache Airflow [6, p. 4], as well as the development of custom automation scripts using
tools such as Jenkins [24] or GitLab CI/CD [2, p. 2]. In this case, the pipeline is divided into separate
steps, each of which is implemented as code or configuration, and then these steps are orchestrated and
executed automatically [23, p. 1706].
3.5. Model performance monitoring
Monitoring the performance of machine learning models is an important MLOps practice that allows
tracking the operation of models in a production environment, identifying problems, and taking
measures to maintain their quality. This practice covers the collection of metrics regarding the operation
of the model, real-time monitoring of these metrics, and alerts in case of detecting deviations from the
norm [25, p. 128].
Model performance monitoring occurs at the model operation stage, after its deployment in the
production environment. It is part of the continuous MLOps cycle and is performed in parallel with
other practices, such as development, testing, and deployment [25, p. 127].
The conditions for using monitoring are the presence of a deployed ML model that processes real
data and generates predictions. In addition, an infrastructure for collecting and storing monitoring data,
for example, a database and visualization tools, should be set up [25, p. 134].
The monitoring process includes several steps (figure 6). First, key model performance metrics are
defined, such as accuracy, error, latency, etc. Then, tools for collecting these metrics from the system
where the model is deployed are set up. Next, the data is aggregated and visualized on dashboards for
convenient analysis. Finally, alerts are set up that are triggered when metrics go outside of acceptable
limits [25, p. 129].
Figure 6: Monitoring process in MLOps [25, p. 129].
Figure 7 depicts the analogy proposed by Bodor et al. [25] between monitoring a machine learning
system and an iceberg. This analogy emphasizes that the health of an ML system depends not only on
visible elements, such as provided services, but also on hidden features that are difficult to track, such
as data and the model itself:
386
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Figure 7: Hidden and visible characteristics associated with monitoring [25, p. 130].
• The top level of the iceberg represents ecosystem health, which includes aspects such as prediction
drift and business key performance indicators (KPIs). The goal is to assess the performance,
reliability, and stability of this top level.
• The service level includes metrics such as latency, cost, and overall system performance.
• The data level contains data quality characteristics, outlier values, and data drift.
• The lowest level of the iceberg is the model, which is characterized by accuracy, concept drift,
and model bias.
A feature of monitoring in the context of MLOps is the need to track not only traditional software
performance metrics (for example, CPU and memory usage), but also metrics specific to ML, such as
model accuracy on new data (figure 8). In addition, MLOps involves automating the monitoring process
and integrating it into the overall model development and deployment pipeline [25, p. 128].
There are a number of tools for setting up model performance monitoring in ML. They include
open-source platforms such as Prometheus, Grafana, ELK stack, as well as commercial solutions from
cloud providers, for example, AWS CloudWatch, Google Stackdriver, Azure Monitor [25, p. 135].
3.6. Experiment management
According to Singh [26], experiment tracking involves systematically storing metadata of machine
learning experiments [26, p. 153].
Czakon and Kluge [27] defines that experiment metadata can include arbitrary scripts to run the
experiment, environment configuration files, information about training and evaluation data, model pa-
rameter and training configurations, ML evaluation metrics, model weights, performance visualizations
(e.g., confusion matrix or ROC curve), etc.
Experiment management (also known as experiment tracking) is part of MLOps focused on supporting
iterative model development – the part of the ML project lifecycle where, in particular, hyperparameter
tuning is performed to achieve the required level of model performance. Experiment management is
closely intertwined with other aspects of MLOps, such as data and model versioning.
387
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Figure 8: Elements for monitoring [25, p. 131].
Data Prediction
collection monitoring
Data Data Experiment Model Model
labeling versioning management versioning deployment
Model architecture
Model training
Model evaluation
Figure 9: Experiment management in the MLOps lifecycle (adapted from Czakon and Kluge [27]).
The main condition for applying experiment management is the iterative nature of the model
development and training process, when many experiments are conducted with different sets of hy-
perparameters, model architectures and training data (figure 9) [27]. This is typical for research and
applied machine learning projects.
The most popular tools for experiment management are MLflow, Neptune.ai, Weights & Biases
(wandb), Guild.ai, Comet.ml and TensorBoard [2]. They allow storing information about each experiment
388
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
in a repository – data used and its version, model parameters and architecture, performance metrics,
model artifacts, etc.
3.7. Model deployment
Model deployment is an important MLOps practice that occurs at the operationalization stage in the
workflow of developing and deploying ML models. This practice involves directly placing a trained and
tested machine learning model into a production environment where it can be used to obtain predictions
on real data [11, p. 1].
According to Kolltveit and Li [11], deployment, i.e. the transition of a packaged and integrated model
into a service state, can occur in several different ways. Models packaged in containers are simply
run directly as standalone services. However, models can be deployed in a target environment that is
different from where they were packaged, and in this case, model transfer must occur using a push or
pull pattern [11, p. 4]:
• in a pull-pattern deployment, the target environment (host application running, e.g., on a server
or edge device) periodically polls for model updates and downloads them when available;
• in a push-pattern deployment, the target environment is notified of the availability of a new
model by the master server (e.g., the server where the model was trained) through a messaging
service, where the message contains metadata including the location of the updated model, or by
initiating model transfer to the target environment through a specific receiving interface.
ML model deployment often occurs by packaging them in containers, for example, using Docker.
This allows standardizing the deployment process and ensuring that the model will run in the same
environment as during development and testing [11, p. 5].
There are different ways to deploy ML models depending on the requirements and architecture of
the system [28, p. 68]:
• the model can be integrated directly into the application code;
• the model can be deployed as a separate service (microservice) with a REST API;
• the model can be loaded into a specialized environment for deploying and scaling ML models (for
example, AWS SageMaker).
ML model deployment is associated with a number of problems, including ensuring low latency and
high throughput for the prediction service [11, p. 5]. Various methods are used to solve them, such as
adaptive queues with timeout for batch prediction, caching, dynamic switching between models of
different accuracy, etc.
In terms of tools, container orchestration systems (Kubernetes), cloud provider services (AWS Sage-
Maker, Azure ML), end-to-end MLOps platforms (MLflow, Kubeflow) are used for ML model deployment
[28, 3].
ML model deployment is a critically important MLOps practice that allows transitioning developed
models into a working state and using them for prediction. It occurs at the operationalization stage
and requires considering a number of factors - from the way the model is packaged to ensuring the
necessary performance of the prediction service. Deployment relies on modern tools for containerization,
orchestration and infrastructure automation.
Based on the generalization [11, 28], a general scheme of machine learning model deployment was
constructed (figure 10):
1. ML Model – a trained and tested machine learning model.
2. Packaging – the model is packaged in an appropriate format (e.g., a Docker container).
3. Model registry – the packaged model is placed in a model registry.
389
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
4. Deployment – the packaged model is deployed in the target environment (cloud, edge devices).
5. Serving – the model serves requests and generates predictions.
6. Monitoring – the performance of the model and environment is monitored. If necessary, model
retraining is initiated.
ML model
Packaging Model registry
Deployment
Monitoring Serving
Figure 10: ML model deployment scheme.
3.8. Lifecycle management
According to Steidl et al. [7], continuous lifecycle management ((end-to-end) lifecycle management) is
a continuous pipeline/flow management, which describes the (automatic) execution of specific tasks
to ensure the management of the lifecycle of artificial intelligence [7, p. 7], which starts with data
collection and ends with the deployment and monitoring of the artificial intelligence model [7, p. 2].
The goal of lifecycle management is to unify and standardize processes, which contributes to increas-
ing the productivity of ML model development and their reliability in the industrial environment.
Key features of lifecycle management in MLOps:
• covers all stages: data collection and preparation, model development, training, validation and
deployment [25, p. 127] (figure 11);
• is applied both at the early stages of model development and for their continuous support after
deployment in the production environment [5, p. 9];
• involves versioning of data, code and models themselves to track changes and ensure reproducibil-
ity;
• involves monitoring model performance;
• automates processes using pipelines [7, p. 8].
Figure 11 shows the ML project development process (pipeline), which consists of a number of steps
that are both linear and iterative (MLOps Pipeline block). The pipeline starts with data extraction,
validation and preparation, then model training, evaluation and validation (experiment management).
After that, the model is deployed in the production environment [25, p. 128].
An important aspect of the lifecycle is versioning of models and data, which occurs both in the
experiment management pipeline and the production pipeline: the latter includes all stages of model
390
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Figure 11: Machine learning project lifecycle in MLOps [25, p. 127].
creation, not just the final result of the experiment management pipeline. The iterative nature of MLOps
provides the opportunity to obtain and maintain the best model, and the combination of monitoring
based on key performance indicators and alert functions enables proactive intervention, ensuring the
quality and reliability of deployed models throughout the lifecycle [25, p. 128].
The main tools used for lifecycle management in MLOps [25, p. 126]:
• platforms for organizing ML workloads and pipelines, such as MLflow, Kubeflow;
• data versioning systems, for example DVC (Data Version Control);
• tools for monitoring model performance in the production environment, such as Neptune.ai.
A comprehensive approach to lifecycle management in MLOps is implemented in Ease.ML (https:
//ease.ml/) – a lifecycle management system designed to simplify the entire development process [29].
The main goal of Ease.ML is to provide systematic recommendations and automation at all stages of the
ML lifecycle, minimizing user effort.
Key features of Ease.ML:
1. Human-in-the-loop process – Ease.ML incorporates user interaction in a structured way, providing
the ability for users to input data and make decisions at critical stages.
2. Probabilistic data model – the system uses a probabilistic database that handles uncertainty in
data, which can arise from incorrect data, weak supervision, or other sources.
3. Interactive environment – Ease.ML uses Jupyter notebooks, allowing users to perform data manip-
ulations, run ML operations, and visualize results in an integrated environment.
4. Lineage graphs – user interactions and operations are tracked in lineage graphs, which represent
the entire ML workflow and ensure reproducibility.
391
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
5. Automatic quality tuning and recommendations – the system provides recommendations for model
improvements based on errors detected in production, guiding users through data cleaning and
collection tasks efficiently.
The Ease.ML process is divided into three sub-processes and consists of eight stages that cover the
entire ML lifecycle:
• Day 0: Pre-ML Subprocess
1. Problem formulation – clearly define the problem and ML goals.
2. Feasibility study – assess the feasibility of an ML solution with the available data and
resources.
• Day 1: AutoML Subprocess
3. Data preparation – clean, preprocess and augment data to make it suitable for training ML
models.
4. Model training, using prepared data, with the use of AutoML to automate model selection
and tuning.
5. Evaluation of trained model performance on validation datasets to determine compliance
with criteria.
6. Selection of the best model and its deployment in the production environment.
• Day 2: Post-ML Subprocess
7. Continuous integration and delivery – integration of the model into the production environ-
ment and setup of CI/CD pipelines to manage model updates and performance monitoring.
8. Model maintenance through continuous monitoring of the model in the production environ-
ment and necessary updates or retraining of the model to adapt to new data or changing
conditions.
3.9. Data security and privacy
Data security and privacy is the practice of protecting information and data used in machine learning
processes from unauthorized access, misuse, and leakage [30, p. 1]. It aims to ensure the integrity,
availability, and confidentiality of data at all stages of the MLOps lifecycle.
Data security and privacy should be considered at all stages of MLOps, from problem definition
to monitoring of the deployed system. Ensuring security at the stages of data management, model
development and deployment is especially critical, as this is where data is most vulnerable [31, p. 8].
The practice of data security and privacy is mandatory in cases where the ML system operates
with sensitive data (personal, financial, medical, etc.) or is deployed in mission-critical environments
(healthcare, automotive industry, manufacturing). However, even for less sensitive applications, an
appropriate level of security must be ensured in accordance with regulatory requirements and user
expectations.
Unlike traditional software development, in the context of MLOps, new attack vectors and vulnerabil-
ities specific to machine learning emerge (figure 12). In particular, ML models are vulnerable to attacks
such as data poisoning, model inversion, and adversarial example attacks [30, pp. 8-15]. This requires
the use of specialized defense strategies.
To ensure data security and privacy in MLOps, both standard security tools (encryption, authenti-
cation, logging, etc.) and specialized ML-oriented solutions are used. Examples of the latter include
libraries for secure federated learning aggregation, tools for testing models for vulnerabilities, frame-
works for privacy-preserving machine learning based on homomorphic encryption or secure enclaves
[31, pp. 10-11].
392
Literature review
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414 9
Model extraction Model stealing
attacks attacks Model poisoning
System Security for
MLBSS attacks
Model-reuse
Model-oriented attacks
attacks
Model adversarial
attacks
Model evasion
Data-oriented Model inference
Model trojan attacks
attacks attacks
attacks
Data adversarial Data poisoning System-level DNN
attacks attacks attacks
System-oriented
Side-channel
attacks
Data backdoor attacks
attacks Covert channel
User privacy attacks
attacks
Platform attacks
Figure 12: Classification of attacks on MLOps systems [30, p. 9].
Fig. 3. The taxonomy for system security of machine learning-based software systems
Data security and privacy is ensured by implementing various control and protection mechanisms at
holding local data samples [199]. The settings have a larger liability of exposing the attack surface to the adversary, in
each stage of MLOps. This includes, in particular, data encryption and anonymization, authentication
which the adversary
and access can manipulate
control, integrity the training
checking, data monitoring,
activity in differentincident
stages. response,
In general, the success
as well ofsecurity
as regular machine learning
algorithms audits andthe
assumes penetration
integritytesting
of the[30, pp. 4-5].
training data and serving data. Availability of the training data is one main
Figure 13 summarizes the main security components and practices in MLOps presented in [30,
concern for security. The goals of data-oriented attacks can be summarised as to: 1. degrade the machine learning model
pp. 22-25].
performance; Essential
2. manipulate the prediction
components of MLOps:outcome at test time.
Review & discussion:
• problem definition includes understanding the problem and system requirements;
Overall, the data-oriented attacks could be categorised into three major types of attacks, which are poisoning attacks,
• data
backdoor attacks managementattacks.
and adversarial covers data
Whilecollection,
poisoning labeling
attacksand verification;
have been a general term summarising most popular
attacks, in this •work
model construction
poisoning and deployment
attacks includes
represents the model selection,
triggerless poisoningbuilding,
attacks optimization and evalua-
while the backdoor attacks is the
tion,
backdoor poisoning as wellTriggerless
attacks. as its deployment in the
poisoning targetrefer
attacks environment;
to successfully mislead the model and system behaviour
• system
without manipulating of maintenance involves
the data during modelmonitoring
inference the functioning
stage. of theattacks,
For backdoor deployed system.
the attack will be activated when
the input dataSecurity
contains crafted
practices in trigger,
MLOps: which could be pixel-level features in an image or designed characters in a
sequence. In Table. 2, we have collectively summarised 15 most recently exemplar studies to cover the modes targeting
1. At the problem definition stage:
on different development stages.
• risk assessment;
For both triggerless poisoning attacks and backdoor poisoning attacks, the attacks mostly happen during the data
• threat modeling.
management stage when the attackers could easily obtain the access to the dataset. This assumption is made against
the centralised2.machine
At the data management
learning models stage:
[7, 71, 91, 161, 172, 182, 190], whilst one specific scenario is identified for the
distributed machine•learning
data flowmodels.
diagram;In [129, 176], the federated learning-based systems are studied that the global
model is poisoned by• aggregating
STRIDE methodology for threat
the edge node modelclassification;
updates learned from malicious participants. The last attacking
• conceptual
stage for backdoor poisoning modeling;
attacks is discovered in [88], in which the model is retrained periodically at system
• data validation;
maintenance stage. Generally, attacks have impacts on the data integrity and availability.
Although several•earlier
data linter;
works [120, 128, 143, 153] have identified the poisoning attacks in different applications,
• data verification.
one earliest work presenting a more realistic assumption by limiting the adversary’s capability and knowledge of the
system was led by Suciu et al. [172]. Neither the feature nor the algorithm knowledge was taken for granted as the
Manuscript submitted to ACM
393
n the development of MLBSS are specific and can not be well managed within the SWEBOK topics, especially in term
of the technical and data-related challenges. This, in turn, calls for adjusting the practices in the context of system
domain and technical
Danylo O.issues.
Hanchuk et al. CEUR Workshop Proceedings 362–414
Problem • Security • Assess and Data
understanding requirement determine acquisition
System configuration engineering data quality
attributes Data labeling
Data flow
diagram Problem Data Data linter
definition management Data
STRIDE Threat Data
validation
Conceptual modelling verification
modelling
Model
System construction
maintenance and
System testing & deployment
debugging ML model
• Post-
deployment • Model testing
monitoring
System Test adequacy
monitoring identification
SelfChecker Fuzzing Testing against
MLDEMON techniques adversarial input
Essential components for MLBSS Identified secure practice
Figure 13: Main security components and practices in MLOps [30, p. 20].
Fig. 4. The essential components [162] and identified secure practices for MLBSS
3. At the model construction and deployment stage:
Currently, the research and development
• identification of the state-of-the-practice
of test adequacy criteria; for MLBSS are mostly focused on two topics
which are technical debt and quality
• testing againstassurance. Generally, quality assurance defines ‘a planned and systematic pattern
adversarial input;
of all actions necessary •tofuzzing
providetechniques.
adequate confidence that the item or project conforms to established technical require
ments’ [20]. For technical debt, at
4. At the system its broadest,
maintenance stage:it was defined as ‘any side of the current system that is considered
sub-optimal from a technical perspective’
• ML model [81,tools
monitoring 177].(MLDEMON,
While the discussion
SelfChecker);of technical debt is related to the compromised
decisions for the poor •software
static analysis;
development [93], the quality assurance dedicates to system quality offline and in
• exploratory
production attached with attacks;aspects, i.e., model quality, safety and fairness analysis [76].
several distinct
• evasion
Technical debt, studied in [177]attacks;
for classical software systems, has recently been reviewed in MLBSS in order to identify
• data
he specific patterns [14, poisoning
93, 162, 175].attacks;
Additional to typical technical debt at code level, the machine learning-based
systems represent extra• trade-offs
manual testing;
to be overcome for practical practices for the long term development, deploymen
and maintenance at the• system
dynamic level.
analysis.
The data dependency debt was extensively discussed as a key contributor fo
MLBSS since it isThus,
onesecurity practices in
of the integral MLOps are integrated
components in MLBSS intoand
all main stages of more
is generally the development lifecycle
difficult to detect as a result o
and include both general methods of security assessment and testing (threat modeling, static/dynamic
ts unstable and underutilised characteristics [162]. Other technical debts, i.e., abstraction debt, configuration deb
analysis) and specialized approaches focused on the peculiarities of machine learning systems (testing
against
data testing debt, adversarial examples,
reproducibility detection
debt, process of data poisoning).
management debt and cultural debt, may also be accrued and demand
ongoing collaborative efforts. Following in [14], 21 primary studies were included to identify the actual technical debt
3.10. Model explainability and interpretability
hat have been investigated in recent works, in which four novel emerging debts of data, model, configuration and
Explainability/interpretability of models is an important MLOps practice for ensuring transparency
ethics were elaborated. Taking
and trust in further
machine actions
learning towards
models. measuring
Explainability refers the technical
to the ability todebt andand
explain targeting to the
understand pay it off, a recen
decision-making
Manuscript submitted to ACM processes of a machine learning model [10, p. 66]. This is especially important when
the model is used to make decisions that have significant consequences, such as in military operations
or law enforcement activities [9].
394
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Explainability as the basis for trust in the ML project allows users to trust the prediction, which
increases transparency. The user can verify which factors contributed to certain predictions, introducing
an additional layer of accountability. The terms “explainability” and “interpretability” are often used
interchangeably, but for MLOPs, explainability is more than interpretability in terms of importance,
completeness, and fidelity of predictions or classifications (figure 14) [4, p. 63608].
Individual case Model
explanation accountability
Explainability Interpretability
Feature importance
Human readability
explanation
Figure 14: Relationship between explainability and interpretability in MLOps.
Explainable Artificial Intelligence (XAI) is a research direction that promotes explainable decision
making [4, p. 63609]. Explainability can be defined as the degree to which an observer can understand
the cause of a decision [32, p. 8]. An ML system is explainable when it is easier to identify causal
relationships between the inputs and outputs of the system. The more explainable a model is, the better
practitioners understand the internal business procedures that occur during model decision making. An
explainable model does not necessarily translate into a model that a human can understand (internal
logic or processes underlying it), but the explainability of the model allows the user to strengthen trust
in the predictions made by the deployed system [4, pp. 63614-63615].
Various methods and tools can be used to achieve model explainability, such as:
• attribution-based methods (integrated gradients, saliency maps) or perturbation-based methods
(SHAP) [33, p. 3], which explain the model’s decision by assigning high scores to the most
influential input features;
• incorporating an attention mechanism into models, which allows focusing on the most relevant
network states and inputs [33, p. 2];
• using a combined reward signal during training, which includes not only target metrics but also
interpretability metrics [33, p. 4].
3.11. Data quality management
Data quality management is an important practice in the MLOps workflow. According to Steidl et al.
[7], the data processing stage includes the full lifecycle of working with data, including pre-processing,
quality assurance, versioning and documentation [7, p. 21].
In Haller’s book [9, pp. 77-84], data quality management is considered in the context of monitoring
and checking data in a production environment to ensure that the data used corresponds to reality.
In the data-centric MLOps lifecycle (figure 15), data quality management is performed at the following
stages:
395
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Figure 15: Data-centric MLOps lifecycle [26, p. 145].
1. Data collection – at this stage, data is created and obtained from various sources: data must be
relevant, complete, consistent and reliable.
2. Data quality assessment and cleaning – collected data undergoes thorough quality checks using
various metrics such as accuracy, completeness, consistency, timeliness, etc.; quality issues
are identified and resolved – incorrect values, missing values, duplicates, noise; data cleaning
techniques are applied to improve data quality.
3. Data augmentation and labeling – cleaned data is enriched with additional information and
labeled according to the target task; at this stage, the quality of data labeling is also controlled to
avoid errors and inaccuracies.
4. Data quality analysis – labeled data is analyzed for quality, its representativeness, class balance,
presence of outliers and anomalies are checked; adjustments are made as needed to improve data
quality.
5. Model training with quality control – based on quality data, ML model training takes place;
during experiments, model and data quality metrics are tracked to ensure stability and reliability
of results.
6. Model deployment with quality assurance – the model is deployed in the production environment
only after thorough testing on quality test data; measures are taken to maintain data quality in
the production environment.
7. Data and model quality monitoring – the deployed model and incoming data are constantly
checked for quality: quality metrics, presence of anomalies in data, distribution shift are tracked;
the model is retrained on new quality data as needed.
To guarantee data quality throughout the entire lifecycle of their use in MLOps, some approaches to
data validation and verification need to be applied:
• data quality assessment by determining how suitable this data is for achieving business goals
(completeness, uniqueness, integrity, validity, accuracy, timeliness) [26, pp. 2-3];
396
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
• data validation in single and cross batches by comparing data characteristics with the expected
schema, as well as checking for data drift [7, p. 11];
• automated modular data tests based on a schema to identify errors in data and prevent them from
entering the model training stage [7, p. 10];
• versioning of data and related artifacts (processing procedures, metadata, etc.) for traceability,
reproducibility of results and compliance with regulatory requirements [7, p. 11];
• documenting data to an extent sufficient to ensure model verifiability [7, p. 12].
Timely detection and elimination of defects in data helps prevent wasting computational resources
on low-quality data [7, p. 11].
In their work, Singh [26] present techniques for assessing the quality of “big data”, which help identify
datasets that may cause problems and unnecessary costs.
To implement data quality management practices, frameworks such as TensorFlow Extended (TFX)
can be used, which have data validation components such as SchemaGen and ExampleValidator [7,
p. 19].
3.12. Configuration management
Configuration files help create more robust software by moving all hardcoded variables into dedicated
locations that can be split up or organized at the developer’s discretion [34, p. 3].
Godwin and Melvin [34] propose a template that supports two types of configuration files – a
config.py file that contains the template configuration and can include additional resources, such as
databases and spreadsheets, and JSON files for storing specific program variables, such as thresholds or
parameters. The config.py file is accessible through import statements, while JSON files are called from
disk at runtime [34, pp. 3-4].
Yongqiang et al. [35] consider the use of a unified data model based on the YANG language for
unifying the description of configuration data and simplifying their management. Neptune Labs [36]
indicate that configuration management tools, such as Ansible (https://www.ansible.com/), Puppet
(https://www.puppet.com/) and Chef (https://www.chef.io/), can be used to automate configuration and
provisioning of MLOps platforms.
3.13. Model deployment strategies
Model deployment strategies:
• are implemented at the final stage of the MLOps process;
• require standardization, automation and encapsulation of models;
• are used for gradual transition of a new model to the production environment;
• rely on containerization.
Peltonen and Dias [28] highlight the following benefits of using containers for model deployment
[28, p. 68]:
• model abstraction and process isolation by running multiple models in individual containers
representing their runtime dependencies;
• ability to create model-specific containers that meet their packaging requirements;
• assignment to different processors (CPU, Mobile GPU, etc.);
• ability to allocate a separate container to facilitate post-processing functions;
397
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
• a model repository and container storage can be used for fast deployment;
• provides a means of standardized deployment;
• almost all existing continuous development pipelines facilitate model deployment in the form of
containers;
• containers can be tailored to specific architectures of edge devices;
• Docker containers are a well-established industry standard;
• ease of rollback in case of failures.
Gunny et al. [37] describe in detail the development cycle and model deployment strategy using
the DeepClean application as an example. A new version of the model is first deployed as a developer
version, undergoes validation in conditions similar to production, and only then replaces the current
model in the production environment. This uses a service architecture with NVIDIA Triton Inference
Server, which supports simultaneous placement of the developer version and the production version of
the model.
The authors identify two main approaches to model deployment [37, pp. 11-12].
In the traditional
Workshop scenario, each user manages their own resources and model versions. Inconsis-
Presentation
tencies in libraries and dependencies, as well as model versions, lead to inconsistent results. Reduced
computational requirements for inference lead to underutilization of hardware resources, depicted
by green rectangles on each node (figure 16). More complex deployment scenarios require the use of
multiple networks, exacerbating existing issues.
Git repository
User node on
compute cluster
Local/shared
storage
Version of DL
software stack
Ops from different
DL frameworks
Figure 16: Traditional
(a) distributed
Traditional deployment scenario [37,deployment
distributed p. 12]. scenario (b)
Figure 2: (a) A traditional deployment scenario in which individual users m
Inconsistencies in libraries and398dependencies as well as model versions lea
demands of inference lead to hardware under-utilization, represented b
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Training Job
User node on
compute cluster
Training Job Cloud or
local model
repository
Local/shared
storage
Version of DL
software stack Containerized
inference service
Ops from different gRPC inference
DL frameworks requests
Figure 17: Deployment using the Inference-as-a-Service scenario
(b) Inference-as-a-Service [37, p. 12]. scenario
deployment
ario
In the Inference-as-a-Service approach, a centralized service orchestrates models and provides unified
which individual users manage their own software and hardware resources.
interfaces for invoking models (figure 17). A centralized model store synchronizes all users and keeps
well as model
themversions lead to
up to date. Pipelines sendinconsistent results.
gRPC inference requests to theReduced computational
service using standardized APIs that
abstract away the details of the inference execution itself. Inference is performed by a containerized
utilization, represented by green rectangles on each node. More complex
service that can efficiently schedule asynchronous model execution, maximizing hardware compute
e networkscapabilities
utilizing multiple
in a portable framework
and scalable manner. In thisbackends, exacerbating
approach, containerization existing
allows creating portable
and isolated model execution environments.
ndardizes inference across all users and coordinates complex concurrent
Choosing the right model deployment strategy allows minimizing operational costs, ensuring consis-
e capacity tent
in amodel
way that for
operation is all
portable and scalable.
users, and facilitating model version monitoring and control.
3.14. Infrastructure automation
equired to Infrastructure
usersas to pick and choose which libraries they need in order to
code is an important MLOps practice that allows treating infrastructure as code for its
ok like the reliable keep theirdeployment
and efficient deployments lightweight.
and management [38, p. 2].In this section,
Infrastructure we will
automation briefly
relates to the
s that look deployment and monitoring stage in the MLOps process (figure 18).
describe the purpose of the most relevant libraries, and leave more It ensures reliable creation of the
necessary environment for deploying machine learning models and automates infrastructure tasks [38,
e Model as p. 3]. detailed information to their documentation.
ed point in Infrastructure as code involves describing infrastructure configuration in a declarative way in special
files (for4.1.1 in YAML, JSON formats) or usingThe
example, hermes.cloudbreak. cloudbreak
special libraryTerraform)
languages (for example, contains or tools
tools.
ream users These files describe the desired state of the infrastructure [38, p. 4].
for provisioning Kubernetes clusters and virtual machines on pri-
e true test Describing infrastructure settings as code allows automating the process of its creation, modification
vate clouds and Thisdeploying workloads onto environments,
those computational
s practices and management
resources and
[38, p. 3].
avoid manual
makes it possible
setup errors [38, p. 4].
to fully reproduce
This approach is appropriate to
quickly deploy
use for Cloud,
creating
resources. While support only currently exists for Google
oyment via complex dynamic infrastructures, when high repeatability and consistency of environments is needed,
the intent of the library is to be written in such a way that the
atable, com-
user interface is agnostic to the actual cloud backend. Moreover, by
using Python contexts to deploy resources, we can ensure that any
399
resources are spun-down once jobs are complete so that unneces-
sary costs are not incurred. cloudbreak is not currently used as
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Data collection Data analysis Model building Model testing
and preparation and exploration and training and validation
Model
Model deployment
performance
(Infrastructure automation)
monitoring
Infrastructure as Code
Model retraining and updating
Figure 18: Infrastructure automation in the MLOps lifecycle.
to increase reliability and reduce time costs for manual configuration [38, pp. 2, 4].
The automated approach allows testing infrastructure as code, applying software development
practices to it (code review, versioning, etc.). This contributes to improving stability and security, allows
quickly tracking and resolving infrastructure issues [38, p. 4].
To implement Infrastructure as code in Azure DevOps, tools such as Azure Resource Manager (ARM)
Templates, Terraform, Ansible, Chef are used [38, pp. 3-4]. They allow describing infrastructure as code
and automatically creating or modifying it.
3.15. Collaboration and communication
Collaboration and communication between various stakeholders is a key MLOps practice for the
successful implementation of machine learning and artificial intelligence projects in organizations.
MLOps emphasizes how cross-functional teams, such as data analysts, system operators, as well as data
and software engineers, collaborate through a harmonized process [7, p. 7].
The essence of the collaboration and communication practice lies in establishing effective interaction
and information exchange between the various teams involved in the process of developing and
implementing ML models – the data science team, development team, operations team, and business
units. This practice is an important component of the development and implementation stage of the
MLOps workflow.
As Kreuzberger et al. [2] point out, MLOps involves close collaboration between data science (machine
learning) teams, which are engaged in data preparation and model development, software development
engineers, who are responsible for integrating models into the production environment, and operations
teams, which ensure the deployment and support of models [2, p. 2]. Effective communication is
necessary at all stages of the ML model lifecycle, from defining business goals to monitoring models in
the production environment. Without well-established collaboration, the development of ML solutions
can be delayed, conflicts of interest and misunderstandings may arise between participants [39, p. 3].
The diagram (figure 19) shows the main participants in the MLOps process and the directions of their
interaction:
1. The Data Science team prepares data and develops machine learning models.
2. The Development team integrates the developed models into software products.
3. The Operations team deploys models in the production environment and provides their support.
4. The Business team defines goals and requirements for ML solutions, and also receives results
from the Data Science and Operations teams.
400
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Defining goals and requirements
Business
Collaboration
Data tools: Slack,
preparation, Trello, GitLab wiki Model
model deployment
development and support
Data Science Development Operations
Integration of models into the product
Figure 19: Diagram of collaboration and communication in MLOps.
To ensure effective communication in MLOps practice, the following approaches and tools are used
[2, p. 4]:
• use of collaboration and knowledge sharing tools, such as Slack, Trello, GitLab wiki;
• regular meetings between teams to discuss status, problems and plans;
• clear definition of roles and areas of responsibility of process participants;
• use of version control systems (Git) for collaboration on code and models;
• automation of CI/CD processes to ensure transparency and reproducibility of development.
3.16. Risk management and compliance
Since ML models often make important decisions that affect people, risk management and compliance
is a critically important MLOps practice, the essence of which is to ensure compliance of developed ML
models and systems with regulatory requirements and standards (compliance) and manage potential
risks associated with their development and operation.
This practice has a cross-cutting nature and manifests itself at different stages of the ML model
lifecycle. Risk management in the context of MLOps involves identifying and mitigating potential risks
associated with ML models, such as data bias, privacy breaches, model accuracy deterioration over
time, etc. Compliance means ensuring that models comply with regulatory requirements, for example,
regarding data protection [7, pp. 18-19]. The main condition for using this practice is the presence of
regulatory requirements or industry standards that the ML system must comply with. Examples can be
GDPR requirements for personal data protection or certifications in the healthcare industry [7, p. 5].
401
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Steidl et al. [7] indicate that compliance aspects should be considered during data preparation, model
training and validation, as well as deployment and monitoring [7, p. 18].
Methods of using this practice include regular data quality checks, testing models for bias and
discrimination, implementing access control and data encryption, documenting model architecture and
development process, monitoring model performance after deployment [7, pp. 11-12]. Yes, compliance
is achieved by [7, p. 18]:
• quality control and origin of data used for model training to avoid violation of regulatory require-
ments;
• documenting and versioning models and data to ensure reproducibility of results and auditing;
• verifying models integrated into the production environment for compliance with requirements;
• continuous monitoring of deployed models for timely detection of potential violations or incorrect
behavior.
The leading tools for ensuring risk management and compliance practice are version control systems
for tracking changes in data and model code, testing tools for identifying problems in models, monitoring
systems for tracking model accuracy in real time [7, pp. 12, 14]. At the same time, interviews with
practitioners conducted by Steidl et al. [7] revealed that ensuring compliance of ML systems in various
domains (for example, in healthcare) is a serious challenge due to the lack of established methodologies
and software tools. At the same time, achieving compliance is a mandatory condition for obtaining the
necessary certifications and permits from regulators.
The diagram in figure 20 shows three main MLOps stages: working with training data, model training,
and system deployment (operationalization). At each stage there are blocks that indicate potential risks
and compliance measures. This diagram illustrates the cross-cutting nature of risk management and
compliance practice in MLOps, showing its presence and interconnections at each stage of the machine
learning model lifecycle.
Figure 21 shows the relationships between the key principles, deployment process, and main MLOps
practices that are applied at the stage of machine learning model deployment. The principles of
automation and reproducibility influence the model deployment process, which is associated with
the following MLOps practices: CI/CD, model deployment, data security and privacy, configuration
management, model deployment strategies, and infrastructure automation.
4. Conclusions
A meta-synthesis of systematic reviews [6, 8, 5] and a review of products and providers [15] was
performed in order to generalize knowledge about the implementation of MLOps practices for the
effective deployment of machine learning models. The main conclusions obtained as a result of the
meta-synthesis are as follows:
1. MLOps is an approach for managing, automating, and operationalizing the processes of developing,
deploying, and supporting machine learning models based on practices from software engineering
and DevOps. MLOps is based on a set of principles, processes, and practices that ensure effective
development, deployment, and support of machine learning models.
2. The main stages of the MLOps lifecycle include the following processes: data collection and
processing, model development and training, deployment, monitoring, and retraining of models.
3. Various frameworks and architectures are used to implement MLOps, such as open source
platforms (MLflow, Kubeflow, TensorFlow Extended), cloud computing platforms (AWS, Google
Cloud, Azure), containerization (Docker), and container orchestration (Kubernetes).
402
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
data model deployment
risks risks risks
data collection model training operationalization
data model deployment
compliance compliance compliance
Figure 20: Diagram of risk management and compliance practice in MLOps.
Automation Model deployment CI/CD
Model deployment
Data security and privacy
Configuration management
Reproducibility
Model deployment strategies
Infrastructure automation
Figure 21: Diagram of relationships between principles, processes and MLOps practices for model deployment.
4. MLOps tools provide a wide range of features to support the machine learning model lifecycle,
with a focus on automation, experiment tracking, versioning, monitoring, and model deployment.
5. The most common ways to deploy machine learning models in production environments are the
use of container technologies, cloud platforms and services, and the deployment of models as
web services.
403
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
6. Adapted software development maturity models, such as CMM, can be used to assess the maturity
level of MLOps processes in organizations.
7. Successful MLOps implementation requires the involvement of specialists from different ar-
eas - software development, data engineering, machine learning, subject matter experts, and
management.
8. The main challenges when deploying machine learning models in production environments are
managing the model lifecycle, ensuring scalability and performance, monitoring and maintaining
models in real-world conditions.
9. Open issues and challenges in MLOps are the need to develop standards and best practices, ensure
interpretability and responsible use of models, effectively manage data, integrate knowledge from
different fields.
10. The main opportunities and development trends of MLOps are the creation of standardized
platforms, application in the context of distributed learning, and integration with other approaches
to managing the lifecycle of data and models. Current and future areas of MLOps application
include a wide range of industries, from finance and healthcare to IoT and natural language
processing.
The conducted meta-synthesis showed that MLOps is a promising approach for the effective deploy-
ment of machine learning models in production environments, which requires further research and
development to address existing challenges and realize potential opportunities.
Next, the key MLOps practices necessary for effective deployment of machine learning models were
analyzed. The main conclusions obtained as a result of the analysis are as follows:
1. MLOps is based on a set of principles, processes and practices that ensure effective development,
deployment and maintenance of machine learning models. The key principles of MLOps are
automation, reproducibility, collaboration, continuous learning and data governance.
2. The main MLOps practices include: continuous integration and delivery (CI/CD), model and data
versioning, ML pipeline automation, model performance monitoring, experiment management,
model deployment and lifecycle management.
3. Additional MLOps practices, such as data security and privacy, model explainability and inter-
pretability, data quality management, configuration management, model deployment strategies,
infrastructure automation, collaboration and communication, risk management and compliance,
are important for ensuring reliability, compliance with requirements and efficiency of MLOps
processes.
4. The application of MLOps practices allows automating and standardizing the processes of devel-
opment, deployment and maintenance of machine learning models, which increases the efficiency
and reliability of ML solutions in the production environment.
5. Successful implementation of MLOps practices requires the use of appropriate tools and platforms,
such as experiment management systems, data and model versioning, infrastructure automation
and monitoring tools, as well as establishing effective collaboration between the different roles
and teams involved in the process of developing and implementing machine learning models.
The analysis showed that the application of MLOps practices is critically important for the successful
deployment of machine learning models in production environments. The implementation of these
practices allows increasing the efficiency, reliability and reproducibility of the processes of development
and operation of ML solutions, which is a necessary condition for their successful use in real business
problems.
404
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
As a result of the comprehensive study, the MLOps practices necessary for the effective deployment
of machine learning models were identified and analyzed. The main conclusions obtained during the
study are as follows:
1. A meta-synthesis of systematic reviews was performed to generalize knowledge about MLOps
practices. The conducted meta-synthesis showed that MLOps is a promising approach for the
effective deployment of machine learning models in production environments, which requires
further research and development to solve existing challenges and realize potential opportunities.
2. A diagram of relationships between MLOps principles, processes and practices is proposed.
This diagram illustrates the interconnections between the key principles, stages of the machine
learning model development and implementation process, as well as the main MLOps practices
that are applied at each stage.
3. The most effective MLOps practices for model deployment have been identified, which include:
continuous integration and delivery (CI/CD), model and data versioning, ML pipeline automa-
tion, model performance monitoring, experiment management, model deployment and lifecycle
management, data security and privacy, model explainability and interpretability, data quality
management, configuration management, model deployment strategies, infrastructure automa-
tion, collaboration and communication, risk management and compliance.
The obtained results have both theoretical and practical significance. The theoretical significance
lies in the generalization and systematization of knowledge about MLOps practices necessary for the
effective deployment of machine learning models. The practical significance of the obtained results
lies in the possibility of their use by organizations for the implementation or improvement of MLOps
processes in order to increase the efficiency and reliability of machine learning model deployment in
production environments.
Further research may be aimed at developing detailed recommendations for the implementation of
individual MLOps practices in organizations, creating new tools and platforms for automating and
managing the lifecycle of machine learning models, as well as studying the effectiveness of applying
MLOps practices in various industries and areas of machine learning model application.
Declaration on Generative AI: During the preparation of this work, the authors used Claude 3 Opus in order to: Text
Translation, Abstract drafting. After using this service, the authors reviewed and edited the content as needed and takes full
responsibility for the publication’s content.
References
[1] P. V. Zahorodko, S. O. Semerikov, V. N. Soloviev, A. M. Striuk, M. I. Striuk, H. M. Shalatska, Com-
parisons of performance between quantum-enhanced and classical machine learning algorithms
on the IBM Quantum Experience, Journal of Physics: Conference Series 1840 (2021) 012021.
doi:10.1088/1742-6596/1840/1/012021.
[2] D. Kreuzberger, N. Kühl, S. Hirschl, Machine Learning Operations (MLOps): Overview, Definition,
and Architecture, IEEE Access 11 (2023) 31866–31879. doi:10.1109/ACCESS.2023.3262138.
[3] G. Symeonidis, E. Nerantzis, A. Kazakis, G. A. Papakostas, MLOps - Definitions, Tools and
Challenges, in: 2022 IEEE 12th Annual Computing and Communication Workshop and Conference
(CCWC), 2022, pp. 0453–0460. doi:10.1109/CCWC54503.2022.9720902.
[4] M. Testi, M. Ballabio, E. Frontoni, G. Iannello, S. Moccia, P. Soda, G. Vessio, MLOps: A Taxonomy
and a Methodology, IEEE Access 10 (2022) 63606–63618. doi:10.1109/ACCESS.2022.3181730.
[5] J. Diaz-de Arcaya, A. I. Torre-Bastida, G. Zárate, R. Miñón, A. Almeida, A Joint Study of the
Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey, ACM
Comput. Surv. 56 (2023) 84. doi:10.1145/3625289.
405
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
[6] G. Recupito, F. Pecorelli, G. Catolino, S. Moreschini, D. D. Nucci, F. Palomba, D. A. Tamburri, A
Multivocal Literature Review of MLOps Tools and Features, in: 2022 48th Euromicro Conference
on Software Engineering and Advanced Applications (SEAA), 2022, pp. 84–91. doi:10.1109/
SEAA56994.2022.00021.
[7] M. Steidl, M. Felderer, R. Ramler, The pipeline for the continuous development of artificial
intelligence models—current state of research and practice, Journal of Systems and Software 199
(2023) 111615. doi:10.1016/j.jss.2023.111615.
[8] A. Lima, L. Monteiro, A. P. Furtado, MLOps: Practices, Maturity Models, Roles, Tools, and
Challenges – A Systematic Literature Review, in: Proceedings of the 24th International Conference
on Enterprise Information Systems - Volume 1: ICEIS, INSTICC, SciTePress, 2022, pp. 308–320.
doi:10.5220/0010997300003179.
[9] K. Haller, Managing AI in the enterprise: Succeeding with AI projects and MLOps to build
sustainable AI organizations, Apress Berkeley, CA, 2022. doi:10.1007/978-1-4842-7824-6.
[10] E. e Oliveira, M. Rodrigues, J. P. Pereira, A. M. Lopes, I. I. Mestric, S. Bjelogrlic, Unlabeled learning
algorithms and operations: overview and future trends in defense sector, Artificial Intelligence
Review 57 (2024) 66. doi:10.1007/s10462-023-10692-0.
[11] A. B. Kolltveit, J. Li, Operationalizing machine learning models: a systematic literature review,
in: Proceedings of the 1st Workshop on Software Engineering for Responsible AI, SE4RAI ’22,
Association for Computing Machinery, New York, NY, USA, 2023, p. 1–8. doi:10.1145/3526073.
3527584.
[12] F. Calefato, F. Lanubile, L. Quaranta, A Preliminary Investigation of MLOps Practices in GitHub, in:
Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering
and Measurement, ESEM ’22, Association for Computing Machinery, New York, NY, USA, 2022, p.
283–288. doi:10.1145/3544902.3546636.
[13] M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer,
J. M. Tetzlaff, E. A. Akl, S. E. Brennan, R. Chou, J. Glanville, J. M. Grimshaw, A. Hróbjartsson, M. M.
Lalu, T. Li, E. W. Loder, E. Mayo-Wilson, S. McDonald, L. A. McGuinness, L. A. Stewart, J. Thomas,
A. C. Tricco, V. A. Welch, P. Whiting, D. Moher, The PRISMA 2020 statement: an updated guideline
for reporting systematic reviews, BMJ 372 (2021) n71. doi:10.1136/bmj.n71.
[14] C. Haertel, D. Staegemann, C. Daase, M. Pohl, A. Nahhas, K. Turowski, MLOps in Data Science
Projects: A Review, in: 2023 IEEE International Conference on Big Data (BigData), 2023, pp.
2396–2404. doi:10.1109/BigData59044.2023.10386139.
[15] R. Cohen, Digital Strategy, Machine Learning, and Industry Survey of MLOps, in: Digital Strategies
and Organizational Transformation, 2023, pp. 137–150. URL: https://tinyurl.com/33z6zpd3. doi:10.
1142/9789811271984_0008.
[16] T. A. Sipe, W. L. Curlette, A meta-synthesis of factors related to educational achievement: a
methodological approach to summarizing and synthesizing meta-analyses, International Journal
of Educational Research 25 (1996) 583–698. doi:10.1016/S0883-0355(96)80001-2.
[17] J. Chrastina, Meta-synthesis of qualitative studies: background, methodology and applications, in:
NORDSCI Conference proceedings, volume 1 of NORDSCI Conference, Saima Consult Ltd, 2018.
doi:10.32008/nordsci2018/b1/v1/13.
[18] S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, T. Zimmermann,
Software Engineering for Machine Learning: A Case Study, in: 2019 IEEE/ACM 41st International
Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019, pp.
291–300. doi:10.1109/ICSE-SEIP.2019.00042.
[19] S. Dhanorkar, C. T. Wolf, K. Qian, A. Xu, L. Popa, Y. Li, Who needs to know what, when?:
Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI
Lifecycle, in: Proceedings of the 2021 ACM Designing Interactive Systems Conference, DIS ’21,
Association for Computing Machinery, New York, NY, USA, 2021, p. 1591–1602. doi:10.1145/
3461778.3462131.
[20] L. E. Lwakatare, I. Crnkovic, J. Bosch, DevOps for AI – Challenges in Development of AI-enabled
Applications, in: 2020 International Conference on Software, Telecommunications and Computer
406
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Networks (SoftCOM), 2020, pp. 1–6. doi:10.23919/SoftCOM50211.2020.9238323.
[21] R. Akkiraju, V. Sinha, A. Xu, J. Mahmud, P. Gundecha, Z. Liu, X. Liu, J. Schumacher, Characterizing
Machine Learning Processes: A Maturity Framework, in: D. Fahland, C. Ghidini, J. Becker,
M. Dumas (Eds.), Business Process Management, volume 12168 of Lecture Notes in Computer Science,
Springer International Publishing, Cham, 2020, pp. 17–31. doi:10.1007/978-3-030-58666-9_
2.
[22] C. Min, A. Mathur, U. G. Acer, A. Montanari, F. Kawsar, SensiX++: Bringing MLOps and Multi-
tenant Model Serving to Sensory Edge Devices, ACM Trans. Embed. Comput. Syst. 22 (2023) 98.
URL: https://doi.org/10.1145/3617507. doi:10.1145/3617507.
[23] F. Bachinger, J. Zenisek, M. Affenzeller, Automated Machine Learning for Industrial Applications
– Challenges and Opportunities, Procedia Computer Science 232 (2024) 1701–1710. doi:10.1016/
j.procs.2024.01.168.
[24] K. Filippou, G. Aifantis, G. A. Papakostas, G. E. Tsekouras, Structure Learning and Hyperparameter
Optimization Using an Automated Machine Learning (AutoML) Pipeline, Information 14 (2023)
232. doi:10.3390/info14040232.
[25] A. Bodor, M. Hnida, N. Daoudi, Machine Learning Models Monitoring in MLOps Context: Metrics
and Tools, International Journal of Interactive Mobile Technologies (iJIM) 17 (2023) pp. 125–139.
doi:10.3991/ijim.v17i23.43479.
[26] P. Singh, Systematic review of data-centric approaches in artificial intelligence and machine
learning, Data Science and Management 6 (2023) 144–157. doi:10.1016/j.dsm.2023.06.001.
[27] J. Czakon, K. Kluge, ML Experiment Tracking: What It Is, Why It Matters, and How to Implement
It, 2024. URL: https://neptune.ai/blog/ml-experiment-tracking.
[28] E. Peltonen, S. Dias, LinkEdge: Open-sourced MLOps Integration with IoT Edge, in: Proceedings of
the 3rd Eclipse Security, AI, Architecture and Modelling Conference on Cloud to Edge Continuum,
ESAAM ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 67–76. doi:10.
1145/3624486.3624496.
[29] L. A. Melgar, D. Dao, S. Gan, N. M. Gürel, N. Hollenstein, J. Jiang, B. Karlas, T. Lemmin, T. Li,
Y. Li, S. X. Rao, J. Rausch, C. Renggli, L. Rimanic, M. Weber, S. Zhang, Z. Zhao, K. Schawinski,
W. Wu, C. Zhang, Ease.ML: A Lifecycle Management System for MLDev and MLOps, in: 11th
Conference on Innovative Data Systems Research, CIDR 2021, Virtual Event, January 11-15, 2021,
Online Proceedings, 2021. URL: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper26.pdf.
[30] H. Chen, M. A. Babar, Security for Machine Learning-based Software Systems: A Survey of Threats,
Practices, and Challenges, ACM Comput. Surv. 56 (2024) 151. doi:10.1145/3638531.
[31] N. K. Gopalakrishna, D. Anandayuvaraj, A. Detti, F. L. Bland, S. Rahaman, J. C. Davis, “If security
is required”: engineering and security practices for machine learning-based IoT devices, in:
Proceedings of the 4th International Workshop on Software Engineering Research and Practice
for the IoT, SERP4IoT ’22, Association for Computing Machinery, New York, NY, USA, 2023, p.
1–8. doi:10.1145/3528227.3528565.
[32] T. Miller, Explanation in artificial intelligence: Insights from the social sciences, Artificial
Intelligence 267 (2019) 1–38. doi:10.1016/j.artint.2018.07.007.
[33] F. Rezazadeh, H. Chergui, L. Alonso, C. Verikoukis, SliceOps: Explainable MLOps for Streamlined
Automation-Native 6G Networks, IEEE Wireless Communications 31 (2024) 224–230. doi:10.
1109/MWC.007.2300144.
[34] R. C. Godwin, R. L. Melvin, Toward efficient data science: A comprehensive MLOps template for
collaborative code development and automation, SoftwareX 26 (2024) 101723. doi:10.1016/j.
softx.2024.101723.
[35] D. Yongqiang, W. Xin, L. Yongbo, Y. Wang, Building Network Domain Knowledge Graph from
Heterogeneous YANG Models, Journal of Computer Research and Development 57 (2020) 699–708.
doi:10.7544/issn1000-1239.2020.20190882.
[36] Neptune Labs, MLOps Landscape in 2024: Top Tools and Platforms, 2024. URL: https://neptune.ai/
blog/mlops-tools-platforms-landscape.
[37] A. Gunny, D. Rankin, P. Harris, E. Katsavounidis, E. Marx, M. Saleem, M. Coughlin, W. Benoit, A
407
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Software Ecosystem for Deploying Deep Learning in Gravitational Wave Physics, in: Proceedings
of the 12th Workshop on AI and Scientific Computing at Scale Using Flexible Computing Infras-
tructures, FlexScience ’22, Association for Computing Machinery, New York, NY, USA, 2022, p.
9–17. doi:10.1145/3526058.3535454.
[38] C. Vuppalapati, A. Ilapakurti, K. Chillara, S. Kedari, V. Mamidi, Automating Tiny ML Intelligent
Sensors DevOPS Using Microsoft Azure, in: 2020 IEEE International Conference on Big Data (Big
Data), 2020, pp. 2375–2384. doi:10.1109/BigData50022.2020.9377755.
[39] R. Sothilingam, V. Pant, E. S. K. Yu, Using i* to Analyze Collaboration Challenges in MLOps Project
Teams, in: A. Maté, T. Li, E. J. T. Gonçalves (Eds.), Proceedings of the 15th International iStar
Workshop (iStar 2022) co-located with 41th International Conference on Conceptual Modeling
(ER 2022), Virtual Event, Hyderabad, India, October 17, 2022, volume 3231 of CEUR Workshop
Proceedings, CEUR-WS.org, 2022, pp. 1–6. URL: https://ceur-ws.org/Vol-3231/iStar22_paper_1.pdf.
A. Using the large language model Claude 3 Sonnet for analyzing
systematic reviews
The queries were created on 10.05.2024. A PDF file with the article text was added to each query.
A report according to the submitted plan was expected from the chatbot. The queries consisted of
universal and variable parts.
Universal part of the query:
Using the added review article file as a data source,
write a detailed report on it according to the following plan.
Variable parts of the query:
1. 1. Year of publication.
2. Research objective (paper aim).
3. Research questions.
2. 4. Information sources (databases, etc.)
5. Inclusion criteria.
6. Exclusion criteria.
7. Quality criteria.
3. 8. MLOps definition (if any).
9. MLOps workflow stages (if any).
4. 10. What frameworks and architectures facilitate MLOps?
5. 11. What MLOps tools can be used to build ML pipelines for
Continuous Deployment? What tools are used in the
activities for operationalizing machine learning models?
6. 12. What are the main features offered by MLOps tools?
7. 13. How are machine learning models deployed in production
environments?
8. 14. What maturity models are used to assess the level of
automation in deploying machine learning models?
9. 15. What roles and responsibilities are identified in the
activities of operationalization of machine learning models?
408
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
10. 16. What challenges are encountered with regard to
deploying machine learning models in production
environments?
11. 17. What are the open issues, challenges and
particularities in MLOps?
18. What are the opportunities and future trends in MLOps?
What are the current and future fields in which MLOps is
thriving?
B. Results of systematic reviews analysis
Table B.1
Results of systematic review analysis [6, 8, 5].
Comparison Review by Review by Review by
object Recupito et al. [6] Lima et al. [8] Diaz-de Arcaya et al. [5]
Year of publi- 2022 2022 2023
cation
Research To identify tools that allow cre- Review of existing literature The main objective of this sys-
objective ating MLOps pipelines for con- to identify practices, standards, tematic literature review is to
tinuous deployment. roles, maturity models, chal- provide an understanding of the
To analyze the main character- lenges and tools used to auto- implementation of MLOps and
istics and functions of these mate activities of operational- AIOps methodologies in both in-
MLOps tools to provide a com- izing machine learning mod- dustry and academia. The au-
prehensive overview of their els into industrial operation thors seek to highlight the chal-
value. (MLOps). lenges, opportunities and future
trends in these areas.
Research ques- We answer the following main RQ1: How are machine learning Q1: What are the open issues,
tions research question: What tools models deployed in production challenges and particularities in
and capabilities enable develop- environments? MLOps and AIOps?
ers to create ML-enabled soft- RQ2: What maturity models are Q2: What are the opportunities
ware systems? used to assess the level of au- and future trends in MLOps?
Which was detailed into two tomation in deploying machine Q3: What are the opportunities
sub-questions: learning models? and future trends in AIOps?
RQ1. What MLOps tools can be RQ3: What roles and responsi- Q4: What platforms and archi-
used to create machine learning bilities are defined in the activi- tectures facilitate MLOps and
pipelines for continuous deploy- ties of operationalizing machine AIOps?
ment? learning models? Q5: What are the current and fu-
RQ2. What are the main fea- RQ4: What tools are used in ture fields in which MLOps and
tures offered by MLOps tools? the activities of operationaliz- AIOps are thriving?
ing machine learning models?
RQ5: What challenges are en-
countered when deploying ma-
chine learning models in pro-
duction environments?
Continued on next page
409
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Continuation of table B.1
Comparison Review by Review by Review by
object Recupito et al. [6] Lima et al. [8] Diaz-de Arcaya et al. [5]
Information Google Scholar – for searching The automated search was To search for relevant arti-
sources scientific literature, such as jour- conducted in the following cles, the authors used several
nals, books and dissertations. electronic research databases: databases and repositories, in-
Google Search – for searching ACM Digital Library, IEEE cluding arXiv, Springer, IEEE.
so-called “gray” literature, such Xplore, ScienceDirect and However, the main source was
as blog posts, developer sites, SpringerLink. the Scopus database from Else-
webinars, GitHub repositories vier, as it contains metadata and
and YouTube videos. abstracts of many publications.
Using both academic (white lit-
erature) and professional (gray
literature) sources allowed the
authors to explore MLOps from
different perspectives – theoret-
ical and practical.
Inclusion crite- The study discusses compo- Studies related to machine Articles published between
ria nents of a minimal end-to-end learning operations (MLOps) in 2018 and 2023, identified by
MLOps workflow. general. search queries, containing new
The study discusses MLOps Studies evaluating the lifecycle ideas and closely related to
practice or machine learning- of machine learning solutions. the topic of MLOps and AIOps
based applications. Studies related to maturity were included in the analysis.
The study relates to the imple- models of the machine learning
mentation of MLOps tool(s). process.
The study describes experi- Studies analyzing roles and
ences, opinions, or practices re- responsibilities involved in the
garding MLOps pipelines. development and deployment
of machine learning solutions.
Studies covering tools for
deploying machine learning
solutions.
Studies identifying challenges
for the development and de-
ployment of machine learning
models.
Exclusion cri- The study does not provide de- Studies not published in En- Publications not in English, re-
teria tails on the design or implemen- glish. tracted publications, publica-
tation of MLOps tool(s). Studies related to the applica- tions from irrelevant publishers,
The study only proposes the de- tion of machine learning mod- subscription materials without
sign of a certain component of els. access, and articles with insuf-
machine learning pipelines. Short papers or posters. ficient citations (depending on
The study does not provide or Studies not related to machine the year of publication) were ex-
reference details on machine learning operations. cluded.
learning automation. Studies whose content is not ac-
The study refers to commercial cessible.
platforms that offer MLOps ap- Articles not relevant to the re-
plications to promote their de- search questions.
velopment and deployment ser-
vices.
Continued on next page
410
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Continuation of table B.1
Comparison Review by Review by Review by
object Recupito et al. [6] Lima et al. [8] Diaz-de Arcaya et al. [5]
Quality crite- The repository must have at Does the study report unam- Comprehensive literature
ria least 100 stars biguous findings based on ev- review and gap identification
The YouTube video must have idence and arguments? Verification of results on a use
been viewed at least 1000 times Does the study represent a re- case
search project and not an expert Number of research questions
opinion? addressed
Is the context being analyzed Publication under an open
fully described in the study? license
Are the research objectives Type of publication (jour-
clearly defined? nal/other)
Are the research results prop- Publication in a high-impact
erly validated? journal
Number of citations
MLOps defini- MLOps is a practice that helps A set of practices and princi- MLOps uses machine learning,
tion (if avail- model, develop and operate ples for operationalizing data DevOps and data engineering to
able) the machine learning lifecycle, science solutions, used to au- bring machine learning systems
drawing on DevOps principles tomate the deployment of ma- into production, facilitating the
and practices. chine learning models into an creation of machine products.
operational environment
MLOps work- Data extraction for integration Data collection Data management
flow stages (if Data analysis Data transformation Distributed training
available) Data cleaning, transformation Continuous training Deployment
and feature engineering to split Continuous model deployment Monitoring
data Presentation of results Retraining
Model training Monitoring of machine learning The need to manage the lifecy-
Model validation to assess the solutions cle and key components of AI
quality applications using specialized
Model deployment in target en- platforms and tools is empha-
vironments sized.
Model monitoring
Frameworks Continuous training pipelines MLflow HPC
and archi- deployed via CI/CD Kubeflow Cloud computing
tectures that Orchestration platforms Polyaxon Edge/IoT platforms
facilitate TensorFlow Extended (TFX) Comet.ml Serverless architectures
MLOps imple- Machine learning cloud plat- Kafka-ML Frameworks for integration
mentation forms MLModelCI Automatic data labeling meth-
ods
Proactive incident management
Platforms for orchestration
Semantically enhanced
pipelines
Architectures for distributed
training and deployment
Frameworks for monitoring
Programming languages
Containerized solutions
AutoML software
Use of APIs
Deep learning and neural
networks
Continued on next page
411
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Continuation of table B.1
Comparison Review by Review by Review by
object Recupito et al. [6] Lima et al. [8] Diaz-de Arcaya et al. [5]
MLOps tools Machine learning cloud MLflow - this open platform Containerized solutions (e.g.,
for creating platforms (AWS SageMaker, has components such as MLflow Docker)
machine learn- AzureML, Google AI Platform) Projects and MLflow Model Reg- Serverless computing (AWS
ing pipelines Orchestration platforms istry Lambda, Azure Functions, etc.)
and opera- (Apache Airflow, Jenkins, Kubeflow Continuous integra-
tionalizing Kubeflow, MLflow, Polyaxon, Kafka-ML tion/continuous deployment
models Seldon Core, Valohai) MLModelCI (CI/CD) tools
Configuration frameworks Monitoring of processes and
(TensorFlow Extended, Gitlab) events
MLOps automation using
AutoML
Containerization for packaging
model dependencies.
API technologies for deploy-
ment as a web service.
Main features Common features related to Experiment tracking Data, model, and code version-
offered by all phases of machine learning Model packaging and version- ing
MLOps tools pipelines ing Workflow orchestration and au-
Data management features ML project management tomation
Model management features Model registry Monitoring of processes and
Continuous integration and de- events
livery Integration with cloud and edge
Model monitoring infrastructure
Hyperparameter tuning Containerization
Portability MLOps automation using Au-
toML
Methods of Machine learning cloud MLOps is considered as a set of Deployment in the cloud Using
deploying platforms (AWS SageMaker, practices and principles for op- cloud computing resources
machine learn- AzureML, DotScience, Google erationalizing Providing isolation
ing models in AI Platform) Some MLOps tools, such as Hybrid approaches
production Orchestration platforms MLflow, Kubeflow, and Kafka- Deployment on edge/IoT de-
environments (Apache Airflow, Jenkins, ML vices
Kubeflow, MLflow, Polyaxon, The role of “Deployment Lead” Deployment directly on IoT de-
Seldon Core, Valohai) vices and mobile devices.
TensorFlow Extended (TFX) TensorFlow Lite and Core ML.
Overcoming limitations
Containerized deployments
Packaging models
Docker.
Serverless architectures
Deploying ML functions as a
service
Reducing costs
Deployment via APIs
Continued on next page
412
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Continuation of table B.1
Comparison Review by Review by Review by
object Recupito et al. [6] Lima et al. [8] Diaz-de Arcaya et al. [5]
Maturity mod- Support for continuous inte- Maturity model proposed by The article does not define spe-
els for assess- gration and continuous deploy- Amershi et al. [18] cific maturity models for assess-
ing the level of ment (CI/CD) Dhanorkar et al. [19] classify or- ing the automation of machine
automation in The ability to automatically ganizations into three levels of learning model deployment, but
deploying ma- tune maturity emphasizes the importance of
chine learning Full automation of model man- Lwakatare et al. [20] describe adapting software development
models agement processes five stages of improvement in practices to the field of machine
development practices learning.
Akkiraju et al. [21] propose an
adaptation of the Capability Ma-
turity Model (CMM)
Roles and re- Data scientists – responsible for Domain specialist – has deep Software developers
sponsibilities developing and training knowledge of the subject area Data specialists/data scientists
identified in Data engineers – responsible Computational scien- Operations engineers
the activities for extracting, processing, trans- tist/engineer – has high Domain experts
of operational- forming, and ensuring the qual- technical skills Management/stakeholders
izing machine ity of data ML scientist/engineer – respon-
learning mod- DevOps engineers – responsi- sible for designing
els ble for automating deployment Provenance specialist – man-
processes and managing opera- ages the supply of data
tional environments Manager – evaluates models
Product managers and business Application developer – devel-
stakeholders – provide require- ops applications
ments for models and partici- Deployment lead – assesses
pate in decision-making aspects
Challenges Complexity of managing Integration of software devel- Managing the ML lifecycle
encountered Ensuring consistency opment Gap between software engineer-
when deploy- Integration of various tools Implementation of ing
ing machine Automation of all stages MLOps/AIOps practices Data management
learning Monitoring model performance Machine learning models need Distributed and parallel execu-
models in Scaling infrastructure monitoring tion
production Ensuring security Identifying infrastructure Diversity of computing infras-
environments components tructure
Deploying and versioning Monitoring
Explainability
Challenges Lack of standardization Integration of software develop- Lack of skilled personnel
encountered Ensuring portability ment processes Data management issues
when deploy- Configuration and integration Implementation of MLOps prac- Complexity of orchestration
ing machine Full automation tices Heterogeneity of hardware
learning Understandability It is necessary to go beyond an- Need for continuous monitor-
models in alyzing model prototypes ing
production Careful monitoring Lack of explainability
environments Determining infrastructure Scaling and performance issues
Addressing scalability
Need for versioning
Automation
Managing the lifecycle
Integration with DevOps
Continued on next page
413
Danylo O. Hanchuk et al. CEUR Workshop Proceedings 362–414
Continuation of table B.1
Comparison Review by Review by Review by
object Recupito et al. [6] Lima et al. [8] Diaz-de Arcaya et al. [5]
Opportunities, Standardization of MLOps Demand for MLOps tools and Involvement of business units
future trends, practices and tools platforms is expected to grow Greater attention to the ML
and areas of Improved support environ- Industries where MLOps is lifecycle
application of ments. actively developing – finance, Better data management prac-
MLOps Advances in automation and healthcare, industry, retail, tices
Integration of MLOps with transportation, and logistics. Use of new hardware platforms
DevOps and DevSecOps. Possible development of spe- Use of containers, serverless
Increased attention to data cialized technologies
management Integration of MLOps with De- Development of versioning
Areas of application vSecOps, MLSecOps concepts tools
Financial services and banking Emergence of new roles and Industries where MLOps is
Healthcare and biotechnology competencies thriving:
Manufacturing and Internet of Traditional industries
Things Innovative industries
Retail and e-commerce Academic disciplines
Telecommunications Communication and network-
Transportation and logistics ing technologies
Healthcare
Scientific activity
414