Automating the end-to-end lifecycle of machine learning models is critical for their efective operationalization. Various tools, frameworks and architectures have emerged to support Machine Learning Operations (MLOps) practices. This paper presents a meta-synthesis of existing reviews to provide a comprehensive overview of such enabling technologies for MLOps. The capabilities and features ofered by common commercial and open-source MLOps platforms are compared. Patterns in the MLOps architecture and design philosophies are identified. The role of containers, orchestration, configuration management, and infrastructure automation in ML pipelines is examined. Approaches for model deployment on cloud and edge are also discussed. The synthesis ofers insights for tool selection and usage to automate enterprise-scale machine learning.
In the modern world, machine learning is becoming an increasingly important technology that finds
application in various fields such as finance, healthcare, industry, retail, etc. [
To solve these problems, the MLOps (Machine Learning Operations) methodology has emerged, which
aims to apply the principles and practices of DevOps to the development and deployment processes
of machine learning models [
At the same time, despite the significant interest in the topic of MLOps from both scientists and
practitioners, there are still certain gaps and unresolved problems in this area. In particular, there are no
generally accepted standards and best practices for implementing MLOps, issues of integrating MLOps
with other approaches (DataOps, ModelOps, AIOps, etc.) are insuficiently researched, and there is a
need to develop new tools and platforms to automate MLOps processes [
This work is aimed at solving the current problem of defining and analysing MLOps practices necessary for the efective deployment of machine learning models. The basis for performing the work is the need to systematize and generalize knowledge about MLOps practices, as well as the need to develop recommendations for their implementation in organizations to increase the eficiency and reliability of deploying machine learning models in production environments.
According to the aim, the following main objectives of the study are defined: 1. Perform a meta-synthesis of systematic reviews to generalize knowledge about MLOps practices necessary for the efective deployment of machine learning models. 2. Analyze the relationships between MLOps principles, processes, and practices. 3. Identify the most efective MLOps practices for deploying machine learning models.
DevOps (Development & Operations) is becoming increasingly widespread, and companies are
applying its methods in various fields [
According to Calefato et al. [
Systematic reviews can provide syntheses of the state of knowledge in a field, from which future research priorities can be identified; they can address questions that otherwise could not be answered by individual studies; they can identify problems in primary research that should be rectified in future studies; and they can generate or evaluate theories about how or why phenomena occur [13, p. 1]. The main aim of a systematic review is facilitating evidence-based decision-making [13, p. 6]. The main diference between systematic and literature reviews is the “place of idea”. The literature review can be idea-driven; thus, all sources can be selected to confirm some idea. Indeed, the systematic review is the scientific method; instead of ideas, we operate with research questions and hypotheses. As a result, the systematic review can produce new evidence-based knowledge.
On 23 February 2024, we proceeded with a search request to the Scopus database by article title:
TITLE ( ( systematic OR review OR survey ) AND mlops )
5 documents were found (table 1), of which 3 [
R. Cohen, Digital Strategy, Machine As part of a digital strategy, machine learning (ML) has become a
comLearning, and Industry Survey of MLOps, mon toolset and capability across many businesses. However, the
opin: Digital Strategies and Organizational erational aspects of machine learning (MLOps) are often overlooked
Transformation, 2023, pp. 137–150. for ML projects until they are already installed and being executed in
URL: https://tinyurl.com/33z6zpd3. the business environment. This chapter provides a review of MLOps
doi:10.1142/9789811271984_0008. [
J. Diaz-de Arcaya, A. I. Torre-Bastida, G. Diaz-de Arcaya et al. [
MLOps is more common in traditional industrial environments.
The papers by Haertel et al. [
The meta-synthesis was performed according to Chrastina [17, pp. 123-125]:
1. Defining the subject of research : MLOps practices for the efective deployment of models.
2. Identifying relevant sources: systematic literature reviews [
7. Publication of the meta-synthesis.
2.3.1. Distribution of reviews by year
Two works [
Common aspects of the objectives of the considered reviews are:
1. All reviews [
Distinct aspects of the review objectives are:
1. The review [
Thus, despite some diferences in focus and breadth of coverage, all the considered reviews are
united by a common goal – to investigate and generalize knowledge about the MLOps methodology, its
practical application, tools, challenges, and state of implementation to promote further development of
this area.
2.3.3. Review research questions
The aim of the meta-synthesis of the research questions of the reviews [
Common aspects of the research questions of the considered reviews are:
1. All reviews [
MLOps tools for creating machine learning pipelines.
2. The review [
Thus, the research questions of the considered reviews cover a wide range of MLOps aspects, from
tools and platforms to challenges, opportunities, and areas of application. Despite some diferences in
the focus of the questions, all reviews seek to explore the key components and factors that influence
the implementation and development of MLOps practices in organizations.
2.3.4. Review information sources
The aim of the meta-synthesis of the information sources of the reviews [
Common aspects of information sources of the considered reviews are:
1. All reviews [
Distinct aspects of information sources of the reviews are:
1. The review [
Xplore, ScienceDirect, and SpringerLink.
3. The review [
Thus, the considered reviews demonstrate diferent approaches to the selection of information
sources. Some studies [
Common aspects of the inclusion criteria in the considered reviews are:
1. All reviews [
Distinct aspects of the inclusion criteria in the reviews are:
1. The review [
Thus, despite some diferences, the inclusion criteria in the considered reviews mainly focus on
studies that directly relate to MLOps, describe practical experience, tools, and processes, and consider
various aspects of the ML model development lifecycle. The reviews [
Common aspects of the exclusion criteria in the considered reviews are:
1. The reviews [
Distinct aspects of the exclusion criteria for information sources from the reviews are:
1. The review [
Thus, the considered reviews apply diferent exclusion criteria to filter out studies that do not meet
their requirements. The common aspect is the exclusion of studies with insuficient descriptions of
MLOps processes and tools, as well as non-English publications. Diferences lie in additional criteria,
such as the exclusion of commercial platforms without technical details [
Common aspects of the quality criteria in the considered reviews are:
1. The reviews [
Distinct aspects of the quality criteria in the reviews are:
1. The review [
YouTube) to assess the quality and relevance of “grey” literature.
2. The review [
Thus, the considered reviews apply diferent approaches to assessing the quality of studies. The
common aspect is the desire to include studies with a complete description of the methodology and
substantiated results. However, the specific quality criteria difer: from the use of popularity indicators
for “grey” literature [
For the mutual translation of the results from diferent works, in addition to systematic reviews [
Common aspects of MLOps definitions in the considered reviews are:
1. All reviews [
Distinct aspects of MLOps definitions in the reviews are:
1. The review [
MLOps to create model-based products.
4. The review [
Thus, despite some diferences in emphasis and wording, all the considered reviews define MLOps
as an approach for managing, automating, and operationalizing the processes of developing,
deploying, and supporting machine learning models based on practices from software
engineering and DevOps. MLOps is a key component for the successful implementation of machine
learning solutions in an industrial environment.
2.4.2. Stages of the MLOps workflow
The aim of the meta-synthesis of the stages of the MLOps workflow in the reviews [
Common stages of the MLOps workflow in the considered reviews are:
1. All reviews [
Distinct aspects of the MLOps workflow stages in the reviews are:
1. The review [
Thus, despite diferent levels of detail and grouping, the considered reviews demonstrate general
consistency regarding the main stages of the MLOps workflow. These stages cover the entire lifecycle
of machine learning models, from data collection and processing to deployment, monitoring, and
retraining of models. Diferences in the presentation of stages reflect diferent approaches to structuring
and describing the MLOps workflow.
2.4.3. Frameworks and architectures that facilitate MLOps implementation
The aim of the meta-synthesis of frameworks and architectures that facilitate MLOps implementation
in the reviews [
Common frameworks and architectures that facilitate MLOps implementation, according to the
considered reviews, are:
1. The reviews [
Distinct aspects of the considered MLOps frameworks and architectures in the reviews are:
1. The review [
Airflow, Jenkins, and Polyaxon.
2. The review [
MLModelCI for managing pipelines and deploying models.
3. The review [
Thus, there are many frameworks and architectural approaches that facilitate MLOps implementation,
from open platforms and libraries to commercial solutions and cloud services. Key factors are support
for automation, scalability, portability, and integration with existing systems and tools. The choice of
appropriate frameworks and architectures depends on the specific requirements and constraints of the
organization, as well as the level of maturity of its MLOps processes.
2.4.4. MLOps tools for creating machine learning pipelines and operationalizing models
The aim of the meta-synthesis of MLOps tools for creating machine learning pipelines and
operationalizing models in the reviews [
Common MLOps tools mentioned in the considered reviews are:
1. The reviews [
Google Cloud AI Platform, Azure Machine Learning, as tools for operationalizing models.
3. The reviews [
Distinct aspects of the considered MLOps tools in the reviews are:
1. The review [
MLModelCI for managing pipelines and deploying models.
3. The review [
Domino Data Lab, Comet, Valohai, etc. (table 2)
Thus, there is a wide range of MLOps tools for creating machine learning pipelines and
operationalizing models, from open platforms such as MLflow, to commercial solutions from cloud providers and
specialized companies. The choice of specific tools depends on the needs and scale of the organization,
as well as compatibility with the existing technology stack.
2.4.5. Main features ofered by MLOps tools
The aim of the meta-synthesis of the main features ofered by MLOps tools in the reviews [
Common features of MLOps tools, highlighted in the considered reviews, are:
1. The reviews [
MLOps workflows, such as model training and deployment pipelines.
3. The reviews [
Distinct aspects of MLOps tool features, considered in the reviews, are:
1. The review [
• support for various machine learning libraries and frameworks;
2. The review [
Thus, MLOps tools provide a wide range of features to support the lifecycle of machine learning
models, with a focus on automation, experiment tracking, versioning, monitoring, and model deployment.
Some tools ofer more specialized features, such as hyperparameter optimization or data management.
The choice of a tool with an appropriate set of features depends on the specific needs and goals of the
organization in the field of MLOps.
2.4.6. Ways of deploying machine learning models in production environments
The aim of the meta-synthesis of ways of deploying machine learning models in production
environments in the reviews [
Common ways of deploying machine learning models in production environments, according to the
considered reviews, are:
1. The reviews [
Docker, which provides model mobility and isolation.
2. The reviews [
Distinct aspects of the considered ways of deploying models in the reviews are:
1. The review [
Thus, the most common ways to deploy machine learning models in production environments are
the use of container technologies, cloud platforms and services, and the deployment of models as web
services. The choice of a specific approach depends on the requirements for latency, scalability, and
availability of models, as well as the existing infrastructure and ecosystem of tools in the organization.
2.4.7. Maturity models for assessing the level of automation in deploying machine learning
models
Lima et al. [
This model, based on the Capability Maturity Model (CMM) and Six Sigma methodology, checks
whether the activity: (1) has defined goals, (2) is consistently implemented, (3) is documented, (4)
is automated, (5) is measured and tracked, and (6) is continuously improved.
2. According to Dhanorkar et al. [
All systematic reviews [
Common roles and responsibilities identified in the considered reviews:
1. All reviews [
Distinct aspects of roles and responsibilities, considered in the reviews:
1. The review [
Thus, despite some diferences in the detail of roles, the considered reviews recognize the need to
involve specialists from diferent areas – software development, data engineering, machine learning,
subject matter experts, and management, for the successful operationalization of machine learning
models. Close collaboration and communication between these roles is critical for implementing MLOps
practices in organizations (figure 1).
2.4.9. Challenges encountered in deploying machine learning models in production
environments
The aim of the meta-synthesis of challenges encountered in deploying machine learning models in
production environments in the reviews [
Common challenges identified in the considered reviews:
1. The reviews [
DS Data Scientist (ML model development)
BE (ML infrastructure management)
ML
ML Engineer / MLOps Engineer (cross-functional management of ML environment and assets:
ML infrastructure,
ML models, ML workflow pipelines, data Ingestion, monitoring)
SE
{…} (applies design patterns and
coding guidelines)
DE Data Engineer
(data management, data pipeline management)
DO (Software engineer with DevOps skills,
ML workflow pipeline orchestration,
CI/CD pipeline management,
monitoring)
FFiigguurer1e: I3nt.ersRectoiolneosf raolens dandtrhesepoinrsibiinlittieesr(ascceorcdtinigoton[s2, pc. o5]n). tributing to the
MLOps paradigm
2. The reviews [
Amarchcinhelietarenicngtpuipreleine.and Workflow
On 4t.hTehebraevsiieswso[f6,t5h]ecoindsiednerticfhiaelledngpesrirnelcatiepdlteose,nscuorimng pdaotanseencutrsit,yaannddprriovalecys,wwhene
deploying machine learning models.
derive a generalized MLOps end-to-end architecture to give ML
5. The reviews [
Additionally, and business stakeholders. in which the different tasks are executed in the different stages. The
Distinct challenges, considered in the reviews:
artifact was designed to be technology-agnostic. Therefore, ML
1. The review [
The considered reviews show that deploying machine learning models in production environments
is associated with a number of challenges, such as managing the model lifecycle, ensuring scalability
and performance, monitoring and maintaining models in real-world conditions. Addressing these
challenges requires an integrated approach that includes automation of MLOps processes, selection
of appropriate infrastructure, ensuring data security and privacy, and efective communication with
business stakeholders.
2.4.10. Open issues, challenges, and peculiarities of MLOps
The aim of the meta-synthesis of open issues, challenges, and peculiarities of MLOps in the reviews
[
Common open issues and challenges of MLOps, identified in the considered reviews:
1. The reviews [
Peculiarities of MLOps, identified in the considered reviews:
1. The review [
Thus, the considered reviews identify a number of open issues and challenges in MLOps, such as the
need to develop standards and best practices, ensure interpretability and responsible use of models,
and efectively manage data. Peculiarities of MLOps, such as the diference from traditional software
development, the importance of the human factor, and the need to integrate knowledge from diferent
ifelds, require consideration when implementing MLOps practices in organizations.
2.4.11. Opportunities, future trends, and areas of application of MLOps
The aim of the meta-synthesis of opportunities, future trends, and areas of application of MLOps in
the reviews [
Opportunities and future trends of MLOps, identified in the considered reviews:
1. The reviews [
Current and future areas of MLOps application, identified in the considered reviews:
1. The reviews [
Thus, the considered reviews outline a number of opportunities and development trends for MLOps, such as creating standardized platforms, applying in the context of distributed learning, and integrating with other approaches to managing the lifecycle of data and models. Current and future areas of MLOps application include a wide range of industries, from finance and healthcare to IoT and natural language processing, which indicates a significant potential impact of this approach.
MLOps is based on a set of principles [2, p. 3] and processes that ensure efective development, deployment, and support of machine learning models (MLOps practices).
MLOps principles define the fundamental foundations of designing machine learning pipelines: • automation: maximum automation of all stages of the machine learning model lifecycle to reduce
manual interventions and improve eficiency; • reproducibility: ensuring the ability to reproduce the results of experiments and model deployment
processes; • collaboration: establishing efective collaboration and communication between diferent teams involved in model development and implementation; • continuous learning and improvement: regular updating of models based on new data and feedback,
continuous improvement of MLOps processes; • data governance: ensuring data quality, security, and confidentiality throughout the model lifecy
cle.
MLOps processes define the sequence of actions for designing and implementing machine learning pipelines: 1. Defining business goals and requirements : aligning the goals of developing machine learning
models with the business strategy of the organization. 2. Data collection and preparation: collecting, cleaning, transforming, and enriching data for model
training. 3. Model development and training: selecting algorithms, developing model architecture, training
and validating models. 4. Model evaluation and testing: evaluating model performance on test data, conducting tests for
reliability, security, and compliance with requirements. 5. Model deployment: packaging models with necessary dependencies, deploying in target environ
ments. 6. Model monitoring and maintenance: tracking model performance, identifying and resolving issues,
updating models as needed. 7. Model lifecycle management: coordinating all stages of model development, deployment, and
support, ensuring compliance with regulatory requirements.
MLOps practices define the most efective methods and technologies for implementing machine learning pipelines: • basic MLOps practices include: – continuous integration and delivery (CI/CD): automation of the processes of building, testing,
and deploying machine learning models; – model and data versioning: tracking changes in models and datasets, ensuring reproducibility
of results; – ML pipeline automation: creating automated workflows for data collection, processing,
model training, and evaluation; – model performance monitoring: tracking model quality metrics in the production environ
ment, detecting performance degradation; – experiment management: organizing, tracking, and comparing diferent experiments with
models and hyperparameters; – model deployment: packaging models with necessary dependencies, deploying in diferent
environments (cloud, edge, etc.); – model lifecycle management: coordinating the processes of model development, testing,
deployment, and monitoring to best ensure compliance with requirements; • additional MLOps practices include: – data security and privacy: ensuring the protection of data used for model training, compliance with regulatory requirements; – model explainability and interpretability: using methods and tools to understand and explain
model behavior, especially in regulated industries; – data quality management: monitoring and ensuring the quality of data used for model
training and evaluation, detecting and handling anomalies; – configuration management : versioning and managing configurations of environments where
models are deployed, ensuring consistency across diferent environments; – model deployment strategies: selecting and implementing appropriate deployment strategies; – infrastructure automation: using Infrastructure as Code to automate provisioning and man
agement of infrastructure for model training and deployment; – collaboration and communication: establishing efective collaboration between data science,
development, operations, and business units; – risk management and compliance: identifying and mitigating risks associated with the use
of machine learning models, ensuring compliance with regulatory requirements.
Figure 2 illustrates the relationships between key MLOps principles (light blue rectangles), main processes (green rectangles), and common practices (orange rectangles). Arrows show how principles influence processes, and processes, in turn, are implemented through specific practices. For example, the principle of automation influences all MLOps processes, from goal definition to model lifecycle management. The model development process is associated with practices such as versioning, pipeline automation, experiment management, and model interpretability. 3.2. CI/CD CI/CD (Continuous Integration/Continuous Delivery) is a key element/practice/implementation of DevOps for automatic testing and deployment of code, data and models in a production environment (figure 3) [ 7, p. 7]. In MLOps, it is extended to automate the process of developing and deploying ML models, including the stages of building, testing, delivery, and deployment [2, pp. 3-4].
The CI/CD process in MLOps includes the stages of build, test, delivery, and deploy [2, p. 4]. However, unlike traditional CI/CD, MLOps may also have additional stages, such as model retraining.
CI/CD in MLOps is part of the MLOps system architecture and provides fast feedback to developers on the success or failure of certain stages, increasing overall productivity [2, pp. 3-4].
Typical triggers for starting the CI/CD process in MLOps on GitHub are git push and pull_request events [12, p. 4]. The events issue_comment, release, and schedule (on a schedule) can also be used.
In the article Steidl et al. [
A feature of the CI/CD process in MLOps is the need to version not only code, but also data and models. This allows for reproducibility and the ability to roll back to previous versions [2, p. 4].
Among the popular tools for implementing CI/CD in MLOps are Jenkins [2, p. 3] and GitHub Actions
[
Figure 4 from the article by Kreuzberger et al. [
and calculation of new features are defined. • B2. Feature Engineering Pipeline. A data processing pipeline is implemented that receives data from various sources, applies transformations, and loads it into a feature store system. 3. C. Experimentation. At this stage, the data scientist (DS) performs data analysis, preparation, and validation, as well as model training and validation. The best model is saved in the Model Registry. Literature rDevaineywlo O. Hanchuk et al. CEUR Workshop Proceedings System Security for
MLBSS
Model extraction attacks
Model stealing
attacks Model-oriented attacks
Model adversarial
attacks Model trojan
attacks System-oriented attacks Platform attacks
Model poisoning
attacks Model-reuse
attacks Model evasion
attacks System-level DNN
attacks Covert channel attacks
Data security and privacy is ensured by implementing various control and protection mechanisms at holding local data samples [199]. The settings have a larger liability of exposing the attack surface to the adversary, in each stage of MLOps. This includes, in particular, data encryption and anonymization, authentication which the aadnvdearscacersys ccaonntmroal,ninipteuglarittey tchheectkraining,inacgtidvaittya minondiitfeorreinngt,sitnacgideesn.tInregspenonesrea,l,asthweeslluacscreesgsuolafrmseaccuhriintye learning algorithms aausdsiutsmaensdtpheenienttreagtirointyteosftitnhge[t3r0a,ipnpin.4g-5d]a. ta and serving data. Availability of the training data is one main concern for security. The goals of data-ormieani ntesdeactutraictyksccoamnpboenseunmtsmanardisperdacatsicteos: 1in.dMegLrOadpes tphreesmenatcehdinien l[e3a0r,ning model
Figure 13 summarizes the pp. 22-25]. performance; E2s.smenatinailpcuolmatpeonthenetpsroefdMicLtOiopns:outcome at test time.
Review & discussion:
• problem definition includes understanding the problem and system requirements;
Overall, the data-oriented attacks could be categorised into three major types of attacks, which are poisoning attacks, backdoor attack•s daantda amdavneargseamriaelnattctoavcekrss. dWathailceoplloecistioonni,nlgabaetltiancgkasnhdavveerbificeaetniona;general term summarising most popular attacks, in this •wmorokdeplociosonnstirnugctaiottnaacnkds rdeeprloeysmenetnsttihneclturdigesgemroledsesl speoleiscotinoinn,gbuaitltdaicnkgs,owphtiimleiztahteiobnaacnkddeovoarluaat-tacks is the backdoor poisonintigona,ttaascwkse.llTarsigitgsedrleepslsoypmoiesnotniinngthaetttaacrkgestreenfevrirtoonsmuecncet;ssfully mislead the model and system behaviour without manipu•lastyinstge mof mthaeindtaentaandcueringvomlvoedseml oinnfietorerinncgetshteagfuen.Fctoiornbiancgkodfotohreadtteapclokys,etdhseyastetamc.k will be activated when the input dataSeccounrittayinprsacctriacfetseidn tMriLgOgpesr:, which could be pixel-level features in an image or designed characters in a sequence. In Table. 2, we have collectively summarised 15 most recently exemplar studies to cover the modes targeting 1. At the problem definition stage: on diferent development stages.
• risk assessment; For both triggerless poisoning attacks and backdoor poisoning attacks, the attacks mostly happen during the data • threat modeling. management stage when the attackers could easily obtain the access to the dataset. This assumption is made against the centralised2m.Aacththineedlaetaarmnianngagmeomdeenlst s[t7a,g7e1: , 91, 161, 172, 182, 190], whilst one specific scenario is identified for the distributed machine•ledaartnaiflonwg dmiaogdrealms.; In [129, 176], the federated learning-based systems are studied that the global model is poisoned by• aSgTgRrIeDgaEtminegththoedoeldoggey nfoordtehrmeaotdcellausspificdaattieosnl;earned from malicious participants. The last attacking stage for backdoor p•ociosnocneipntguaalttmaocdkesliinsgd;iscovered in [88], in which the model is retrained periodically at system maintenance stage. G•ednaetraavllayl,idaatttiaocnk;s have impacts on the data integrity and availability.
Although several•eadraltiaerlinwteorr;ks [120, 128, 143, 153] have identified the poisoning attacks in diferent applications, one earliest work pr•esdeanttainvgeraificmatoiorne. realistic assumption by limiting the adversary’s capability and knowledge of the system was led by Suciu et al. [172]. Neither the feature nor the algorithm knowledge was taken for granted as the f the technical and data-related challenges. This, in turn, calls for adjusting the practices in the context of syste omain and tecDhanniycloalOi.sHsaunecsh.uk et al. CEUR Workshop Proceedings
Currently, the research and development of the state-of-the-practice for MLBSS are mostly focused on two topic
• identification of test adequacy criteria;
hich are technical deb•t taenstdinqguaaglaiitnystaasdsvuerrasnarciea.l iGnepnute;rally, quality assurance defines ‘ a planned and systematic patter
f all actions necessary •tofupzrzoivnigdteecahdneiqquueast.e confidence that the item or project conforms to established technical
requireents’ [
Technical debt, studie•deivnas[i1o7n7a]tftoacrkcsl;assical software systems, has recently been reviewed in MLBSS in order to identif e specific patterns [ 1•4,d9a3ta, 1p6oi2s,o1n7in5g].aAttadcdkist;ional to typical technical debt at code level, the machine learning-base ystems represent extra• tmraadnuea-oltfsesttoinbge; overcome for practical practices for the long term development, deployment nd maintenance at th•e sdyynstaemmic laenvaelyl.sis.
The data dependency debt was extensively discussed as a key contributor f LBSS since it isThounse,soecfutrhitey ipnratectgicreasl icnoMmLpOopnseanretsinitnegMraLteBdSiSntoanadll misaignesnteagraeslloyf mthoerdeevdeilficoupmltetnot ldifeetceycctleas a result of and include both general methods of security assessment and testing (threat modeling, static/dynamic s unstable and underutilised characteristics [162]. Other technical debts, i.e., abstraction debt, configuration debt analysis) and specialized approaches focused on the peculiarities of machine learning systems (testing ata testing debagt,arinesptraoddvuercsiabriilailtyexdaembptl,eps,rdoecteecstsiomnaonf adgateampeonistondienbgt).and cultural debt, may also be accrued and deman
ngoing collaborative eforts. Following in [
Explainability/interpretability of models is an important MLOps practice for ensuring transparency
thics were elaborated. Taking further actions towards measuring the technical debt and targeting to pay it of, a recent
and trust in machine learning models. Explainability refers to the ability to explain and understand the
anuscript submittdeedctiosioAnC-Mmaking processes of a machine learning model [10, p. 66]. This is especially important when
the model is used to make decisions that have significant consequences, such as in military operations
or law enforcement activities [
Explainability as the basis for trust in the ML project allows users to trust the prediction, which increases transparency. The user can verify which factors contributed to certain predictions, introducing an additional layer of accountability. The terms “explainability” and “interpretability” are often used interchangeably, but for MLOPs, explainability is more than interpretability in terms of importance, completeness, and fidelity of predictions or classifications (figure 14) [4, p. 63608].
Individual case explanation
Model accountability Explainability
Interpretability Feature importance explanation
Human readability
Explainable Artificial Intelligence (XAI) is a research direction that promotes explainable decision making [4, p. 63609]. Explainability can be defined as the degree to which an observer can understand the cause of a decision [32, p. 8]. An ML system is explainable when it is easier to identify causal relationships between the inputs and outputs of the system. The more explainable a model is, the better practitioners understand the internal business procedures that occur during model decision making. An explainable model does not necessarily translate into a model that a human can understand (internal logic or processes underlying it), but the explainability of the model allows the user to strengthen trust in the predictions made by the deployed system [4, pp. 63614-63615].
Various methods and tools can be used to achieve model explainability, such as: • attribution-based methods (integrated gradients, saliency maps) or perturbation-based methods (SHAP) [33, p. 3], which explain the model’s decision by assigning high scores to the most influential input features; • incorporating an attention mechanism into models, which allows focusing on the most relevant
network states and inputs [33, p. 2]; • using a combined reward signal during training, which includes not only target metrics but also
interpretability metrics [33, p. 4].
Data quality management is an important practice in the MLOps workflow. According to Steidl et al.
[
In Haller’s book [9, pp. 77-84], data quality management is considered in the context of monitoring and checking data in a production environment to ensure that the data used corresponds to reality.
In the data-centric MLOps lifecycle (figure 15), data quality management is performed at the following stages:
1. Data collection – at this stage, data is created and obtained from various sources: data must be relevant, complete, consistent and reliable. 2. Data quality assessment and cleaning – collected data undergoes thorough quality checks using various metrics such as accuracy, completeness, consistency, timeliness, etc.; quality issues are identified and resolved – incorrect values, missing values, duplicates, noise; data cleaning techniques are applied to improve data quality. 3. Data augmentation and labeling – cleaned data is enriched with additional information and labeled according to the target task; at this stage, the quality of data labeling is also controlled to avoid errors and inaccuracies. 4. Data quality analysis – labeled data is analyzed for quality, its representativeness, class balance, presence of outliers and anomalies are checked; adjustments are made as needed to improve data quality. 5. Model training with quality control – based on quality data, ML model training takes place; during experiments, model and data quality metrics are tracked to ensure stability and reliability of results. 6. Model deployment with quality assurance – the model is deployed in the production environment only after thorough testing on quality test data; measures are taken to maintain data quality in the production environment. 7. Data and model quality monitoring – the deployed model and incoming data are constantly checked for quality: quality metrics, presence of anomalies in data, distribution shift are tracked; the model is retrained on new quality data as needed.
To guarantee data quality throughout the entire lifecycle of their use in MLOps, some approaches to data validation and verification need to be applied: • data quality assessment by determining how suitable this data is for achieving business goals (completeness, uniqueness, integrity, validity, accuracy, timeliness) [26, pp. 2-3]; • data validation in single and cross batches by comparing data characteristics with the expected schema, as well as checking for data drift [7, p. 11]; • automated modular data tests based on a schema to identify errors in data and prevent them from entering the model training stage [7, p. 10]; • versioning of data and related artifacts (processing procedures, metadata, etc.) for traceability, reproducibility of results and compliance with regulatory requirements [7, p. 11]; • documenting data to an extent suficient to ensure model verifiability [7, p. 12].
Timely detection and elimination of defects in data helps prevent wasting computational resources on low-quality data [7, p. 11].
In their work, Singh [26] present techniques for assessing the quality of “big data”, which help identify datasets that may cause problems and unnecessary costs.
To implement data quality management practices, frameworks such as TensorFlow Extended (TFX) can be used, which have data validation components such as SchemaGen and ExampleValidator [7, p. 19].
Configuration files help create more robust software by moving all hardcoded variables into dedicated locations that can be split up or organized at the developer’s discretion [34, p. 3].
Godwin and Melvin [34] propose a template that supports two types of configuration files – a config.py file that contains the template configuration and can include additional resources, such as databases and spreadsheets, and JSON files for storing specific program variables, such as thresholds or parameters. The config.py file is accessible through import statements, while JSON files are called from disk at runtime [34, pp. 3-4].
Yongqiang et al. [35] consider the use of a unified data model based on the YANG language for unifying the description of configuration data and simplifying their management. Neptune Labs [36] indicate that configuration management tools, such as Ansible (https://www.ansible.com/), Puppet (https://www.puppet.com/) and Chef (https://www.chef.io/), can be used to automate configuration and provisioning of MLOps platforms.
Model deployment strategies: • are implemented at the final stage of the MLOps process; • require standardization, automation and encapsulation of models; • are used for gradual transition of a new model to the production environment; • rely on containerization.
Peltonen and Dias [28] highlight the following benefits of using containers for model deployment [28, p. 68]: • model abstraction and process isolation by running multiple models in individual containers representing their runtime dependencies; • ability to create model-specific containers that meet their packaging requirements; • assignment to diferent processors (CPU, Mobile GPU, etc.); • ability to allocate a separate container to facilitate post-processing functions; • a model repository and container storage can be used for fast deployment; • provides a means of standardized deployment; • almost all existing continuous development pipelines facilitate model deployment in the form of containers; • containers can be tailored to specific architectures of edge devices; • Docker containers are a well-established industry standard; • ease of rollback in case of failures.
Gunny et al. [37] describe in detail the development cycle and model deployment strategy using the DeepClean application as an example. A new version of the model is first deployed as a developer version, undergoes validation in conditions similar to production, and only then replaces the current model in the production environment. This uses a service architecture with NVIDIA Triton Inference Server, which supports simultaneous placement of the developer version and the production version of the model.
The authors identify two main approaches to model deployment [37, pp. 11-12].
WInotrhkesthraodiptioPnarlessceennatriaot,ieoacnh user manages their own resources and model versions. Inconsistencies in libraries and dependencies, as well as model versions, lead to inconsistent results. Reduced computational requirements for inference lead to underutilization of hardware resources, depicted by green rectangles on each node (figure 16). More complex deployment scenarios require the use of multiple networks, exacerbating existing issues.
Git repository
User node on compute cluster Local/shared storage Version of DL software stack Ops from different DL frameworks (b) Figure 2: (a) A traditional deployment scenario in which individual users Inconsistencies in libraries and39d8ependencies as well as model versions le demands of inference lead to hardware under-utilization, represented ser node on ompute cluster ocal/shared torage ersion of DL oftware stack ps from different L frameworks rio Model as point in eam users true test
practices yment via table, com
Training Job Training Job
Cloud or local model repository Containerized inference service gRPC inference requests
In the Inference-as-a-Service approach, a centralized service orchestrates models and provides unified hich individual users manage their own software and hardware resources.
interfaces for invoking models (figure 17). A centralized model store synchronizes all users and keeps ell as modtheelmvueprtosidoatne.sPilpeelaindestsoendingRcPoCninsfiesretnecnerteqrueesstustlottsh.e Rseervdicueucseindg sctoanmdarpdiuzetdaAtPioIs nthaatl abstract away the details of the inference execution itself. Inference is performed by a containerized tilization, represented by green rectangles on each node. More complex service that can eficiently schedule asynchronous model execution, maximizing hardware compute networkscapuatbiillitiiezsiinnagpomrtaublletainpd lscealfabrlaemmanenwer.oInrtkhisbapapcrokacehn,cdonst,aienexriazactieonrballaowtisncrgeateinxgipsotrtianblge dardizes ainndfiesorlaetendcmeodaelcerxoecsustioanlelnvuirsonemresntas.nd coordinates complex concurrent
Choosing the right model deployment strategy allows minimizing operational costs, ensuring consiscapacity in a way that is portable and scalable.
tent model operation for all users, and facilitating model version monitoring and control.
p. 3]. quired to
users to pick and choose which libraries they need in order to Infrastructure as code is an important MLOps practice that allows treating infrastructure as code for its k like the reliable kanedeepficietnhtedierpldoyempelnotyanmdemnatnsagleimgehntt w[e3i8g, hp.t2.]I. nInftrhasitsruscetucrteiaounto,mwateionwreillaltebsrtioetflyhe that look deploymdeentsacnrdibmeontihtoerinpgusrtapgoesinethoefMthLOepmsporosctesrse(figleuvrea1n8)t. Iltibenrsaurreiserse,liaabnledcrleeaativone omftohere necessary environment for deploying machine learning models and automates infrastructure tasks [38, detailed information to their documentation.
Infrastructure as code involves describing infrastructure configuration in a declarative way in special ifles (for e4x.a1m.1ple, inhYeArMmLe, sJS.OcNlfoorumdatbs)roerauskin.g Tspheceiaclllanoguudagbers(efoarkexlaimbprlae,rTyercroafnortma)inorstotoolso. ls These filefsodrescprriboevthiseidoesniriendgstaKteuobfethrenineftraesstruccltuusrete[3r8s,pa.n4]d. virtual machines on priDescribing infrastructure settings as code allows automating the process of its creation, modification vate clouds and deploying workloads onto those computational and management [38, p. 3]. This makes it possible to fully reproduce environments, quickly deploy resourcerseasnoduavrociedsm. aWnuahl isleetuspuerproprosr[3t8,opn.4ly]. TchuisrrapepnrotlaycheisxaipsptrsopfroiarteGtoouosgelfoer Ccreloatuindg, complex dynamic infrastructures, when high repeatability and consistency of environments is needed, the intent of the library is to be written in such a way that the user interface is agnostic to the actual cloud backend. Moreover, by using Python contexts to deploy resources, we can ensure that any 399 resources are spun-down once jobs are complete so that unnecesData collection and preparation
Data analysis and exploration
Model building and training
Model testing and validation
Model deployment (Infrastructure automation)
Infrastructure as Code
Model retraining and updating
Model performance monitoring to increase reliability and reduce time costs for manual configuration [38, pp. 2, 4].
The automated approach allows testing infrastructure as code, applying software development practices to it (code review, versioning, etc.). This contributes to improving stability and security, allows quickly tracking and resolving infrastructure issues [38, p. 4].
To implement Infrastructure as code in Azure DevOps, tools such as Azure Resource Manager (ARM) Templates, Terraform, Ansible, Chef are used [38, pp. 3-4]. They allow describing infrastructure as code and automatically creating or modifying it.
Collaboration and communication between various stakeholders is a key MLOps practice for the successful implementation of machine learning and artificial intelligence projects in organizations. MLOps emphasizes how cross-functional teams, such as data analysts, system operators, as well as data and software engineers, collaborate through a harmonized process [7, p. 7].
The essence of the collaboration and communication practice lies in establishing efective interaction and information exchange between the various teams involved in the process of developing and implementing ML models – the data science team, development team, operations team, and business units. This practice is an important component of the development and implementation stage of the MLOps workflow.
As Kreuzberger et al. [
The diagram (figure 19) shows the main participants in the MLOps process and the directions of their interaction: 1. The Data Science team prepares data and develops machine learning models. 2. The Development team integrates the developed models into software products. 3. The Operations team deploys models in the production environment and provides their support. 4. The Business team defines goals and requirements for ML solutions, and also receives results from the Data Science and Operations teams.
Data preparation,
model development
Defining goals and requirements
Business Collaboration tools: Slack, Trello, GitLab wiki
Model deployment and support Data Science
Development
Operations
Integration of models into the product
To ensure efective communication in MLOps practice, the following approaches and tools are used [2, p. 4]: • use of collaboration and knowledge sharing tools, such as Slack, Trello, GitLab wiki; • regular meetings between teams to discuss status, problems and plans; • clear definition of roles and areas of responsibility of process participants; • use of version control systems (Git) for collaboration on code and models; • automation of CI/CD processes to ensure transparency and reproducibility of development.
Since ML models often make important decisions that afect people, risk management and compliance is a critically important MLOps practice, the essence of which is to ensure compliance of developed ML models and systems with regulatory requirements and standards (compliance) and manage potential risks associated with their development and operation.
This practice has a cross-cutting nature and manifests itself at diferent stages of the ML model
lifecycle. Risk management in the context of MLOps involves identifying and mitigating potential risks
associated with ML models, such as data bias, privacy breaches, model accuracy deterioration over
time, etc. Compliance means ensuring that models comply with regulatory requirements, for example,
regarding data protection [7, pp. 18-19]. The main condition for using this practice is the presence of
regulatory requirements or industry standards that the ML system must comply with. Examples can be
GDPR requirements for personal data protection or certifications in the healthcare industry [ 7, p. 5].
Steidl et al. [
Methods of using this practice include regular data quality checks, testing models for bias and discrimination, implementing access control and data encryption, documenting model architecture and development process, monitoring model performance after deployment [7, pp. 11-12]. Yes, compliance is achieved by [7, p. 18]: • quality control and origin of data used for model training to avoid violation of regulatory requirements; • documenting and versioning models and data to ensure reproducibility of results and auditing; • verifying models integrated into the production environment for compliance with requirements; • continuous monitoring of deployed models for timely detection of potential violations or incorrect behavior.
The leading tools for ensuring risk management and compliance practice are version control systems
for tracking changes in data and model code, testing tools for identifying problems in models, monitoring
systems for tracking model accuracy in real time [7, pp. 12, 14]. At the same time, interviews with
practitioners conducted by Steidl et al. [
The diagram in figure 20 shows three main MLOps stages: working with training data, model training, and system deployment (operationalization). At each stage there are blocks that indicate potential risks and compliance measures. This diagram illustrates the cross-cutting nature of risk management and compliance practice in MLOps, showing its presence and interconnections at each stage of the machine learning model lifecycle.
Figure 21 shows the relationships between the key principles, deployment process, and main MLOps practices that are applied at the stage of machine learning model deployment. The principles of automation and reproducibility influence the model deployment process, which is associated with the following MLOps practices: CI/CD, model deployment, data security and privacy, configuration management, model deployment strategies, and infrastructure automation.
A meta-synthesis of systematic reviews [
model compliance deployment compliance 4. MLOps tools provide a wide range of features to support the machine learning model lifecycle, with a focus on automation, experiment tracking, versioning, monitoring, and model deployment. 5. The most common ways to deploy machine learning models in production environments are the use of container technologies, cloud platforms and services, and the deployment of models as web services.
Model deployment Data security and privacy Configuration management Model deployment strategies
Infrastructure automation 6. Adapted software development maturity models, such as CMM, can be used to assess the maturity level of MLOps processes in organizations. 7. Successful MLOps implementation requires the involvement of specialists from diferent areas - software development, data engineering, machine learning, subject matter experts, and management. 8. The main challenges when deploying machine learning models in production environments are managing the model lifecycle, ensuring scalability and performance, monitoring and maintaining models in real-world conditions. 9. Open issues and challenges in MLOps are the need to develop standards and best practices, ensure interpretability and responsible use of models, efectively manage data, integrate knowledge from diferent fields. 10. The main opportunities and development trends of MLOps are the creation of standardized platforms, application in the context of distributed learning, and integration with other approaches to managing the lifecycle of data and models. Current and future areas of MLOps application include a wide range of industries, from finance and healthcare to IoT and natural language processing.
The conducted meta-synthesis showed that MLOps is a promising approach for the efective deployment of machine learning models in production environments, which requires further research and development to address existing challenges and realize potential opportunities.
Next, the key MLOps practices necessary for efective deployment of machine learning models were analyzed. The main conclusions obtained as a result of the analysis are as follows: 1. MLOps is based on a set of principles, processes and practices that ensure efective development, deployment and maintenance of machine learning models. The key principles of MLOps are automation, reproducibility, collaboration, continuous learning and data governance. 2. The main MLOps practices include: continuous integration and delivery (CI/CD), model and data versioning, ML pipeline automation, model performance monitoring, experiment management, model deployment and lifecycle management. 3. Additional MLOps practices, such as data security and privacy, model explainability and interpretability, data quality management, configuration management, model deployment strategies, infrastructure automation, collaboration and communication, risk management and compliance, are important for ensuring reliability, compliance with requirements and eficiency of MLOps processes. 4. The application of MLOps practices allows automating and standardizing the processes of development, deployment and maintenance of machine learning models, which increases the eficiency and reliability of ML solutions in the production environment. 5. Successful implementation of MLOps practices requires the use of appropriate tools and platforms, such as experiment management systems, data and model versioning, infrastructure automation and monitoring tools, as well as establishing efective collaboration between the diferent roles and teams involved in the process of developing and implementing machine learning models.
The analysis showed that the application of MLOps practices is critically important for the successful deployment of machine learning models in production environments. The implementation of these practices allows increasing the eficiency, reliability and reproducibility of the processes of development and operation of ML solutions, which is a necessary condition for their successful use in real business problems.
As a result of the comprehensive study, the MLOps practices necessary for the efective deployment of machine learning models were identified and analyzed. The main conclusions obtained during the study are as follows: 1. A meta-synthesis of systematic reviews was performed to generalize knowledge about MLOps practices. The conducted meta-synthesis showed that MLOps is a promising approach for the efective deployment of machine learning models in production environments, which requires further research and development to solve existing challenges and realize potential opportunities. 2. A diagram of relationships between MLOps principles, processes and practices is proposed.
This diagram illustrates the interconnections between the key principles, stages of the machine learning model development and implementation process, as well as the main MLOps practices that are applied at each stage. 3. The most efective MLOps practices for model deployment have been identified, which include: continuous integration and delivery (CI/CD), model and data versioning, ML pipeline automation, model performance monitoring, experiment management, model deployment and lifecycle management, data security and privacy, model explainability and interpretability, data quality management, configuration management, model deployment strategies, infrastructure automation, collaboration and communication, risk management and compliance.
The obtained results have both theoretical and practical significance. The theoretical significance lies in the generalization and systematization of knowledge about MLOps practices necessary for the efective deployment of machine learning models. The practical significance of the obtained results lies in the possibility of their use by organizations for the implementation or improvement of MLOps processes in order to increase the eficiency and reliability of machine learning model deployment in production environments.
Further research may be aimed at developing detailed recommendations for the implementation of individual MLOps practices in organizations, creating new tools and platforms for automating and managing the lifecycle of machine learning models, as well as studying the efectiveness of applying MLOps practices in various industries and areas of machine learning model application. Declaration on Generative AI: During the preparation of this work, the authors used Claude 3 Opus in order to: Text Translation, Abstract drafting. After using this service, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content. Software Ecosystem for Deploying Deep Learning in Gravitational Wave Physics, in: Proceedings of the 12th Workshop on AI and Scientific Computing at Scale Using Flexible Computing Infrastructures, FlexScience ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 9–17. doi:10.1145/3526058.3535454. [38] C. Vuppalapati, A. Ilapakurti, K. Chillara, S. Kedari, V. Mamidi, Automating Tiny ML Intelligent Sensors DevOPS Using Microsoft Azure, in: 2020 IEEE International Conference on Big Data (Big Data), 2020, pp. 2375–2384. doi:10.1109/BigData50022.2020.9377755. [39] R. Sothilingam, V. Pant, E. S. K. Yu, Using i* to Analyze Collaboration Challenges in MLOps Project Teams, in: A. Maté, T. Li, E. J. T. Gonçalves (Eds.), Proceedings of the 15th International iStar Workshop (iStar 2022) co-located with 41th International Conference on Conceptual Modeling (ER 2022), Virtual Event, Hyderabad, India, October 17, 2022, volume 3231 of CEUR Workshop Proceedings, CEUR-WS.org, 2022, pp. 1–6. URL: https://ceur-ws.org/Vol-3231/iStar22_paper_1.pdf.
The queries were created on 10.05.2024. A PDF file with the article text was added to each query. A report according to the submitted plan was expected from the chatbot. The queries consisted of universal and variable parts.
Universal part of the query: Using the added review article file as a data source, write a detailed report on it according to the following plan.
Variable parts of the query: 1. 1. Year of publication.
2. Research objective (paper aim).
3. Research questions. 2. 4. Information sources (databases, etc.) 5. Inclusion criteria. 6. Exclusion criteria.
7. Quality criteria. 3. 8. MLOps definition (if any).
9. MLOps workflow stages (if any). 4. 10. What frameworks and architectures facilitate MLOps? 5. 11. What MLOps tools can be used to build ML pipelines for
Continuous Deployment? What tools are used in the activities for operationalizing machine learning models? 6. 12. What are the main features offered by MLOps tools? 7. 13. How are machine learning models deployed in production
environments? 8. 14. What maturity models are used to assess the level of
automation in deploying machine learning models? 9. 15. What roles and responsibilities are identified in the activities of operationalization of machine learning models? 10. 16. What challenges are encountered with regard to deploying machine learning models in production 11. 17. What are the open issues, challenges and particularities in MLOps? 18. What are the opportunities and future trends in MLOps? What are the current and future fields in which MLOps is thriving?
2022
Review by
Lima et al. [
Review by
Diaz-de Arcaya et al. [
Comparison Review by
object Recupito et al. [
To identify tools that allow cre- Review of existing literature The main objective of this sysating MLOps pipelines for con- to identify practices, standards, tematic literature review is to tinuous deployment. roles, maturity models, chal- provide an understanding of the To analyze the main character- lenges and tools used to auto- implementation of MLOps and istics and functions of these mate activities of operational-AIOps methodologies in both inMLOps tools to provide a com- izing machine learning mod- dustry and academia. The auprehensive overview of their els into industrial operation thors seek to highlight the chalvalue. (MLOps). lenges, opportunities and future trends in these areas.
Research ques- We answer the following main RQ1: How are machine learning Q1: What are the open issues, tions research question: What tools models deployed in production challenges and particularities in and capabilities enable develop- environments? MLOps and AIOps? ers to create ML-enabled soft- RQ2: What maturity models are Q2: What are the opportunities ware systems? used to assess the level of au- and future trends in MLOps? Which was detailed into two tomation in deploying machine Q3: What are the opportunities sub-questions: learning models? and future trends in AIOps? RQ1. What MLOps tools can be RQ3: What roles and responsi- Q4: What platforms and archiused to create machine learning bilities are defined in the activi- tectures facilitate MLOps and pipelines for continuous deploy- ties of operationalizing machine AIOps? ment? learning models? Q5: What are the current and fuRQ2. What are the main fea- RQ4: What tools are used in ture fields in which MLOps and tures ofered by MLOps tools? the activities of operationaliz-AIOps are thriving? ing machine learning models? RQ5: What challenges are encountered when deploying machine learning models in production environments?
Continued on next page
Comparison Review by Review by Review by
object Recupito et al. [
Inclusion crite- The study discusses compo- Studies related to machine Articles published between ria nents of a minimal end-to-end learning operations (MLOps) in 2018 and 2023, identified by MLOps workflow. general. search queries, containing new The study discusses MLOps Studies evaluating the lifecycle ideas and closely related to practice or machine learning- of machine learning solutions. the topic of MLOps and AIOps based applications. Studies related to maturity were included in the analysis. The study relates to the imple- models of the machine learning mentation of MLOps tool(s). process.
The study describes experi- Studies analyzing roles and ences, opinions, or practices re- responsibilities involved in the garding MLOps pipelines. development and deployment of machine learning solutions.
Studies covering tools for deploying machine learning solutions.
Studies identifying challenges for the development and deployment of machine learning models.
Exclusion cri- The study does not provide de- Studies not published in En- Publications not in English, reteria tails on the design or implemen- glish. tracted publications, publicatation of MLOps tool(s). Studies related to the applica- tions from irrelevant publishers, The study only proposes the de- tion of machine learning mod- subscription materials without sign of a certain component of els. access, and articles with insufmachine learning pipelines. Short papers or posters. ifcient citations (depending on The study does not provide or Studies not related to machine the year of publication) were exreference details on machine learning operations. cluded. learning automation. Studies whose content is not acThe study refers to commercial cessible. platforms that ofer MLOps ap- Articles not relevant to the replications to promote their de- search questions. velopment and deployment services.
Continued on next page
Comparison Review by Review by Review by
object Recupito et al. [
Number of citations MLOps defini- MLOps is a practice that helps A set of practices and princi- MLOps uses machine learning, tion (if avail- model, develop and operate ples for operationalizing data DevOps and data engineering to able) the machine learning lifecycle, science solutions, used to au- bring machine learning systems drawing on DevOps principles tomate the deployment of ma- into production, facilitating the and practices. chine learning models into an creation of machine products.
operational environment MLOps work- Data extraction for integration Data collection Data management lfow stages (if Data analysis Data transformation Distributed training available) Data cleaning, transformation Continuous training Deployment and feature engineering to split Continuous model deployment Monitoring data Presentation of results Retraining Model training Monitoring of machine learning The need to manage the lifecyModel validation to assess the solutions cle and key components of AI quality applications using specialized Model deployment in target en- platforms and tools is emphavironments sized.
Model monitoring Frameworks Continuous training pipelines MLflow and archi- deployed via CI/CD Kubeflow tectures that Orchestration platforms Polyaxon facilitate TensorFlow Extended (TFX) Comet.ml MLOps imple- Machine learning cloud plat- Kafka-ML mentation forms MLModelCI
Comparison Review by Review by Review by
object Recupito et al. [
API technologies for deployment as a web service.
Main features Common features related to Experiment tracking Data, model, and code versionofered by all phases of machine learning Model packaging and version- ing MLOps tools pipelines ing Workflow orchestration and auData management features ML project management tomation Model management features Model registry Monitoring of processes and Continuous integration and de- events livery Integration with cloud and edge Model monitoring infrastructure Hyperparameter tuning Containerization Portability MLOps automation using AutoML Methods of Machine learning cloud MLOps is considered as a set of Deployment in the cloud Using deploying platforms (AWS SageMaker, practices and principles for op- cloud computing resources machine learn-AzureML, DotScience, Google erationalizing Providing isolation ing models in AI Platform) Some MLOps tools, such as Hybrid approaches production Orchestration platforms MLflow, Kubeflow, and Kafka- Deployment on edge/IoT deenvironments (Apache Airflow, Jenkins, ML vices
Kubeflow, MLflow, Polyaxon, The role of “Deployment Lead” Deployment directly on IoT deSeldon Core, Valohai) vices and mobile devices. TensorFlow Extended (TFX) TensorFlow Lite and Core ML. Overcoming limitations Containerized deployments Packaging models Docker.
Serverless architectures Deploying ML functions as a service Reducing costs Deployment via APIs
Continued on next page
Comparison Review by Review by Review by
object Recupito et al. [
Akkiraju et al. [21] propose an adaptation of the Capability Maturity Model (CMM) Roles and re- Data scientists – responsible for Domain specialist – has deep Software developers sponsibilities developing and training knowledge of the subject area Data specialists/data scientists identified in Data engineers – responsible Computational scien- Operations engineers the activities for extracting, processing, trans- tist/engineer – has high Domain experts of operational- forming, and ensuring the qual- technical skills Management/stakeholders izing machine ity of data ML scientist/engineer – responlearning mod- DevOps engineers – responsi- sible for designing els ble for automating deployment Provenance specialist – manprocesses and managing opera- ages the supply of data tional environments Manager – evaluates models Product managers and business Application developer – develstakeholders – provide require- ops applications ments for models and partici- Deployment lead – assesses pate in decision-making aspects Challenges Complexity of managing Integration of software devel- Managing the ML lifecycle encountered Ensuring consistency opment Gap between software engineerwhen deploy- Integration of various tools Implementation of ing ing machine Automation of all stages MLOps/AIOps practices Data management learning Monitoring model performance Machine learning models need Distributed and parallel execumodels in Scaling infrastructure monitoring tion production Ensuring security Identifying infrastructure Diversity of computing infrasenvironments components tructure
Deploying and versioning Monitoring
Explainability Challenges Lack of standardization Integration of software develop- Lack of skilled personnel encountered Ensuring portability ment processes Data management issues when deploy- Configuration and integration Implementation of MLOps prac- Complexity of orchestration ing machine Full automation tices Heterogeneity of hardware learning Understandability It is necessary to go beyond an- Need for continuous monitormodels in alyzing model prototypes ing production Careful monitoring Lack of explainability environments Determining infrastructure Scaling and performance issues Addressing scalability Need for versioning Automation Managing the lifecycle Integration with DevOps
Continued on next page
Comparison Review by Review by Review by
object Recupito et al. [
Integration of MLOps with transportation, and logistics. Use of new hardware platforms DevOps and DevSecOps. Possible development of spe- Use of containers, serverless Increased attention to data cialized technologies management Integration of MLOps with De- Development of versioning Areas of application vSecOps, MLSecOps concepts tools Financial services and banking Emergence of new roles and Industries where MLOps is Healthcare and biotechnology competencies thriving: Manufacturing and Internet of Traditional industries Things Innovative industries Retail and e-commerce Academic disciplines Telecommunications Communication and networkTransportation and logistics ing technologies Healthcare
Scientific activity