The successful deployment of machine learning (ML) models in production environments remains a significant challenge despite advancements in ML algorithms and model development. MLOps, a set of practices that combines machine learning and DevOps principles, has emerged as a promising approach to address this challenge. This paper presents a meta-synthesis of systematic reviews to provide a comprehensive understanding of MLOps practices, tools, and challenges for effective ML model deployment. The meta-synthesis was conducted following Chrastina's [1] methodology and included three systematic reviews and one review of MLOps products and providers. The meta-synthesis identified key MLOps principles, such as automation, reproducibility, collaboration, continuous learning, and data governance. It also revealed the main stages of the MLOps workflow, including data collection and processing, model development and training, deployment, monitoring, and retraining. Various frameworks and architectures that facilitate MLOps implementation were discussed, such as open-source platforms, cloud computing platforms, containerization, and container orchestration. The study highlighted the main features offered by MLOps tools, including automation, experiment tracking, versioning, monitoring, and model deployment. The most common methods of deploying ML models in production environments were identified as the use of container technologies, cloud platforms and services, and deployment of models as web services. The meta-synthesis also discussed the importance of adapted software development maturity models for assessing the maturity level of MLOps processes in organizations. Furthermore, the paper emphasized the critical roles and responsibilities involved in ML model operationalization activities, such as data scientists, data engineers, DevOps engineers, and domain experts. The main challenges encountered when deploying ML models in production environments were discussed, including managing the model lifecycle, ensuring scalability and performance, monitoring and maintaining models in real-world conditions. Open issues and challenges in MLOps were also identified, such as the need to develop standards and best practices, ensure interpretability and responsible use of models, and effectively manage data. The findings of this meta-synthesis can guide organizations in adopting MLOps practices to improve the efficiency, reliability, and scalability of their ML model deployments.
Machine learning (ML) has become an increasingly important technology, finding applications in various domains such as finance, healthcare, manufacturing, retail, and more [2,3,4]. The successful deployment of ML models in production environments is crucial for organizations to reap the benefits of this technology. However, despite significant progress in developing ML algorithms and models, their effective deployment in production environments remains a challenging task [5,6]. This is due to several factors, such as the need for scalability, reproducibility, security, and reliability of models, as well as the complexity of integrating development and operation processes.
To address these challenges, the methodology of MLOps (Machine Learning Operations) has emerged, aiming to apply DevOps principles and practices to the processes of developing and deploying ML models [7,5]. MLOps covers a wide range of practices, such as automation of ML pipelines, versioning of data and models, monitoring of model performance, experiment management, and more [7,6]. Studies show that applying MLOps practices can significantly improve the efficiency and reliability of deploying ML models in production environments [5,6].
However, despite significant interest in MLOps from both researchers and practitioners, there are still gaps and unresolved issues in this field. In particular, there are no generally accepted standards and best practices for implementing MLOps, the issues of integrating MLOps with other approaches (DataOps, ModelOps, AIOps, etc.) are insufficiently explored, and there is a need to develop new tools and platforms to automate MLOps processes [7,5,6].
This paper aims to address the relevant problem of defining and analyzing the MLOps practices needed for effective deployment of machine learning models. The basis for this work is the need to systematize and generalize knowledge about MLOps practices, as well as the need to develop recommendations for their implementation in organizations to improve the efficiency and reliability of deploying machine learning models in production environments.
The rest of the paper is organized as follows. Section 2 describes the methodology used for the metasynthesis. Section 3 presents the results of the meta-synthesis, including the definition of MLOps, stages of the MLOps workflow, frameworks and architectures facilitating MLOps implementation, MLOps tools, features offered by MLOps tools, methods of deploying ML models, maturity models for assessing the level of automation, roles and responsibilities in ML model operationalization activities, challenges encountered when deploying ML models, open issues and challenges in MLOps, and opportunities and future trends in MLOps. Section 4 concludes the paper and outlines directions for future research.
The meta-synthesis was performed according to Chrastina [1] in the following steps:
1. Defining the research subject: MLOps practices for effective model deployment. 2. Identifying relevant sources: systematic literature reviews [7,5,6] and a review of MLOps products and providers [8]. 3. Close examination to determine the common time period, commonalities and differences in the aims, research questions, sources, inclusion, exclusion and quality criteria, MLOps definitions and stages. 4. Determining the relationship between the works by identifying and grouping key themes.
contradictions in the results from different works, and generalizing the results from different works. 6. Synthesis of results.
The meta-synthesis included three systematic reviews [7,5,6] and one review of MLOps products and providers [8]. The systematic reviews were selected based on a search in the Scopus database using the following query: TITLE ( ( systematic OR review OR survey ) AND mlops ). The review of MLOps products and providers [8] was included to provide additional insights into the practical aspects of MLOps implementation.
The meta-synthesis focused on the following key themes: MLOps definition, stages of the MLOps workflow, frameworks and architectures facilitating MLOps implementation, MLOps tools, features offered by MLOps tools, methods of deploying ML models, maturity models for assessing the level of automation, roles and responsibilities in ML model operationalization activities, challenges encountered when deploying ML models, open issues and challenges in MLOps, and opportunities and future trends in MLOps.
The results of the meta-synthesis were expressed in the form of a narrative synthesis, summarizing the key findings and insights from the included reviews. The synthesis aimed to provide a comprehensive understanding of MLOps practices, tools, and challenges for effective ML model deployment, as well as to identify open issues and future trends in this field.
Despite some differences in emphasis and formulations, all the reviews examined define MLOps as an approach for managing, automating, and operationalizing the processes of developing, deploying, and maintaining machine learning models based on practices from software engineering and DevOps. MLOps is a key component for the successful implementation of machine learning solutions in an industrial environment.
Despite different levels of detail and grouping, the reviews demonstrate overall consistency regarding the main stages of the MLOps workflow. These stages cover the entire lifecycle of machine learning models, from data collection and processing to deployment, monitoring, and retraining of models. Differences in the presentation of stages reflect different approaches to structuring and describing the MLOps workflow.
The main stages of the MLOps workflow identified in the reviews include:
• Data collection and processing: collecting, cleaning, transforming, and enriching data for training models.
There are many frameworks and architectural approaches that facilitate the implementation of MLOps, from open platforms and libraries to commercial solutions and cloud services. Key factors are support for automation, scalability, portability, and integration with existing systems and tools. The choice of appropriate frameworks and architectures depends on the specific requirements and constraints of the organization, as well as the maturity level of its MLOps processes. The reviews highlight the following frameworks and architectures as key components of the MLOps ecosystem:
• Open-source platforms and frameworks, such as MLflow, Kubeflow, and TensorFlow Extended (TFX). • Cloud computing platforms and services, such as AWS, Google Cloud, and Azure, for deploying and scaling MLOps solutions. • Containerization (e.g., using Docker) and container orchestration (e.g., using Kubernetes) architectures for ensuring portability and scalability of MLOps solutions.
• Pipeline orchestration platforms, such as Apache Airflow, Jenkins, and Polyaxon.
• Specific frameworks and platforms, such as Kafka-ML and MLModelCI, for managing the lifecycle of ML models.
There is a wide range of MLOps tools for building machine learning pipelines and operationalizing models, from open platforms such as MLflow to commercial solutions from cloud providers and specialized companies. The choice of specific tools depends on the needs and scale of the organization, as well as compatibility with the existing technology stack. The reviews highlight the following popular MLOps tools:
MLOps tools provide a wide range of features to support the machine learning model lifecycle, with a focus on automation, experiment tracking, versioning, monitoring, and model deployment. Some tools offer more specialized features, such as hyperparameter optimization or data management. The choice of a tool with an appropriate set of features depends on the specific needs and goals of the organization in the field of MLOps. The reviews identify the following key features offered by MLOps tools:
• Experiment tracking and model versioning.
• Automation and orchestration of MLOps workflows, such as training and deployment pipelines.
• Monitoring of model performance and degradation in production environments.
• Data management features, such as data extraction, transformation, and monitoring.
• Support for various machine learning libraries and frameworks.
• Automated hyperparameter optimization and model testing.
• Integration with existing systems and support for collaborative work of teams.
The most common methods of deploying machine learning models in production environments are the use of container technologies, cloud platforms and services, and deployment of models as web services.
The choice of a specific approach depends on the requirements for latency, scalability, and availability of models, as well as the existing infrastructure and ecosystem of tools in the organization. The reviews describe the following key methods and aspects of deploying ML models:
• Deploying models using container technologies, such as Docker, which ensures model mobility and isolation. • Deploying models in cloud environments using platforms and services from major providers, such as AWS, Google Cloud, and Azure.
• Deploying models as web services using REST API or other protocols to provide access to predictions in real-time. • Using orchestration platforms (Apache Airflow, Jenkins, Kubeflow, MLflow, Polyaxon, Seldon Core, Valohai) for automated scaling and container management. • Built-in deployment capabilities of some MLOps tools, such as MLflow, Kubeflow, and Kafka-ML. • Deploying models not only in the cloud but also on edge devices using specialized frameworks, such as TensorFlow Lite and Core ML. • The main stages and features of the ML model deployment process using CI/CD pipelines
All systematic reviews [7,5,6] indicate that the level of automation of MLOps processes is one of the key factors in assessing an organization's maturity in this area. Although the reviews do not provide a comprehensive description of MLOps maturity models, they emphasize the importance of assessing the level of automation in the processes of developing, testing, and deploying models as a key factor in an organization's maturity in this area. Adapting existing software development maturity models to the specifics of MLOps can be an effective approach to assessing and improving machine learning processes in an organization.
The reviews mention several maturity models for assessing the level of improvement in the machine learning solutions development process:
• The maturity model proposed by Amershi et al. [9], based on the Capability Maturity Model (CMM) and the Six Sigma methodology, which checks whether an activity: (1) has defined goals, (2) is consistently implemented, (3) is documented, ( 4) is automated, ( 5) is measured and tracked, and ( 6) is continuously improved. • According to Dhanorkar et al. [10], organizations can be classified into three levels of maturity in developing machine learning solutions: (1) data-oriented, (2) model-oriented, (3) pipeline-oriented. • Lwakatare et al. [11] describe five stages of development practices improvement: (1) a manual process driven by data science, (2) a standardized process of experimental-operational symmetry, (3) an automated ML workflow process, (4) integrated software development and ML workflow processes, and (5) an automated and fully integrated CD and ML workflow process. • Akkiraju et al. [12] proposed an adaptation of the CMM model with the definition of five maturity levels for each assessed capability: (1) initial, (2) repeatable, (3) defined, (4) managed, and (5) optimizing.
Despite some differences in the detail of roles, the reviews examined recognize the need to involve professionals from various fields -software development, data engineering, machine learning, subject matter experts, and management -for the successful operationalization of machine learning models. Close collaboration and communication between these roles is critical for the implementation of MLOps practices in organizations. The reviews identify the following key roles and responsibilities in ML model operationalization activities:
• Data scientists/ML scientists, who are responsible for developing, training, and experimenting with ML models. • Data engineers/data providers, who are responsible for extracting, processing, transforming, and ensuring the quality of data for training models. • DevOps engineers, ML/MLOps engineers, and software engineers, who are responsible for operationalizing models, automating deployment processes, creating pipelines, and managing environments.
• Managers, leadership, and business stakeholders, who define requirements for models regarding their deployment, make decisions, and support the MLOps strategy. • Domain experts, who provide domain knowledge and participate in data labeling in specific domains.
Some reviews also mention additional roles, such as computational scientists/engineers, ML scientists/engineers, provenance specialists, application developers, and deployment leads, who have specific responsibilities in the MLOps process.
The reviews demonstrate that deploying machine learning models in production environments is associated with a number of challenges, such as managing the model lifecycle, ensuring scalability and performance, monitoring and maintaining models in real-world conditions. Addressing these challenges requires a comprehensive approach that includes automating MLOps processes, selecting appropriate infrastructure, ensuring data security and privacy, and effective communication with business stakeholders.
The main challenges identified in the reviews include:
• The complexity of managing the machine learning model lifecycle, including versioning, tracking, and reproducibility of models and data, as well as the problem of ensuring the scalability and performance of models in real-world conditions with large volumes of data and requests. • Challenges related to monitoring and maintaining models in a production environment, including detecting data drift and model performance degradation. • Challenges related to integrating software development with the machine learning pipeline.
• Challenges related to ensuring data security and privacy when deploying machine learning models. • The quality, availability, preparation, labeling, and integration of data from various sources is a significant challenge that requires a lot of time and resources, and the problem of interpreting and explaining the results of model operation to end-users and business stakeholders is highlighted. • The gap between software engineering and machine learning skills -data scientists often do not understand the requirements of certain production environments, and software developers do not have sufficient machine learning skills. • The effective distribution, parallelization, and orchestration of data and ML tasks.
• The diversity of computing infrastructure.
The reviews identify a number of open issues and challenges in MLOps, such as the need to develop standards and best practices, ensure interpretability and responsible use of models, and effectively manage data. The particularities of MLOps, such as the difference from traditional software development, the importance of the human factor, and the need to integrate knowledge from various fields, require consideration when implementing MLOps practices in organizations.
The main open issues and challenges in MLOps identified in the reviews include:
• The need to develop methods and tools to ensure interpretability, reproducibility, and responsible use of machine learning models in the context of MLOps. • The importance of developing approaches to data management in MLOps, including ensuring data quality, privacy, and security. • The need to develop and implement MLOps standards and best practices to ensure consistency and compatibility between different tools and platforms.
• The importance of the human factor in MLOps, including the need to ensure effective communication and collaboration between different roles and teams and to train qualified personnel with cross-functional skills in programming, data processing, and operational activities.
The reviews also highlight the following particularities of MLOps:
• MLOps must take into account the specifics of the machine learning model development process, which differs from traditional software development. • MLOps should be considered in the context of the overall digital strategy of the organization and emphasize the need to align MLOps practices with business goals and needs.
The reviews outline a number of opportunities and trends in the development of MLOps, such as the creation of standardized platforms, application in the context of distributed learning, and integration with other approaches to data and model lifecycle management. Current and future areas of MLOps application include a wide range of industries, from finance and healthcare to IoT and natural language processing, indicating the significant potential impact of this approach.
The main opportunities and future trends in MLOps identified in the reviews include:
• The potential for the development of standardized MLOps platforms and tools that will simplify and accelerate the deployment of machine learning models in production. • The prospects for integrating MLOps with other approaches, such as DataOps, ModelOps, and DevSecOps, to ensure comprehensive management of the machine learning model lifecycle. • Significant opportunities for further academic research and development due to the fact that MLOps is still at an early stage. • The expectation of an increase in demand for MLOps tools and platforms with the spread of artificial intelligence solutions. • The emergence of new roles and competencies related to MLOps as the industry develops.
• Opportunities for applying MLOps practices in the context of distributed and federated model learning, which will allow efficient use of decentralized data. • Involving business units and educating management on MLOps principles.
• The use of hardware platforms such as FPGA and IoT to improve performance and privacy.
The reviews identify the following current and future areas of MLOps application:
• Industries such as finance, healthcare, retail, marketing, and manufacturing, where machine learning models are already actively used to solve real business problems. • The potential for applying MLOps in the field of IoT and edge computing, where machine learning models can be deployed on resource-constrained devices. • 5G and 6G technologies, educational and scientific activities.
• Transport and logistics.
This paper performed a meta-synthesis of systematic reviews and a review of MLOps products and providers to summarize knowledge about implementing MLOps practices for effective deployment of machine learning models. The main findings obtained as a result of the meta-synthesis are as follows:
1. MLOps is an approach for managing, automating, and operationalizing the processes of developing, deploying, and maintaining machine learning models based on practices from software engineering and DevOps. MLOps is based on a set of principles, processes, and practices that ensure the effective development, deployment, and maintenance of machine learning models. with a focus on automation, experiment tracking, versioning, monitoring, and model deployment. 5. The most common methods of deploying machine learning models in production environments are the use of container technologies, cloud platforms and services, and deployment of models as web services. 6. Adapted software development maturity models, such as CMM, can be used to assess the maturity level of MLOps processes in organizations. 7. Successful implementation of MLOps requires the involvement of professionals from various fields -software development, data engineering, machine learning, subject matter experts, and management. 8. The main challenges in deploying machine learning models in production environments are managing the model lifecycle, ensuring scalability and performance, monitoring and maintaining models in real-world conditions. 9. The open issues and challenges in MLOps are the need to develop standards and best practices, ensure interpretability and responsible use of models, effectively manage data, and integrate knowledge from various fields. 10. The main opportunities and trends in the development of MLOps are the creation of standardized platforms, application in the context of distributed learning, and integration with other approaches to data and model lifecycle management. Current and future areas of MLOps application include a wide range of industries, from finance and healthcare to IoT and natural language processing.
The meta-synthesis showed that MLOps is a promising approach for the effective deployment of machine learning models in production environments, which requires further research and development to address existing challenges and realize potential opportunities.
The results obtained have both theoretical and practical significance. The theoretical significance lies in the generalization and systematization of knowledge about MLOps practices necessary for the effective deployment of machine learning models. The practical significance of the results obtained lies in the possibility of their use by organizations to implement or improve MLOps processes in order to increase the efficiency and reliability of deploying machine learning models in production environments.
Further research may be aimed at developing detailed recommendations for the implementation of individual MLOps practices in organizations, creating new tools and platforms for automating and managing the lifecycle of machine learning models, as well as studying the effectiveness of applying MLOps practices in various industries and areas of machine learning model application.
Declaration on Generative AI: During the preparation of this work, the authors used Claude 3 Opus in order to: Text Translation, Abstract drafting. After using this service, the authors reviewed and edited the content as needed and takes full responsibility for the publication's content.