Today, machine learning (ML) is widely used in industry to provide the core functionality of production systems. However, it is practically always used in production systems as part of a larger end-to-end software system that is made up of several other components in addition to the ML model. Due to production demand and time constraints, automated software engineering practices are highly applicable. The increased use of automated ML software engineering practices in industries such as manufacturing and utilities requires an automated Quality Assurance (QA) approach as an integral part of ML software. Here, QA helps reduce risk by ofering an objective perspective on the software task. Although conventional software engineering has automated tools for QA data analysis for data-driven ML, the use of QA practices for ML in operation (MLOps) is lacking. This paper examines the QA challenges that arise in industrial MLOps and conceptualizes modular strategies to deal with data integrity and Data Quality (DQ). The paper is accompanied by real industrial use-cases from industrial partners. The paper also presents several challenges that may serve as a basis for future studies. Quality assurance, machine learning, automated software engineering, software testing, data integrity, data quality, MLOps.
ceptualize an infographic diagram of the overview of the software pipeline are pre-determined for a given appliMLOps architecture in Figure 1, which shows the contin- cation, and we discovered a lack of implementation for nEvelop-O Engineering (B. S. Ahmed) • RQ 1: What are the industrial QA challenges when conventional software testing and QA are not viable for MLOps? We collaborated with partners in the manufacturing and utilities sector to discover and outline the in cyber-physical systems in Industry 4.0 and human- (DQ) professionals and tools for data QA [7].
Traditionally, the software development life cycle includes quality assurance (QA) practices and automated tools to assess the quality of the software system. Due to the increasing functionalities of recent developments centered development in Industry 5.0, ML programs that have core functionality are becoming increasingly ML software systems. ML software development in operation is an ongoing development in automated software uous training and deployment of ML software. Recent advances toward a continuous development cycle of ML software are known as MLOps [ 1] or AIOps [2]. An MLOps pipeline, for example, in Amazon Web Services (AWS) [3] and Microsoft Azure [ 4], generally consists of ML software development and automated deployment and monitoring (or Ops). Then, trained ML software is deployed for real-time prediction and classification tasks uous and automated development of ML software. Such ML software, which does not follow the conventional way of authoring software code, is a black box [ 5] and is data-driven [6]. Thus, shifting the requirements for QA, which previously relied on exposure to software code for Software Quality Assurance (SQA) and Data Quality
Data and SQA ensure compliance with the level of
accuracy, data security, and performance scripted in an
organization’s QA policies and enforced by the SQA plan in
classical software development in operation (DevOps) [ 8].
ing approaches and data QA in a software development
pipeline. Recent developments in SQA of ML are enabled
by methods such as domain-specific software testing [
and mutation testing [
When it comes to data QA, data integrity, trustworthiness, and DQ, they are measured by individual data dimensions such as timeliness, uniqueness, relevancy, and location engineering that sets up automated pipelines for contin- Such a QA plan in MLOps requires robust software test
data management
sensor (Input)
data management
QA challenges in industrial MLOps for real Use Case (UC)s. • RQ 2: What approach can we take when dealing with QA in industrial MLOps? We propose concept for a modular approach to data QA, where each component of data QA is an ML software in the industrial MLOps architecture mentioned in Figure 1, with automated training, selection and deployment of data dimensions. Furthermore, we conceptualize a practical and applicable MLOps architecture to enable QA and the continuous delivery of ML software.
Network architecture has undergone significant advances
in recent years [
Although the architecture categories difer for each application, they combine to form a hybrid network architecture for a suite of automated ML applications. Furthermore, because the hardware resources and time constraints available on the IoT device, the edge or fog network, and the cloud are vastly diferent, this creates a challenge for QA for industrial MLOps.
Working in collaboration with industrial partners, we have identified the following challenges in relation to QA that are applicable to any architecture: • Modelling challenges: External factors such as anomaly and noise deviate from the sensor data or signals from its ideal measurements. Furthermore, QA for ML must automatically accommodate class imbalance and drifts [ 14], where the current instance of the containerized ML application has observed changes in the test data compared to the data on which it was trained.
There is a lack of robust AutoML strategies and algorithms for such external factors in automated QA.
To accommodate both lightweight and cloud-based ML • Resource, time, and scalability: While some in- software pipelines for industrial applications, we dedustrial processes are sparse, others are frequent signed a modular QA solution. The modular architecture enough to produce big data. For big data, it is is predicated on the notion that every data QA step needs challenging to perform the necessary data QA its unique collection of dimensions. And the set of data stages in a timely manner. In addition, indus- QA dimensions for each data QA step is determined by tries scale up frequently, and system integration the answer to the following question in an organization’s of new hardware may difer from existing ma- SQA plan: What are the automated steps taken as a result chinery. The rapid adoption of new hardware in of data QA? existing QA processes poses a scalability issue. • Architectural constraints: Industrial MLOps 3.1. ML software architecture architectures have network components, each with its capabilities and limitations. Network Knowledge of automated steps or actions serves as the components, such as routers and switches, have foundation for QA strategy. To make this behavior poscapabilities such as the number of data packets sible, we divided the overall architecture of QA for ML they can handle at a given time, network schedul- into three phases: (i) definition and formulation, (ii) diing, and routing. Additionally, these components mension selection, and (iii) QA model training, each anare subject to network attacks, such as distributed swering the following questions: denial of service, for example [15]. This contributes to the total latency that a data packet • Definition and formulation: How does an organeeds to traverse from IoT sensors to QA mi- nization define and formulate QA dimensions such croservices running on edge/fog and is a chal- as trustworthiness, relevancy, and privacy? lenge for a robust and scalable architecture. • Dimension selection: What minimum QA di• Lack of production data in manufacturing mensions are necessary to achieve the QA actions? processes: Some manufacturing operations are • QA model training: How the selected QA dimentime-intensive or infrequent/sparse, typically oc- sions can be trained and weighed automatically in curring on average once or twice a day. One such industrial MLOps? example is electroslag remelting in the steel industries [16]. This indicates that such operations 3.1.1. Definition and formulation will have a few hundred manufacturing events In the first stage, given the objectives and QA policies of over the course of a year, and the lack of data is an organization, formal definitions and mathematical fora challenge for the ML software. Although data mulations need to be developed for each QA dimension. augmentation or a principled approach to gener- While the rest of the QA for the ML strategy is automated, ating synthetic data has been used to fill the gaps, all QA dimensions require a definition. The definition it poses a QA challenge to strategically split the changes as the organization’s policy shifts. Although limited ground truth data for robust training and some QA dimensions are universal, others depend on the testing [17]. objectives of a company. Sensor noise, for example, is • Compliance with regulatory, export control, an ubiquitous component that afects all streaming data and ISO standards: Data is often subject to reg- from industrial IoT sensors. Other QA indicators, such ulatory requirements from government entities. as contextual DQ and data integrity depend on the QA For example, with medical equipment that re- strategy of the organization. Furthermore, for MLOps, quires additional regulatory requirements for in- it is also important to define QA actions. For example, creased safety, user data is subject to the General a QA action is to identify whether the streaming data Data Protection Regulation (GDPR1) in Europe is relatively clean of noise and anomalies with intrinsic and defense data are subject to export control. DQ metrics and applicable to current ML software in These regulations are local to a region and are production with contextual DQ metrics. not universally applicable. In automated and continuous ML software development, adhering to 3.1.2. Dimension selection such regional regulations is challenging for automated QA.
Following the definition of all dimensions of QA, the next step is to determine which dimensions are relevant to a 1https://gdpr-info.eu/ Robustness testing test data
ML software (new version)
ML robustness
test oracle Response variables case, with QA actions, the streaming data are set up to be sent to the QA for ML strategy in a diferent software pipeline or ofline storage for future study.
3.1.3. QA model training The final step before deployment is QA for the ML model training. An approach to this problem is to iteratively perform dimension selection and model training, where the QA for ML model is trained with dimensions fixed, and dimension selection is performed to promote sparsity in the QA for ML graph while keeping the model ifxed. Both steps are performed iteratively until the QA for ML model converges. For each parent in the model training process, robustness testing, a software testing methodology that examines the boundary conditions of a software [ 18], is carried out with the test data for a binary pass/fail score, as shown in Figure 2.
The outcomes of each parent’s robustness testing are then considered for the desired automated actions, allowing action-based QA for ML. For example, a score of ”pass” for ”intrinsic DQ” and ”fail” for ”contextual DQ” means that the streaming data are relatively clean and usable without the need of data cleaning (i.e., less noise, anomaly-free, and without NaN/missing values) but is not relevant for the current ML software in production. Still, it might be relevant for other ML software. In this • Automated ML model retraining: ML software, including AutoML software, once trained in ‘cloud services’ is automatically deployed to the edge as an Edge AI software on the ‘edge node’. Once deployed for the first time, data flows from the network to the delivery and collection microservices to the cloud API layer. Once model degradation has occurred, the ML software is retrained or adapted in the cloud. A new version of the ML software is automatically deployed on the edge node. • Real-time and automated decision and action: Data transfer from the network to the delivery and collection microservices to the microservices for UCs. Microservices for UCs are then sent to Action Services to execute the appropriate action. The action signal is sent back to the network. • Automated QA assessment: Similar to realtime and automated decision and action, data 2https://www.docker.com/ 3https://kubernetes.io/ s tr o a u t c A d n a ,r k o w t e N ,r s o s n e S T o litIr a s u d n I
Delivery & Collection Services Application Data Monitoring Data
Action Services Cloud Actions
Config & Monitor Data and Software QA NaN/Missing Values Anomaly Detection
Microservices for UCs
Industrial UC Peak Usage
Detection Demand Response
Forecasting Drift Detection
Classifcation
System & Network UC
Network Optimization System Optimization Platform Optimization e illi-treaennOm flileoudonCO o r R
API Layer Automated Deployment Trained Model
Runtime
Library Docker File Docker Hub
Cloud Services ML Model Development Analytics Classifcation
Quality Assurance Data Quality Assessment
Testing Drift Adaptation
ML Model Retraining lfows from the network to microservice delivery and collection to microservices for UCs and Action Services. The microservices for UCs here are containers that evaluate the quality of the incoming data. Furthermore, the quality of the containerized ML software for model degradation. Further, the action signal from the action microservice is sent to the Cloud API layer for the execution of the appropriate cloud service(s). • Message forwarding for cloud services: In this scenario, for long-term storage, the message or data are forwarded from the network to the delivery and collection microservice to the cloud API layer for persistent storage.
The modular QA and the MLOps architecture collectively enable real-time execution of real-time ML and action supported by data and SQA processes. However, research in this area is still in its infancy, and we have identified the following vectors for future direction: • Software testing: The principles of Chaos Engineering state that systems react diferently based on surroundings and trafic patterns. This is true for software, and automated software engineering and software testing procedures, such as shadow testing, allow the software to be analyzed in terms of how it performs in diferent environments. The log data of the software in various environments is then analyzed. The use of SQA opens the possibility of potential research to automate shadow testing and other automated software tests. An industrial application where automated software testing with QA is useful is in compressed air. Compressing air is an energy-intensive process that is used in industries to remove dust and cooling [19]. In addition to wasting air/gas and increasing operational expenses, leaks can also be a point of entry for contaminants to enter the system. Therefore, the process must be continuously monitored for leaks to reduce overall energy and operational cost. Streaming data in compressed air are often subject to anomaly and noisy measurements. Monitoring and predicting future air pressure readings based on historical data is one such UC. • Automated regularization, weights, and parameter estimation of ML software: ML models in ML software often need to be fine-tuned with regularization, weights, and other parameters to obtain a more accurate convergence. Furthermore, ML models can get caught in local minima, and one of the best practices is to train multiple times to use the trained model with the optimum performance. In a machine learning software configuration with continuous training and automatic deployment, it is not always possible to perform this repeated training and fine tuning. In modern implementations, systematic parameter search looks for the best match among a prede
This work has been funded by the Knowledge Foundation of Sweden (KKS) through the Synergy Project AIDA - A Holistic AI-driven Networking and Processing Framework for Industrial IoT (Rek:20200067).
Physical and Virtual Worlds (ITU K), 2021, pp. 1–9.
[16] J. B. Boštjan Arh, Bojan Podgornik, Electroslag
remelting: A process overview, Materiali in
tehnologije 50 (2016) 971–979.
[17] A. Chatterjee, B. S. Ahmed, E. Hallin, A. Engman,
Testing of machine learning models with limited
samples: An industrial vacuum pumping
application, arXiv preprint arXiv:2208.04062 (2022).
[18] J. M. Zhang, M. Harman, L. Ma, Y. Liu, Machine
learning testing: Survey, landscapes and horizons,
IEEE Transactions on Software Engineering 48
(2020) 1–37.
[19] M. Benedetti, F. Bonfa, I. Bertini, V. Introna, S.
Ubertini, Explorative study on compressed air systems’
energy eficiency in production and use: First steps
towards the creation of a benchmarking system for
large and energy-intensive industrial firms, Applied
energy 227 (2018) 436–448.
[