Machine learning extracts models from huge quantities of data. Models trained and validated over past data can be deployed in making forecasts as well as in classifying new incoming data. The real world which generates data may change over time, making the deployed model an obsolete one. To preserve the quality of the currently deployed model, continuous machine learning is required. Our approach retrospectively evaluates in an online fashion the behaviour of the currently deployed model. A drift detector detects any performance slump, and, in case, can replace the previous model with an up-to-date one. The approach experiments on a dataset of 8642 hematochemical examinations from hospitalized patients gathered over 6 months: the outcome of the model predicts the RT-PCR test result about CoViD-19. The method reached an area under the curve (AUC) of 0.794 , 6% better than ofline and 5% better than standard online-binary classification techniques.
and replicating the human way of thinking.AI evolved
very rapidly, and nowadays aims to “reason about huge
quantities of data” [
ML processes huge quantities of stored data to derive a model: the model enables us to make forecasts about values for future data. Forecasts are the more precise and the more accurate ones the more the model adheres to the real world. However, the real world may experience changes, thus drifting away from the original behavior, i.e., the one over which the model was initially defined.
As a consequence, the performance of the model slumps.
Two urgent needs arise: a) detect the performance slump of the model; b) decide when new training on more recent data is needed, to re-couple the model with the real world.
The current paper aims at identifying a criterium to
detect the need for re-coupling the model with the real
world. As an application scenario, the paper considers
data from blood examinations from [
A properly trained model can behave well in the real
Published in the Workshop Proceedings of the EDBT/ICDT 2023 Joint scenario for some time; however, the performances
deConference (March 28-March 31, 2023), Ioannina, Greece grade over time [
The paper is structured as follows: Section 2 describes the state of the art in ML; Section 3 focuses on the approach we propose here; Section 4 describes some details about the prototype implementation; Section 5 reports about the on-the-field deployment of the model; finally, Section 6 draws some concluding remarks. erating data. Manual retraining is functional but expen- Therefore, if context transformations are known, the consive and time-consuming. The current enterprise ap- troller will retreat to an earlier composition instead of proach ofers MLOPs (Machine Learning OPerations) retraining itself. Otherwise, it will train new ML models. as a potential solution for automated retraining. How- Consequentially, if a context is not taken into consideraever, minimal efort was devoted to developing adaptive tion explicitly, substantial bias can be introduced. machine-learning frameworks that can continuously up- A completely new technique is proposed by Bach et date models. Operating data must be integrated into al. [10], somehow similar to the one later on used in the models quickly to enable reliable predictions. Online ma- current paper. The novel vision is to use the interacchine learning allows the ongoing recalibration of ML tion between two learners, a stable online learner with models in fast-changing circumstances. a reactive one, and their contrasts in accuracy to man
Deploying a two-fold system could benefit. One part age concept drift. The stable learner beats the reactive of the system, once suitably trained, takes care of reading learner when acquiring a new concept, but the reactive new incoming data and of making suitable forecasts or learner surpasses the stable learner in the time behind suitable classifications over the most recent data. The which the concept changes. second part of the system guarantees experiment tracking Concept drift in ML refers to changes in the proband automated deployment: this part of the system auto- lem’s relationships between input and output data gradmatically warns in case of a performance slump, retrains ually [11]. With more focus on the issues relevant to the the model, and deploys the most recent model. current paper, the detection of concept drifts is to be performed in the data streams over time. While concept drift 2.2. Related Work detection can be performed by some statistical process control as DDM [12], or temporal window [13] distriMajor relevant literature refers to auto-adaptive machine bution as ADWIN [14], the remediation of concept drift learning and concept drift. can be collected by the taxonomy presented by Gama at
Auto-adaptive machine learning [
A first try is from Widmer et al. [
The approach by Gomes at al. [
Another similar approach is also proposed by Nasci- practices update the existing model regarding the most mento et al. [9]. They suggest a control module that recent data samples [16]. Relevant features of the underconfigures ML models, monitors the context modifica- lying model are continuously adapted, e.g., the weights tions, and holds a history of the trained ML-based models of a neural network. Adaptions can either be founded on that deliver the best result for one of the operational con- triggers, such as precise concept drift detectors, or can texts. When the context changes, the rate of the outcomes be taken out regularly on an evolving story. Triggered achieved by the ML model currently in use may drop. transformation methods are defined as knowledgeable or operational strategies, whereas growing strategies are viewed as random or inactive. Triggered transformation methods use a single prediction model based on a drift detection algorithm (e.g., ADWIN [14]). If a drift warning is signalled, the ML model is adjusted [17, 18].
Our approach is based on a detector strategy, where the system contains a primary neural network for output prediction and a secondary neural network for the model update in the background (Figure 1). Thus, we ifrst identify a model by ML techniques, then we monitor the up-to-dateness of the model over time.
primary network
performace estimator x (input) y (predicted) d e t c e t e d itfr d
yl (actual) Data from [t0 - t1] M1inucsoemdintogadnaataly,ze M2inucsoemdintogadnaataly,ze used to train and to M2 trained and validated M3 trained and validated
validate M1 during [t1 – t2] during [t2 – t3] t0 t1 t2
t3 M1 is no longer a good performer
M2 is no longer a good performer
Real-world datasets usually do not include details regarding the exact starting time and ending time of the drift, since they are usually afected by hidden aspects that cannot be calculated. Thus, comparing the general predictive accuracy of other drift-handling techniques on real-world datasets is a prevalent practice. Metrics such as accuracy or Area Under the Curve (AUC) are used for classification problems, and the Mean Absolute Error (MAE) is used for regression ones.
the performance slump, so that, if a performance degradation is detected, the currently deployed neural network (initially, it is the primary network of Figure 1) is replaced by an updated neural network (the secondary network of Figure 1). Thus, whenever a revision of the model is evaluated as necessary, the optimizer (e.g., Adam or Stochastic Gradient Descent – SGD) revises the secondary model parameters after a time window has elapsed. The optimization is a stochastic gradient descent method and minimizes the loss function (i.e., cross-entropy in our approach)
dataset instance i. The primary neural network (Figure 1) accepts as input, builds up a first model, receives a new incoming value , and delivers a predicted result .
After the actual outcome ′ is observed, it is used to update the model of the secondary neural network (Figure 1). A performance estimator (Figure 1) follows Cross-entropy: the predicted () and ground truth (′) results. − log (; ) + (1 − ) log(1 − (; ))
The primary neural network (; ) takes instance measurements as inputs to predict outcomes. We use We use a batch task collecting the recently gathered to represent the weights and biases of the primary neural data (, ) of size (number of samples in the considnetwork, initially trained using the training dataset. ered time span). We perform some batch SGD updates
The secondary neural network (; ) is being up- times: is suitably chosen (see Section 5.1).The perfordated behind the scenes by an online optimizer. is mance estimator signals the degradation of the operating initialized to the same parameters as the initial param- neural network and activates a replacement. eters . Afterward, at the evaluation of new incoming Our proposed approach initially requires the primary samples, is constantly updated, using new incoming network to identify a model based on data collected over data within the latest time window, as the time window a time span; the identified model is then deployed. Next, occasionally slides forward (Figure 2). the approach compares the results () predicted by the deployed model (which at the beginning is the model
Algorithm 1 Online evaluation 1: Input: , , collected data {, } 2: # stores results predicted by the network 3: #′ stores results actually observed 4: for in incoming data do 5: . add(, (; )) # update predicted results 6: ′ . add(, ′ ) # update observed results 7: end for 8: if Detector(, ′ ) then 9: ← # replace the primary neural network 10: end if
The plain ML workflow has a streamlined structure and it is set up by three major steps: data acquisition and preparation (DAP), to process and augment data for use by models; model development and training (MDT), to build 11: ← ( , ′ ) the model based on a set of goals; and model deployment and operations (MDO), for integrating model predictions into the business and build some system-wide features. from the primary network) with the actual ones (′), us- The market ofers several very good tools, some of them ing a specific metric based on the problem we are solving. taking care of all three steps: however, in most cases, one According to the metric (line 8 of Algorithm 1), if the per- tool performs very well on one step (DAP, or MDT or formance of the currently deployed model is worse than MDO), only, and to date, no tool performs very well on the performance of that same model during the previous all the steps. Thus, to select the proper tools, we build up time span (i.e., we observe a model drift), the currently de- an evaluation matrix of the major open-source tools. ployed model is getting worse, and it needs to be replaced. In our approach, we enrich the plain ML workflow The model from the secondary network is continuously to fulfill the requirements of Figure 1, which includes a retrained with new incoming data, keeping this replace- re-training and a re-deployment of the process model. ment model ready to jump in: when requested, the model The resulting enriched workflow is depicted by Figure 3, from the secondary network (the one continuously re- where the steps are suitably orchestrated. trained) replaces the previously deployed model, thus tbheecodmetiencgtotrhceonmepwalryesdtehpelomyeetdrimcwo ditehli.nInthoetahcetruawl otirmdse: inizMiaotdiliezlaAtion traMtineoisdnteginlgaAnd Peesrftoimrmataionnce MuopddealteB smpeatnriwciftohr tthhee mine-tursice fmroomdetlhdeepcrreeavsioeus,stthimeiens-upasen.mIfotdheel Log mSoadveelA luP lsedoAm andB tsbeagT lfroeodm itrcnuodpo is replaced by the continuously retrained model from the Pul secondary network. Algorithm 1 resumes the implemen- Model promdoudcetilon Model tation in each time span. registry deployment
The critical issue is now about the selection of the met- Figure 3: The enriched workflow, derived from Figure 1, where
ric according to which a drift is detected . Usually, it is re- the steps are suitably orchestrated.
quired to select a decision threshold for a deployed model
to achieve some action (e.g., to predict true or false). One
traditional method in ML is picking a threshold from a set As a result, we choose the following tools (Figure 3):
of potential thresholds to get good tradeofs on specific
metrics, such as accuracy and sensitivity. Nonetheless,
frequently such thresholds are manually set. Therefore,
if a model updates new samples, the previously manually
set threshold may be weak. Manually correcting multiple
thresholds across numerous models is breakable. One
mitigation approach for this issue occurs in [
In our approach, we detect drifts by measuring all the performances via AUC, which is a benchmark for binary
classification. Given that: = + and
= + , the Area Under the Curve AUC is computed as: − (1 − ) Thus, according to the metrics, to detect drifts correctly AUC has to be evaluated as new data are processed. • Airflow orchestrator: Airflow is a platform created to write, plan and monitor workflows programmatically. Airflow features scalability, dynamicity for pipeline generation, and extensibility to define own operators. Airflow orchestrates Model A initialization, Model A training and testing, Performance estimation, Model B update. • MLflow model registry: MLflow is an opensource platform for end-to-end ML lifecycle management. It allows one to track experiments, manage and deploy models, and host MLflow templates as a REST endpoint. MLflow is the repository of all the identified models ( Model registry). • BentoML deployment: BentoML is a highperformance framework for serving and deploying ML models either in an ofline or online fashion. It supports multiple ML frameworks, such as Tensorflow, Keras, and many others. BentoML is used to deploy the model in the online production system (Model deployment).
in a binary classification problem: predicting an output value according to some incoming values.
Given the current CoViD-19 pandemics and also in the
light of [19, 20], we choose a dataset containing data from
blood examinations from the Hospital Israelita Albert
Einstein in São Paulo, Brazil [
We select the hyperparameters of our neural network based on a grid search using the Keras Tuner library [21]. In particular, we use a HyperBand Tuner [22], an algorithm designed for hyperparameter optimization with early stopping for fast convergence.
As shown in Table 1, we discover some potential hyperparameters for our one-hidden-layer neural network. The model is, however, supported by a learning rate scheduler with a patience level of 15, monitoring the degradation of the validation loss.
Hyperparameter
Learning rate
Neurons Activation function
Optimizer Dropout
Value 1e-4 224 Relu Adam 0.2
approach to blood examinations from [
training and validation data are from Feb 25ℎ, 2020 to Mar 25ℎ, 2020 (validation split is 0.8); model testing data are from Mar 26ℎ, 2020 to Aug 11ℎ, 2020. Approximately, 5 months of the evaluation samples include the evolution of the CoViD-19 virus and a concentration of RT-PCR positive test results.
The algorithm for model identification is a neural network, which uses a grid search. The data split is a K-fold
ploy it and run it on new incoming data, i.e., from Mar 26ℎ, 2020 to Aug 11ℎ, 2020. As the model drifts away from the ground truth, we mark that timestamp as 1 drift, 2 drift etc., and update the model consequently.
Figure 4 reports drift detections over time, where drifts are gradual and not recursive or sudden [11]. The more data come in, the higher the chance that the model must be replaced by a more recent one. This is particularly true in the earlier stages of the experiment: then, drifts become less “dense”. Whenever the AUC detects a drift, the new model continuously retrained by the secondary network replaces the previously deployed one.
We compare our model with some classic ofline and online models: we consider the same training data and the same validation data. We consider here some current ML models for binary classification, including Logistic Regression (LR), Random Forest (RF), XGBoost [23], and other ofline artificial neural networks; we also compare model (0.745) and 5% better than that of the best online model (0.756). A comprehensive view of the models indicates the primacy of online strategies. our approach with some standard online learning methods that are functional, including online LR and adaptive 5.3. Discussion RF [16, 24]. The base neural network of our approach We now explore the shifts in patient attributes over the comprises one hidden layer with a RELU activation func- pandemic, explaining the need for online models. Again, tion connected to a dropout layer to limit the overfitting. in the light of explainable AI [25, 26], we examine the Figure 5 reports about our approach, one online model advantages of a model replacement mechanism and the (online LR), and two ofline models ( RF and XGBoost), impacts of hyperparameters of the neural network. measuring their respective AUC. Figure 6 depicts the hemoglobin (Hgb) count during the 2 and 3 time spans, i.e. after the 2 and the 3 drift. During the 2 time span, positive (CoViD-19 0.85 = “yes”) and negative (CoViD-19 = “no”) outcomes are coupled to balanced Hgb count: i.e., positive and neg0.8 ative patients feature similar levels of Hgb counts. In CU the 3 time span, the Hgb counts of patients with a A negative outcome difer from the Hgb counts of patients 0.75 with a positive outcome: thus, during the 3 drift, the Hgb count can be a diferentiating feature between positive and negative patients. Because of the deteriorating 0.7 1 2 3 4 likeness of the patients during the 2 time span, the Drift detection instance trained models are instantaneously impaired, including our initial neural network. The rationale for this shift requires further investigation:however, a first guess could probably be in the direction of variants on the side of the CoViD-19 virus, where variants lead to changes in the hematological examinations.
However, our approach and the online logistic regres- (ML) to infer knowledge from huge amounts of data is sion model support a robust AUC over time despite oscil- a challenging endeavour. In this context, significantly lations. The proposed approach yields the most elevated altered data – aka concept drift – and their efects on the AUC in the third frame. A comprehensive comparison of forecast quality are limiting a more widespread of ML. all the models used is presented in Table 2. In this paper, we focus on a novel concept of the drift
For the average AUC over all the time spans, our ap- handling methodology for ML systems in a real-world proach (0.806) performs 3% better than the best ofline environment. We build an online deep learning system and the best online models (0.782). The distinctions are for forecasting outcomes, using MLOPs practices and more relevant in the third frame, where our approach tools, and aim at keeping the in-use model constantly (AUC 0.794) is 6% higher than that of the best ofline up-to-date. Typically, a vital issue of the approach is ) L d / gm15 ( n i b o l g o m eH10 2 drift
No
Yes 3 drift the choice of the time interval between two successive updates of the in-use model. Manual decisions regarding the length of the windows are uncertain and impractical.
Thus, we keep a secondary network constantly updated by new incoming data: as the performance of the in-use model, measured in terms of area under the curve (AUC), are worsening, the model from the secondary networks jumps in, replacing the currently in-use model.
As a validation scenario, we deploy our approach on
real-world clinical data collected during the CoViD-19
pandemics by the Hospital Israelita Albert Einstein in
São Paulo, Brazil [
The generality of our approach contains some particular restrictions. Despite these favorable outcomes, the results are based on one dataset, only. Identifying other real-world datasets with incremental concept drift practices will add value and reliability to our approach. Due to its nature, our approach is appropriate for handling incremental concept drift. Sudden or reoccurring concept drifts will likely demand a diverse strategy, such as changing between two utterly distinct forecasting models, e.g., one model for the everyday problems and one for the harsh circumstances [16], or one forecast model for summertime and one for wintertime, respectively.
Consequently, more analysis is required to explore the correct matching of drift-handling scenarioes. Also, developing diferently-sized detection windows on the forecast implementation requires additional study. Finally, the triggered adaption method executed in this work is founded on the belief that actual labels are obtained soon after the model computes a forecast, but this assumption may be not always valid or feasible.
Acknowledgments G.P. is partially funded by the EU H2020 program: “PERISCOPE: Pan European Response to the ImpactS of CoViD-19 and future Pandemics and Epidemics” (grant n. 101016233).