Replay-based continuous learning (CL) methods are utilized to adapt deployed deep learning (DL) models to evolving data while minimizing catastrophic forgetting. However, the primary challenge is that most CL methods in the literature do not focus on real operational data and regression tasks. These key aspects are explored in this study, which applies and evaluates replay-based CL method in a machine learning pipeline that uses real operational data to retrain a DL model for a maritime use case. This pipeline was adapted to the application context based on experiments and discussions with domain experts, by enhancing the sample management. Our results show that even with a small set of replayed samples, the model performance can be improved and catastrophic forgetting can be limited.
With the growing integration of technology in the
maritime industry, data-driven approaches utilizing operational
and environmental data from sensors onboard have become
prevalent for machine learning (ML) -based performance
monitoring [
CL characterizes learning from dynamic data
distributions and describes the ability of ML models to continuously
learn from newly acquired data while retaining previously
learned knowledge [
Several CL methods have been proposed to address
catastrophic forgetting, broadly categorized as
parameter isolation/architecture-based, regularization-based, and
replay-based methods [
The main contributions are twofold. First, we implement a replay-based CL method applied to a pre-trained ANN for FOC prediction using real-world ship sensor data. Additionally, a comparative analysis is performed alongside two baseline models, one trained with all_data and the other with current_data. Second, the paper proposes an approach to selecting an optimal exemplar set in the replay-based CL method, taking into consideration the requirements of the target application.
CL aims to extract knowledge from an (infinite) stream of
data to gradually extend the existing knowledge of the model
without catastrophic forgetting [
CL methods are extensively described in the literature
(we refer the reader to [
All historical training data
generation data current_data replay_data all_data
Replay-based CL methods select and store a small subset
of past data (i.e., exemplars), in limited storage space to
be replayed during training sessions along with new data
[
The ML training pipeline depicted in Figure 1 illustrates our experimental setup for testing the replay-based CL method. The implementation code is available on GitHub1. 3.1. Training Data and Pipeline The training data consisted of historical sensor time series data collected from an anonymized container ship over two years. The historical experiment dataset includes over 175,000 rows, with data points spaced 5 minutes apart. The dataset’s columns are detailed in Table 1. Utilizing a timestamp column, the experiment dataset was divided into five 1https://github.com/UH-MLOps/replay-cl-in-ml-pipelines 80000 e z i tS60000 e S g inn40000 i a r T 20000 0 equal parts, each serving as a distinct training dataset for the pre-trained ANN model. These divisions are henceforth referred to as training generations. Data distribution plots for each training generations are in the GitHub repository.
In the replay-based approach, samples from previous
datasets are kept in an exemplar set and replayed during
model training alongside new data. The experiments are set
up to first train an ANN model with the data of generation
zero. In our study, the architecture of an optimal ANN model
for predicting ship FOC, as discussed in [
The training pipeline manages data generations, handles training data and exemplar sets, and follows a conventional training process, with each dataset split into training and validation sets using an 80% split ratio. The models are trained from scratch, with hyperparameters, such as the number of hidden layers, remaining unchanged. The number of training epochs is fixed at 30 for all models, except for the generation zero model, which undergoes training for 80 epochs. The trained models were evaluated based on their calculated mean squared error (MSE) when predicting on the validation sets for each generation. The architecture and hyperparameter configurations of the pre-trained ANN model are presented in Table 1.
Equal Subset Size
Increasing Subset Size 5000 4000 3000 2000 1000 0 Gen 1 samples Gen 2 samples Gen 3 samples Gen 4 samples Gen 5 samples Gen 6 samples 3 4
Generation 1 2 5 6 1 2 3 4 Generation 5 6 3.2. Lower and Upper Boundary Models The main diference between all models is the amount of training data used during the retraining runs. The upper boundary model, named all_data, is trained on the current generation data and all historical datasets of previously seen generations combined. The all_data model is expected to perform best since it is trained on all available datasets.
The lower boundary model, named current_data, is trained only with the new dataset available in the current generation. This model acts as the lower bound and is expected to perform poorly compared to the others, demonstrating the efect of catastrophic forgetting because training data is limited to the current generation. For this model, the training pipeline is set up with an exemplar set size of zero such that no historical samples are collected. Figure 2 shows the sizes of the training sets that are used for each model generation. The training set always consists of the current generation training data plus the samples stored in the exemplar set. The diferent sizes of the datasets are expected to influence the runtime of each training and the resulting model’s performance. For reproducibility, the pipeline ensures that all libraries that use randomness are seeded before the experiment runs. 3.3. Replay-based CL Approach In each generation, , the model is trained based on a new dataset combined with the exemplar set 1:− 1 = {1, 2, . . . , − 1}. The exemplar set comprises selected samples from the previous training datasets. The training objective for generation is minimize the total loss : =
∑︁ {,}∈∪1:− 1 (, , ) is the loss function that calculates the model error given input data , the expected output , and the model parameters . This error is calculated for every input in the combined dataset and summed to give the total loss . Once the model is trained, the current dataset gets sampled for exemplars, which are added to the exemplar set. The training pipeline automatically evaluates the trained model before the next training generation starts, where the same model is retrained on the next dataset. 3.3.1. Exemplar Set and Selection The exemplar set is used for storing and replaying samples collected from previous datasets. An important property is that the exemplar set always stores a fixed amount of samples, meaning old ones are dropped from the set when new samples are selected. An exemplar set 1:− 1 at generation consists of subsets that contain samples from previous generations data 1:− 1 = {1, 2, . . . , − 1}, where each subset contains samples from the -th generation dataset. These subsets’ combined size equals the total size set for the exemplar set.
The choice of is critical when setting up the pipeline.
To reduce the amount of data and computation during the
training process, should be as small as possible while
still fulfilling the goal of mitigating catastrophic forgetting
[
In literature, the exemplar set is often implemented such that each subset has the same number of samples: || = . To always maintain the total size of the exemplar set, the subsets’ size must be updated after each new sampling. This means subsets already in the exemplar are reduced to free up space for adding new exemplars from the current dataset. The downside of this approach is that the number of new samples introduced gets smaller with every generation, limiting the influence of each sample set during future training sessions.
For this study, domain experts suggested that in the context of ship performance prediction current historical data is more relevant than older data when training a model. Given this knowledge, the exemplar set was extended with two new functionalities. First, a limit on the number of subsets kept in the exemplar set was implemented. This way, the number of subsets can grow up to stored subsets before the oldest subset gets dropped when new data is sampled, keeping the number of stored sets constant. This also ensures static subset sizes that will stay constant over multiple generations when the limit of is reached. Secondly, a diferent operation mode for the exemplar set was implemented. In this new mode, the subsets are not kept at equal sizes, and newer data is prioritized by storing more samples from recent datasets. The following formula was implemented to calculate the subset sizes at generation when the exemplar set contains − 1 subsets: 2 || = * ( − 1) , ∈ [1, − 1]
For example, in generation = 4 the calculated subset sizes are |1:− 1| = |{ 61 , 62 , 36 }| = .
Figure 3 shows these functionalities in our experiment setup. The setup has six generations of data, the total exemplar set size is = 5000, and the limit is set to = 5 generations that are kept in the exemplar set. After generation 5 the oldest subset for generation 1 is removed because of the set limit.
This section presents experiment results collected with the training pipeline. In all belwo evaluations and plots, the base model represents generation zero, and the retrained models represent generations one to four. Thus, generation zero shows the performance of the shared base model, while generations one to four show the assessments of the retrained models using the diferent training setups. 4.1. Replay-based CL Model Performance (RQ1) RQ1 evaluates how the replay model compares to the other two baseline models trained either on all_data and current_data. Figure 4 shows the performance (MSE) of the three models against the current generation validation set. 0.06 ) E SM0.05 (r o r rE0.04 d e r au0.03 q S n ae0.02 M 0.01
The replay model has the lowest MSE in generations one and two, but it gets overtaken by the all_data model in generations three and four. The plot shows the best performance for all three models in generation three and the poorest in generation four. The poor performance is likely due to the changes in data patterns/distributions in the training and validation datasets. In our GitHub2 repo we provide distribution plots for the three most influential sensor measurements, that show a high overlap between the distributions of training and testing data in generation three, and a diferent distributions between training and test data in generation four. In summary, the replay-based CL approach has a slightly better but comparable performance to the current_data approach. The all_data approach is worse in the first generations compared to the replay and 2https://github.com/UH-MLOps/replay-cl-in-ml-pipelines current_data approach but then improves in the third and fourth generations, beating the other two approaches.
When considering a production setup, the runtime of a training run is significant to evaluate since a long training process binds computing resources, which could accumulate high costs of operation when the training pipeline is run repeatedly over a long period, performing many lengthy retrainings. A model’s training runtime depends on the model’s architecture, i.e., the number of neurons in the model and the amount of training data used.
Figure 5 shows the average training runtime for each training approach. The experiments were conducted on an average laptop with no specialized hardware, e.g. GPU, which could speed up the training process. The training runtimes for the all_data model are increasing in every generation, which is explained by the increasing training dataset in every generation (see Figure 2). While this is an unfavourable behaviour regarding computing time, the results in the section 4.2 show that the all_data model performs the best for mitigating catastrophic forgetting. This shows an important trade-of between training runtime and model performance.
The training runtimes for the replay and current_data models are constant in each retraining since their training dataset are also of constant size. The only diference between the two models is the slightly increased runtime of the replay model, which can be explained by the 3500 rows of exemplars added to the current generation training data. 7 6 ) in5 m ( e4 m i T iig3 n n r2 a T 1 0
2 Generation 3 4 RQ2 evaluates the capability of the replay-based CL approach to mitigate catastrophic forgetting while also generalizing its learned knowledge efectively to predict unseen future events successfully. The evaluation of catastrophic forgetting is done by testing the replay model on historical data. If the performance is lower than before the retraining, the model has ’forgotten’ what it has learned. To evaluate this, each model generation is evaluated against the validation data of generation zero. The experiment results assessed by the MSE are shown in Figure 6.
The replay model in generation one performs slightly worse than the other two models but then improves its performance in later generations. This shows that catastrophic forgetting can be limited even with a limited exemplar set that gets replayed during training. Comparing the performance of the replay and the all_data model, it can be seen 0.07 0.06 that the replay model performs only slightly worse in most generations, which is impressive considering the significant size diference of the training datasets between the models.
The Evaluation of model generalization is done by the performance of the replay model in predicting the unseen validation data of generation four. Figure 7 shows the result of these experiments.
The experiments show good generalization results for all three models, which confirms that all models successfully learn the data patterns of the sensor data. The replay model generalization performance is close to the all_data model and overall better than the current_data model. The worst overall performing model of the retrained models is the current_data model, especially in generation three, with an error of 0.06. Interestingly, there is a significant improvement in the performance of the replay model compared to the current_data model, considering the only diference between the two are the 3500 rows of historical data available to the replay model. This limited set of historical samples brings the accuracy of the replay model close to the performance of the all_data model, which has access to all historical data for training. 4.3. Exemplar Set Size (RQ3) RQ3 investigates the optimal exemplar set by conducting multiple experiments with diferent sizes to evaluate the selected exemplar set size. The selected size of 3500 rows is compared against a set half its size with 1750 rows and a set twice its size with 7000 rows. Since the primary goal of replay-based CL is to reduce the possibility of catastrophic forgetting, the same evaluation as in Section 4.3 is performed to assess the model’s performance on the validation data of generation zero. Figure 8 shows the prediction performance of three replay-based models.
0.055
Although the replay1750 model starts well in generation one, it quickly loses prediction accuracy in later generations, showing the efect of catastrophic forgetting in a similar pattern as the current_data model in Figure 6. This confirms the expectation that an exemplar set with only 1750 rows is insuficient in the scenario of our study. The replay3500 and the replay7000 model show better performances across all generations. The replay7000 model performs best except in generation three, showing a relatively constant mean squared error between 0.020 and 0.026 across all generations.
This section discusses the results and concludes the study with general observations. From the findings presented for RQ1 in Section 4.1, the replay-based approach showed good performance across all training generations. This is a significant result since the model’s main task in a production scenario is to accurately predict the ship’s current performance until new data is available. The replay-based CL approach has a lower error rate than the current_data approach in all training generations. The replay-based approach also has the smallest combined error over all training generations out of the three approaches. This shows that the samples in the exemplar set help the replay model successfully improve its prediction performance in contrast to the current_data approach with no access to historical data.
Interestingly, the all_data model performs worst out of
the three in generations one and two. A possible
explanation for this is that the all_data model likely has higher
model stability compared to the other two model types,
meaning it is less likely to change during training. This
stability is caused by the extensive dataset on which it gets
trained. Since all historical data is present during training,
the model is optimized to predict all data with an equally
high accuracy. In contrast to that, the replay approach (and
the current_data approach) focuses the training on the
current generation data and can therefore achieve a higher
accuracy on this smaller current dataset. Considering the
suggestions of the domain expert mentioned in Section 3.3.1
regarding the use of historical data for model training, it
can be argued that higher model plasticity seen in the replay
and current_data approaches is favorable, because it allows
the model to adapt closer to the current dataset. Generally,
changes in performance are expected in the context of
maritime ship performance prediction since ships experience a
broad range of weather conditions and operational settings
over their lifetime. If specific patterns are not contained in
a model’s training data, it is nearly impossible for the model
to accurately estimate a new operational condition [
RQ2 (Section 4.2) aimed to evaluate the efect of
catastrophic forgetting on the three compared models. In the
context of ship performance prediction, the model must
accurately predict a ship’s performance in various operational
conditions. Since not all conditions are encountered within a
limited timespan of the ship’s lifetime, the prediction model
must remember previously encountered conditions even if
they have not been part of the last training data. As
expected, the current_data model performed worst in this test
since it did not consider historical data during retraining and
successively forgot knowledge it had learned in earlier
generations. More importantly, the diference in performance
were compared between the replay and all_data models.
The replay model performed exceptionally, considering its
limited dataset compared to the all_data model.
Considering the domain context of the models, this result shows
successful mitigation of forgetting across many generations
when using a replay-based CL approach. The slightly worse
performance of the replay model is explained by the limited
amount of historical data the model has access to during
training. These two diferences in performance and size
of training data highlight the trade-of that has to be
considered when setting up a CL training pipeline, which is
also discussed by [
RQ3 (Section 4.3) considers diferent sizes for the
exemplar set used in a replay-based CL approach. Most literature
[
The experiments show that the best-performing model for
mitigating catastrophic forgetting is the replay7000 model,
which has the most training data. However, this model also
has the longest training runtime. It was decided to use the
replay3500 model with an exemplar set size of 3500 for the
experiments since its performance regarding mitigation of
catastrophic forgetting was considered as satisfactory and
also its quicker training time was determined as beneficial
for experiments on the limited available hardware. As
mentioned above and also supported by [
The work was funded by the VesselAI EU project3, enabling the collaboration between the Uni. of Helsinki and NAPA4.