The work is devoted to resolving the contradiction between the speed of modeling nonlinear dynamic objects and the accuracy of constructing models in the form of neural networks. The purpose of the work is to reduce the time required to build nonlinear dynamics continuous-time models in the form of neural networks while ensuring the specified modeling accuracy. This purpose is achieved by developing a modeling method based on the superposition of a set of pre-trained neural networks (support models) that reflect the main properties of the subject area. The scientific novelty of the developed method lies in using pre-trained neural networks with time delays as support models for modeling nonlinear dynamic objects. Unlike existing pre-training methods, the developed method allows building simpler models with reduced training time while ensuring the specified accuracy. The practical benefit of the results of the work lies in the development of the support models method algorithm, which allows significantly reducing the training time of neural networks with time delays without losing the accuracy of the model. identification; nonlinear dynamics; pre-training; neural networks training speed 1 ICST-2025: Information Control Systems & Technologies, September 24-26, 2025, Odesa, Ukraine Corresponding author. These authors contributed equally.
The current stage of modeling development, which is mainly based on using intelligent technologies,
is marked by a number of requirements from practice both for high accuracy of models and for the
speed of their construction [
Achieving high accuracy of nonlinear dynamics continuous-time modeling today is carried out
through using machine learning methods, in particular, neural networks (NN) [
The problem of increasing the speed of modeling remains one of the most urgent, especially in
industries related to the personalization of models, which must adapt to changes in user behavior or
the environment (e.g., in authentication tasks, biomedical applications, human-machine systems),
while operating in real time [
It should be noted that various approaches are being actively researched as part of efforts to
accelerate NN learning. The most common of these are based on regularization and normalization
methods, which promote faster convergence and reduce the likelihood of overfitting [
This direction seems promising in modeling objects with a high degree of internal complexity
and interaction, first of all, nonlinear dynamic objects [
The purpose of the work is to reduce the time for building nonlinear dynamics continuoustime models in the form of NN while ensuring the specified modeling accuracy by developing a method based on the NN models pre-training.
To reach this goal, the following tasks were formulated.
1. Development of the modeling method based on pre-training by superposition of a set of pretrained NN (support models) reflecting the basic characteristics of the subject area.
2. Building of support models in the form of NN reflecting the basic nonlinear and dynamic characteristics of the subject area.
3. Study of the speed of modeling complex nonlinear dynamics using the developed method of support models.
The approach to pre-training NN in practice faces a number of significant limitations [
Let S be the domain for which there is enough labeling data NS (DS dataset): θT 0 = θS : arg min LT ( fθT (x jT ), y jT ) = EθT
tT where LT is the loss function adopted for the target model.
DT={(xjT, yjT)}, (2) where xjT is the vector of the independent variables, yjT is the corresponding target variable (label), j NT.
Let f T T parameters, trained on DT dataset, which ensures the accuracy E T and the duration t T of target model training.
S of the general model. Using them as starting values T0 during training, the target model f T, a specified accuracy level E T is achieved in the minimum time t T:
DS={(xiS, yiS)}, where xiS is a vector of independent variables, yiS is the corresponding target variable (label), i NS.
Let f S be the general NN model with S parameters, which is trained on DS dataset.
Let T be the target task in the subject area S, for which there are marked-up data of limited NT size (DT dataset): (1) (3)
To accelerate the construction of general models of nonlinear dynamics, a new approach is proposed in the work. It consists in using a set of separate pre-trained NN (support models), each of which reflects separate basic characteristics of the domain.
To construct a set of support models, a set of support datasets DRk is used (k g, g is the number of basic characteristics of the domain). Each of the support datasets DRk describes a separate basic characteristic of the domain. On the basis of these datasets, support models f Rk with the parameters
Rk are built. By combining support models that reflect the characteristics of the target problem, a general model f S is built. It has a set of specified properties (nonlinear and dynamic). The target model f T S, DT) is built by pre-training the general model f S obtained on the basis of the set of support models f Rk on the DT dataset.
This approach allows to preserve the advantages of pre-training, since support models obtained once can be repeatedly used for different domains and target tasks, significantly reducing the total time and resources for training models without collecting additional data [17-19].
The structural scheme of the training process based on support models is presented in Fig. 1.
The algorithm of the suggested method consists of the following steps.
Step 1. Selection of basic domain properties and formation of a set of datasets DRk reflected the selected properties.
Step 2. Construction of support models set f Rk in the form of separate NN corresponding to the established properties of the domain. Training of built models based on the generated datasets DRk.
Step 3. Determination of the list of p properties of the target problem from the set of g basic properties of the domain and construction of the general model f S based on the superposition of the corresponding support models f Rh (h p, p g) obtained in Step 2.
Step 4. Training of the target model f T S, DT) based on the general model f S obtained in Step 3.
Step 5. Determination of the accuracy indicators E T and training time t T of the target model f T. In case of unsatisfactory quality indicators of the target model f T, the control transfers to Step 2 to
Rk of the support models f Rk, and, if necessary, to Step 1 to correct the set of basic properties of the domain and the set of datasets DRk, reflecting the selected properties.
The basic properties of the domain in the work are understood as the characteristics of objects that reflect the essential features of their behavior. These properties can include real or abstract parameters that are important for solving the modeling problem [20].
The procedure for selecting the basic properties of the domain and generating datasets DRk reflecting the selected properties is as follows.
1. Defining the range of tasks to be solved in a given domain; analyzing properties that are important for objects in a given domain, significantly affect the results of modeling and should be taken into account when forming the dataset DS.
2. Determination of signal types (for example, periodic, random, pulsed) that best reflect the properties of the objects under study.
3. Formation of the dataset DS based on the list of basic properties of the domain established in paragraph 1, the set of input signals and reactions of the object formed in paragraph 2.
4. Segmentation of the dataset DS into separate datasets DRk according to the defined list of basic properties of the domain.
In the above sequence of steps, the task of determining the type of signals that best reflect the properties of the object under study remains the least formalized. Automating the selection of support models is crucial for scaling the method to complex, multidimensional dynamical systems and bridges the gap between human intuition and automated processing, allowing the application of machine learning methods for objective identification and segmentation.
To automate the selection of support models, improve the reproducibility and reduce the subjective factor of the support model method, this paper proposes two approaches: input data clustering [21] and feature contribution analysis [22].
1. Clustering-based approach: to automatically segment the overall DS dataset into DRk subsets that clearly demonstrate elementary properties of the object under study.
2. Feature contribution analysis approach: to identify and quantify the underlying properties of the subject area Pk.
To modelling nonlinear dynamics, the work uses time-delayed NN (TDNN) [23]. Due to their simplicity and versatility, TDNNs are most widely used in modeling problems of nonlinear dynamic objects. In practice, the TDNN structure is most often used, consisting of three layers: input, hidden, and output [24]. The size of the layers in this TDNN structure is determined as follows: • • • the input layer consists of M neurons and is responsible for the memory (dynamic characteristics) of the model, the hidden layer consists of K neurons and is responsible for the nonlinear characteristics of the model, the output layer contains the number of Y neurons, which is equal to the number of outputs of the model.
For each support model, a labeled data set DSi={(xPkHj(t), fVi[xPkHj(t)])} is formed based on the signals x(t) at the input of the object and the responses y(t)= fVi[x(t)] at its output. Typical signals are often used as input signals: impulse x(t)=a (t), step x(t)=a (t), linear x(t)=at, and harmonic x(t)=a sin(t) signals of various amplitudes a ∈ (0, 1]. Time delays at the input are implemented through shifts in time series and the inclusion of previous values in the input vector.
The main steps for implementing time delays and forming a training sample are as follows: • • •
Step 1. Selecting the number of delays. Determine the number of delays M that will be used. Step 2. Forming the input vector. For each time moment tk, the input vector is formed as a sequence of current and previous values (2.1).
Step 3. Transferring delays to the network. Each formed input vector is transmitted to the input layer of the neural network, where it is processed as a regular input.
The algorithm for forming a training sample with time delays in pseudocode is shown below. Algorithm 1: time_delay 1: Input: DSi, M, x(t), y(t) 2: Output: x, y 3: foreach DSi as x(t), y(t) 4: for k =0, ..., T-M+1 do 5: xk x(tk), x(tk ), x(tk ), ..., x(tk M+1)] 6: yk y(tk) 7: end for 9: end foreach
After completing the process of pre-training the set of support models f Rk, a general model f R is built on their basis. This model is composed of a set of p support models f Rh that correspond to the available basic properties of the object. The selection of support models that reflect the basic properties of an object is generally subjective. To reduce subjectivity and increase the reproducibility of the selection of support models, it is advisable to use methods of clustering input data or feature contribution analysis.
After completing the preliminary training process for the family of support models that correspond to the existing basic characteristics of the object, a rough model f S(DS) is constructed based on them. Assuming that the general f S and support f Rk models are constructed in the form of NN with the same structure (dimension of the parameter vector dim S)=dim Rk)), the definition of the general model is reduced to calculating the arithmetic operations on corresponding components
Rk.
At the same time, several approaches to the superposition of support models are considered: additive superposition is used when each support model is responsible for independent aspects of the system (e.g., dynamics and environmental impact). The outputs of the support models f Si(DSi) are determined based on the calculation of the arithmetic mean of the corresponding components of the parameter vectors of the support models Sv: where i are the indices of the corresponding elements of the
S Rk models. multiplicative superposition is used in the case of interacting processes (e.g., where some processes modify others). In this case, the outputs of the support models f Sv(DSv) are multiplied: vectors of the • • • combined superposition methods use complex combinations of support models, such as weighted sums of outputs of nonlinear functions to combine the outputs of several models, the application of nonlinear functions to the outputs of individual models, etc. Such methods include determining the parameter vector S of the approximate model as the maximum value among the corresponding components of the parameter vectors of the support models Sv: θiS = 1 h
θiRk h k=1
b θiS = θiSv v=1 (4) (5) θiS = max(θiSv ), v = 1,b (6)
Thus, the advantage of forming an approximate model using the support models method is the absence of a training procedure, which reduces computational complexity and significantly speeds up the process of building an approximate model. At the same time, the dimension of the approximate model f S (the dimension of the parameter vector S) remains the same as in the base models, i.e., the complexity of the approximate model does not increase. Another advantage of forming a general model using expressions (4)-(6) is the absence of a training procedure, which significantly speeds up the process of constructing a approximate model.
The algorithm for synthesizing a general model based on the superposition of support models is given below.
Algorithm 2: general_model_sp 1: Input: V, H, P, f (DSi), DSi, M, K 2: Output: f (DS) 3: foreach V as Vi :f (DSi) 4: DSi dataset_formation(Vi, H, P, DSi) 5: f (DSi init(M, K) 5: f (DSi train[f (DSi), time_delay(DSi)] 6: end foreach 7: for i =1, ..., v do 8: f (DS sp[f (DSi)] 9: end for
Here, the init function initializes the model structure, train function learns the model, time_delay function forms a dataset with time delays, sp function perform a superposition of support models according to one of expressions (4) (6).
The study of the support models method is carried out on the example of a nonlinear dynamics test object. A simulation model of a test object in the form of a sequence of a nonlinear link with saturation and a dynamic link of the first order [23, 24] is shown in Fig. 2.
As typical characteristics from the set of properties of the domain of nonlinear dynamics, a nonlinear characteristic in the form of saturation and a dynamic link of the first order were chosen to describe the behavior of the test object. For the test object, a labeled DS dataset generated on the base of signals x(t) at the input of the object and the responses y(t) at its output. The inputs are pulsed x(t)= (t), stepped x(t)= (t), linear x(t)=at and harmonic x(t)=asin(t) signals of different amplitudes a ∈ (0, 1).
Based on the dataset DS, two support datasets are formed: • • input stepped signals x(t)= form of saturation DR1;
(t) and responses y(t) of the object with nonlinearity in the • • • • • input stepped signals x(t)= (t) and responses y(t) of the object in the form of a dynamic link of the first order DR2.
The experiment consists in studying the training speed of the target model of the test object, built by various methods: affiliation mark: a superscript number followi based on the general model f S(DS), pre-trained on the general DS dataset; based on individual support models f R1(DR1), f R2(DR2), pre-trained on datasets DR1 and DR2; based on the general model f R(f R1, f R2) in the form of a superposition of support models f R1(DR1) and f R2(DR2).
The structure of the target model f is chosen to be identical to the model based on the pre-trained general model f (DS), the support models f (DR1), f (DR2) and the superposition of the support models f (f , f ) and is a three-layer TDNN.
According to the results of additional research, the number of neurons M=30 in the input and K=30 in the hidden layers was adopted.
The model is trained by the method of backpropagation of the error with updating the network parameters by the Levenberg-Marquardt method. Pre-training is limited to 50 epochs to prevent overtraining and preserve the ability to adapt.
Fig. 3 shows the dependencies of mse loss functions on the number of training epochs for target models built on the basis of the general model f (DS), support models f (DR1), f (DR2) and superposition of support models f (f , f ).
Results of an experiment to study the learning speed of a target model of a test object built on the basis of a general model, on the basis of separate support models, and on the basis of a superposition of support models, are presented in Table 1.
The experiment shows the advantage of using support NN at the modelling nonlinear dynamics, namely, a significant reduction in the training time of the TDNN model (by 4.6 times) compared to the pre-trained general model with comparable accuracy of both models.
Applying separate support models as general ones can also reduce the training time of an accurate model (by 1.8 times).
The paper successfully solves the problem of reducing the time of building nonlinear dynamics continuous-time models in the form of neural networks while ensuring the specified accuracy of modeling. To resolve the conflict between the accuracy of modeling nonlinear dynamic objects and the speed of model construction, a modeling method was developed based on pre-training through the superposition of support models that reflect the basic properties of the subject area.
The effectiveness of the developed method for modeling nonlinear dynamics was proven when solving the problem of modelling a test nonlinear dynamic object. The experiment demonstrates a 4.6-fold reduction in the time of building a target model using support models compared to the traditional modeling method based on pre-training. The advantages of the proposed approach are the ability to quickly adapt to changing operating conditions, high speed of building the target model while ensuring the specified modeling accuracy. In addition, the developed method allows improving the efficiency of model training in the lack of labeled data for the target task. The disadvantages of the proposed approach, inherited from methods based on pre-training, are the dependence of the modeling results on the quantity and quality of data of the target dataset.
The practical limitations of the application of the proposed method are the a priori need for support models built on a sufficient amount of qualitative data. Insufficient data or poor data quality can significantly reduce the accuracy of support models and, as a result, significantly reduce the training time of an accurate model.
Thus, the area of effective application of the proposed method is allocated: lack of marked data of the target task in the presence of a general dataset of sufficient size; no significant discrepancies between the characteristics of the general and target datasets.
To improve and expand the scope of application of the support models method, it is necessary to take into account the real conditions of the external environment by expanding the experimental part at different levels of noise distortion and under conditions of time drift of the observed parameters. In order to fully assess the potential of the proposed method and determine its place among advanced solutions in further research, it is planned to expand the range of test objects, including systems with different dynamics, multidimensional systems, objects with delays, etc.
During the preparation of this paper, the authors used Google Gemini to check the coherence of the text of the article and to identify technical inaccuracies. The authors are solely responsible for the content of the publication. [16] J. Siebert, L.Joeckel, J. Heidrich, et al., Construction of a quality model for machine learning systems, Software Qual J. 30 (2022) 307 335. doi: 10.1007/s11219-021-09557-y. [17] P. Karampiperis, N. Manouselis, T. B. Trafalis, Architecture selection for neural networks, in: Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02.
Honolulu: HI, USA, 2, 2002, pp. 1115 1119. doi: 10.1109/IJCNN.2002.1007650. [18] K. Pal, B. V. Patel, Data classification with K-fold cross validation and holdout accuracy estimation methods with 5 different machine learning techniques, in: Fourth International Conference on Computing Methodologies and Communication (ICCMC). Erode: India. 2020, p. 83 87. doi: 10.1109/ICCMC48092.2020.ICCMC-00016. [19] K. Nakamichi, et al., Requirements-driven method to determine quality characteristics and , in: IEEE 28th International Requirements Engineering Conference (RE), Zurich, Switzerland, 2020. p. 260 270, doi: 10.1109/RE48521.2020.00036. [20] T. Warren Liao, Clustering of time series data a survey, Pattern Recognition, 38 11 (2005) 18571874. doi: 10.1016/j.patcog.2005.01.025. [21] G. Leylaz, S. Wang, JQ Sun, Identification of nonlinear dynamical systems with time delay, Int.
J. Dynam. Control. 2022. 10, 13 24. doi: 10.1007/s40435-021-00783-7. [22] L. Wenyuan, L. Zhu, F. Feng, at al., A time delay neural network based technique for nonlinear microwave device modelling, Micromachines. Basel, 11 9 (2020) 831. doi: 10.3390/mi11090831. [23] O. Fomin, et al. Interpretation of dynamic models based on neural networks in the form of integral-power series. in: Arsenyeva, O., Romanova, T., Sukhonos, M., Tsegelnyk, Y. (eds). Smart Technologies in Urban Engineering. Lecture Notes in Networks and Systems. Springer. Cham. 536 (2022) 258 265. doi: 10.1007/978-3-031-20141-7_24. [24] O.O. Fomin, A.A. Orlov, Modeling nonlinear dynamic objects using pre-trained time delay neural networks. Applied Aspects of Information Technology. 7 1 (2024) 24 33. doi: 10.15276/aait.07.2024.2.