Adaptive nonlinear neo-fuzzy bagging metamodel and its fast online learning procedure are proposed intended for optimal combining of the results of different computational intelligence systems that simultaneously solve the same problem. It is assumed that the data is processed by the members of the ensemble and by the metamodel online in real time. The proposed metamodel is intended for solving a wide class of Data Stream Mining problems under the conditions of data non-stationarity, and when the processing speed is of the utter importance. Simulation based on the short-term electric load forecasting problem confirmed theoretical results. The metamodel demonstrated significant improvement over the results of the member models.
Recently, computational intelligence has emerged as a powerful tool across various fields, particularly in Data Mining. It encompasses a variety of systems such as artificial neural networks, fuzzy systems, neuro-fuzzy systems, and neo-fuzzy systems. These systems are employed to tackle an extensive array of tasks. Just to mention a few:
On the other hand, every task can be addressed using various computational intelligence systems, each with distinct architectures, training principles, data requirements, and operational speeds, which leads to different data processing results. For instance:
Artificial Neural Networks (ANNs): Known for their high accuracy, especially deep neural networks, they require vast volumes of training samples (which may not always be feasible due to data scarcity) and computational resources, i.e. a lot of time and computing power for their training in multi-epoch mode.
Fuzzy Systems: These systems are known for their ability to handle imprecise or incomplete data, making them suitable for real-world problems where data may be uncertain. Neuro-Fuzzy Systems: Combining the strengths of neural networks and fuzzy logic, these systems offer both learning capabilities and robustness to noisy data.
Selecting the appropriate computational intelligence system for a given problem is an intricate
task that lacks a formal solution, primarily relying on the empirical knowledge of researchers and
occasional intuition. This process is not trivial due to the vast array of systems available, each with
its own strengths and weaknesses. To address this challenge, one promising approach is bagging [
However, nonlinear approaches to bagging have been largely overlooked, with only limited
research available in this domain [
The aggregation of system outputs poses another challenge. While linear combinations have been
extensively studied [
In light of these considerations, we propose a novel approach using the neo-fuzzy methodology
[
In summary, while choosing the right computational intelligence system remains challenging due reliance on experience and intuition, innovative approaches like bagging ensembles and adaptive metamodels offer promising pathways. The exploration of nonlinear solutions, particularly through neo-fuzzy approaches, holds the potential to advance our ability to effectively combine diverse systems for improved problem-solving in dynamic environments.
The architecture of the adaptive neo-fuzzy bagging metamodel is illustrated in Figure 1. This system
represents an enhanced version of the traditional neo-fuzzy neuron [
The metamodel receives input signals from ensemble members that work simultaneously to solve the same problem. Each member provides its output in the form of a time sequence: ( ), … , ( ), … , ( ), where represents the discrete time index = 1,2, … These sequences are organized into a vector format denoted as ( ) = ( ), … , ( ), … , ( ) , which is then passed to the layer of nonlinear synapses. (1) (2)
, which are determined through a learning process tailored to optimize performance.
At the output of each of the nonlinear synapses a signal is formed ( ) = ( ) = , ( ) = ( ) , … , Triangular functions in the following form are most often used as membership functions in ( ) =
( ) − ⎧ ⎪ , − ⎨ ⎪ ⎩0 otherwise, , − ( ) , − , ,
, , , , , if ( ) ∈ , , where , , , ,
, are coordinates of the membership functions centers, which are usually distributed evenly along the coordinate axes.
One significant benefit of utilizing triangular functions within fuzzy logic systems lies in their efficiency during the learning process. When processing incoming observations – denoted as ( ) – these triangular functions ensure that only two neighboring membership functions are activated at any given time. This means that, rather than recalculating or adjusting all relevant weights across the system, which would be computationally intensive, only a specific subset of synaptic weights needs to be updated.
By having only two neighboring functions triggered at each step of the learning process, we minimize the number of adjustments required – specifically, adjusting 2 synaptic weights per moment. This targeted approach not only reduces computational load but also enhances efficiency, making real-time processing more feasible. Moreover, this property contributes to scalability since it limits the complexity that can arise as data volume increases.
In essence, the use of triangular functions streamlines the learning process by confining adjustments to a localized set of weights and membership functions, thereby optimizing performance in dynamic or large-scale applications. This efficiency is crucial for maintaining responsiveness and accuracy in systems where resources are constrained or rapid data processing is necessary. can increase computational demands, which may impact real-time performance if not carefully managed. constraints:
The output signals of nonlinear synapses – ( ), … , ( ), … , ( ) are fed to an additional layer of tuned weights ∗, … , ∗, … , ∗ (which is absent in the standard neo-fuzzy neuron), that are present in most known bagging systems and are subject to unbiasedness ∗ = ∗ = 1, where – ( × 1 ) vector formed by ones.
Signals ( ) after passing through synaptic weights ∗ are combined in the output adder, forming the optimal output signal ∗( ) =
∗ ( ) = ( ) ∗, where ∗ = ∗, … , ∗, … , ∗ , ( ) = ( ), … , ( ), … , ( ) . (3) (4) ( ) = ( − 1) ∗( ) = ( ) ∗( − 1), which, after passing through the set of weights ∗, are combined in the output adder in the form where ( − 1), ∗( − 1) – estimates obtained based on − 1 previous observations.
Adaptive metamodel learning can be implemented based on errors backpropagation. First, the parameter vector ∗ is tuned, and then all the parameter vectors of nonlinear synapses , = 1,2, … , are tuned.
A standard quadratic learning criterion can be used to tune the vector ∗
Thus, in the process of its training, two sets of parameters are configured in the proposed metamodel: ( × 1) vector ∗ and ( + 1) vectors of parameters of nonlinear synapses , = 1,2, … , .
When the signal ( ) is fed to the metamodel, the following signals are formed at the outputs of nonlinear synapses (5) (6) (7) (8) (10) (11) (12) 1 ∗( ) =
2 subject to the unbiasedness constraints ( ) − ∗( ) ∗ = 1,
( ) ( ) = ,
( )‖ ( )‖ − ( − 1) ( ) we get the optimal learning procedure in terms of speed where ( ) – external reference signal, also used for all ensemble members training.
To solve the problem, quadratic programming can be used to find the saddle point of the Lagrange function ( ∗, ) = 1 ( ( ) − ( ) ∗) + ∗ − 1 , (9)
2 where is an undefined Lagrangian multiple, which in this case is also a tuned parameter.
To find a saddle point, it is convenient to use the Arrow-Hurwitz algorithm, which in this case takes the form ∗( ) = ∗( − 1) − ( )∇ ∗ ( ∗, ), ( ) = ( − 1) + ( ) ( ∗, )⁄ , or ∗( ) = ∗( − 1) + ( ) ( ) ( ) − ( − 1) , ( ) = ( − 1) + ( ) ∗( ) − 1 , where ( ) = ( ) − ( ) ∗( − 1) – learning error, ( ), ( ) – search step parameters that determine the convergence rate of the search process.
The process of tuning the parameter vector ∗ can be optimized for speed by appropriately
choosing the step ( ). By specifying this step in the form [
, ( ) = ( ) − ( ) ∗( ) = ( ) − ∗( ) ( )
= 1 2 ∗( ) =
( − 1) ∗ ( ) . ( ) = ( − 1) + ( ) −
( − 1) ∗ ( )
∗ ( )
∗ ( )
To adjust the parameters vector ( ), the same optimal Kaczmarz-Widrow-Hoff algorithm can
or its modification with additional smoothing properties, which has proven effective in training
be used in the form
neo-fuzzy neurons [
Introducing (
× 1) vector of tuned parameters of all nonlinear synapses ( ) = ( ), … , ( ), … , ( ), ( ), … , ( ), … , ( ), ( ) and a modified vector formed ∗( ) ∗( ) by all membership functions and adjusted output weights ∗ ( ) = ( ) , … , ∗( ) ( ) , ∗( ) ( ) , … , ∗( ) ( ) , … , ∗( )
, it is easy to write the transform implemented by the metamodel with the adjusted parameters vector ∗( ) in the form ∗( ) = ∗( − 1) + ( ) = ( − 1) + ( ) ∗( ) − 1 .
( ) ( ) ( ) − ( − 1) ( )‖ ( )‖ − ( − 1) ( ) ,
It is easy to see that in the absence of unbiasedness constraints, this procedure completely
coincides with the optimal one-step Kaczmarz-Widrow-Hoff algorithm [
After adjusting the output parameters vector ∗( ), we can proceed to adjusting the nonlinear synapses parameters based again on the quadratic criterion ( ) = ( − 1) + ( ) ( ) −
( − 1) ∗ ( ) ∗ ( ) , ( ) = ( − 1) + ∗ ( )
, 0 ≤ ≤ 1, where is a smoothing parameter defining a compromise between the filtering and tracking properties of the learning algorithm.
The proposed training procedure for the adaptive neo-fuzzy nonlinear bagging metamodel is designed for online information processing when data is received by the system in real time.
We applied the proposed bagging approach to the short-term electric load forecasting problem (STLF), specifically focusing on 1-step ahead forecasting of daily electric load in one of Ukraine’s regional power systems. To see, why bagging is the approach of choice to solve STLF problems, let’s start with a short overview of STLF itself.
Electric load forecasting is a critical task for utility companies, enabling them to manage electricity generation and distribution efficiently. Among various forecasting horizons, one-day ahead (onestep) forecasting stands out as particularly challenging due to its dynamic nature and the need for real-time accuracy.
Here are the most common challenges inherent to short-term electric load forecasting.
The first challenge lies in the sheer volume and diversity of data that must be processed to generate accurate forecasts. Historical consumption patterns provide a foundation for predictions, but this dataset is augmented by numerous other variables like weather conditions, economic indicators, calendar events, grid conditions, etc. The integration of these diverse data points is essential for accurate predictions but presents a significant challenge due to their different measurement scales, varying nature, and potential inconsistencies.
Electricity demand fluctuates continuously, shaped by human activities such as turning on appliances, working schedules, and leisure time consumption patterns. This dynamic behavior makes it difficult to predict with high precision even just one day ahead. Difficulties arise from time-of-day variations, weekday vs. weekend pattern differences, and seasonal changes.
The relationship between various factors influencing electricity demand is often nonlinear and
complex. Traditional statistical models, which rely on linear relationships, may struggle to capture
these nuances effectively. The impact of temperature on load isn’t always directly proportional; there
can be saturation points beyond which further changes in temperature don’t significantly affect
demand [
This complexity necessitates the use of advanced modeling techniques capable of handling nonlinear relationships, such as machine learning algorithms like Artificial Neural Networks and Support Vector Machines.
External factors beyond immediate control can disrupt even the most sophisticated forecasting models. These include, but are not limited to sudden weather changes, unforeseen events (e.g. sporting events, strikes, or other unexpected occurrences), technical problems in the grid, etc.
These uncertainties require forecasting models to be robust and adaptable, capable of incorporating real-time adjustments as new information becomes available.
The need for real-time processing and high-frequency updates imposes significant computational demands. To keep forecasts accurate, data must be processed quickly enough to reflect the latest conditions. Advanced models require substantial computational resources for training and updating as new data comes in.
This challenge is compounded by the need for scalability, ensuring that forecasting systems can handle increased data loads without compromising performance or accuracy.
Selecting the right forecasting model is a significant challenge due to the dynamic nature of load data. Different models perform better under various conditions, requiring careful selection based on historical performance and expected future scenarios. Ensuring that chosen models remain effective over time requires ongoing validation and adjustment as patterns evolve.
To overcome these challenges, experts employ a variety of strategies, including but not limited to:
The use of various classes of models – from linear regression to deep neural networks to find a better match to a particular forecasting task.
Online data processing – using real-time data feeds to update forecast as new information is received.
Ensemble forecasting – combining predictions from multiple models leverages diverse strengths and reduces reliance on any single model’s potential biases. This is the focus of this paper.
Hence, short term electric load forecasting is a multifaceted challenge requiring advanced techniques, robust computational infrastructure, careful model selection and tuning, dynamic data processing capabilities, and ongoing validation. The complexity stems from the intricate interplay of numerous variables, the nonlinear relationships between factors influencing demand, and the need for real-time accuracy. Applying bagging approaches can help cope with at least some of the mentioned challenges.
The original time series consisted of = 337 samples. We generated six forecast series ( = 6), each with 337 samples, using six different independent computational intelligence models. In this setup, we treated the time series as a data stream, meaning that both forecasting and metamodel operations were performed in an online mode. This approach ensured that the entire dataset was processed sequentially – sample by sample ( = 1,2, . . . , ) – without requiring multiple passes or divisions into training, validation, and test sets.
The original time series (see Figure 3) exhibited several distinct trends corresponding to different
seasons, periodic patterns (primarily weekly), sudden changes, and outliers. The presence of these
features made the forecasting task particularly challenging. Additionally, there was a significant
random component in the data because electric load in large power systems depends on numerous
external factors, many of which are inherently unpredictable or chaotic – for example, weather
conditions [
Given these characteristics – non-stationarity, noise, and complexity – the performance of any single forecasting model or method is unlikely to be consistently superior across the entire dataset. In other words, different models tend to perform better on specific parts of the series but underperform on others. This variability highlights a common challenge in time series forecasting: no single model or method dominates in all situations. It is precisely this kind of scenario where ensemble methods like bagging come into play.
By leveraging the strengths of multiple models through bagging, we aimed to improve overall forecasting accuracy by combining their predictions. The idea was to minimize errors that might arise from relying on a single model and instead capture more robust insights by aggregating results from diverse perspectives within the ensemble. This approach has shown promise in addressing the inherent limitations of individual forecasting methods while providing a more balanced and accurate prediction across the entire time series.
In our simulation, we utilize six specialized STLF models within an ensemble framework. Each of these models has unique inputs and distinct structural differences, which collectively contribute to a diverse predictive capability. Some models focus on historical weather patterns as inputs, while others prioritize calendar events like holidays that affect electricity usage. The diverse structures ensure that each model interprets and processes these inputs differently – some may use linear regression techniques, while others employ neural networks capable of identifying complex patterns.
Deploying six specialized STLF models with varied inputs and structures allows us to comprehensively capture the multifaceted nature of electric load data. This strategic diversity ensures that our ensemble can account for a wide array of factors influencing demand across different parts of the time series.
Figure 4 illustrates the time series data and forecasts for the past 30 days. While long-term trends are reasonably captured by all models, the variability in the data within shorter timeframes is difficult for every forecasting method used. As a result, none of the models demonstrate a significant advantage over the others.
We utilized the proposed metamodel to derive an optimal combination ∗( ) of the six forecasts from the ensemble member models. By visually inspecting the plot, it becomes evident that the combined forecast ∗( ) closely aligns with the actual data series ( ). This proximity is further corroborated by an in-depth error comparison presented in Table 1.
To evaluate the accuracy of our predictions, we employed the Mean Absolute Percentage Error (MAPE) criterion. Widely recognized in short-term electric load forecasting research, MAPE is a robust metric that quantifies forecast errors as a percentage of actual values, which makes a clear physical sense. The best performing ensemble member model achieved a MAPE of 4.858%. Through the application of the nonlinear bagging procedure, the MAPE was reduced to 3.3751%. This represents an approximate reduction by a factor of 1.44 times compared to the original best model’s performance alone.
In this study, we introduced an innovative bagging method based on an adaptive nonlinear neofuzzy metamodel. This system is designed to integrate the results from multiple computational intelligence systems working towards solving similar tasks, such as approximating values or making forecasts.
Our approach processes data in real-time and is particularly suited for environments where data conditions change rapidly over time (non-stationarity). This adaptability is crucial because many real-world applications, such as electric load forecasting, involve dynamic factors influencing demand. The speed of processing is also vital; ensuring timely decisions are made based on current data.
We validated our metamodel through a simulation involving short-term electric load forecasting. Utilizing an ensemble of six independent forecasting models to predict electricity consumption, we compared their results with those generated by our proposed metamodel. The outcomes demonstrated that our method outperformed the individual models, decreasing the MAPE error by a factor of 1.44 times relative to the best model in the ensemble.
Looking ahead, we aim to enhance this metamodel’s structure by increasing its flexibility. This will enable it to more effectively adapt to various types of relationships in data, particularly nonlinear ones, which are common in complex real-world scenarios like electricity demand forecasting. By doing so, we expect the metamodel to become even more robust and versatile in handling diverse and dynamic data environments.
This research was partially funded by the European Union (through the EURIZON H2020 project, grant agreement 871072).
The authors have not employed any Generative AI tools.