The time series prediction and forecasting is an important area in the field of Machine Learning. Around ten years ago, a kind of Multilayer Neural Network was introduced under the name of Flexible Neural Tree (FNT). This model uses meta-heuristic techniques to determinate its topology and its embedded parameters. The FNT model has been successfully employed on time-series modeling and temporal learning tasks. The activation function used in the FNT belongs to the family of radial basis functions. It is a parametric function and the parameters are set employing an heuristic procedure. In this article, we analyze the impact on the performance of the FNT model when it used other family of neuron activation functions. For that, we use the hyperbolic tangent and Fermi activation functions on the tree nodes. Both functions have been extensively used in the field of Neural Networks. Moreover, we study the FNT technique with a linear variant of the Fermi function. We present an experimental comparison of our approaches on two widely used time-series benchmarks.
The first FNT model was developed with a Gaussian activation function in the functional nodes.
In this paper we analyze the FNT performance when another kind of activation functions is used in the nodes. We study the nodes with hyperbolic tangent and Fermi activation function, both family of functions were extensively studied in the field of Neural Networks. Additionally, we test the model with a linear variant of the Fermi activation neurons. We present an experimental comparison of the different activation functions on two widely used time-series benchmarks. The first one is a simulated benchmark commonly used in the forecasting literature called 20-order fixed NARMA data set. The another one is a real data set about the Internet traffic from an European Internet service provider.
The paper is organized as follows. In Section 2, we present a description of the Flexible Neural Tree model. Section 3 contains the description of other activation functions. Next, we present our experimental results. Finally, the last part presents the article conclusion. 2
About ten years ago a new kind of Neural Network (NN) was introduced under
the name of Flexible Neural Tree (FNT) [
The tree construction The topology of the tree is generated using a pre-defined instruction set. We consider the following function set F = {+2, +3, . . . , +Nf }. A node operator +i is an internal vertex instruction with i inputs. In the original FNT, the activation neuron function of any unit i is the following parametric function: f (ai, bi, x) = e−( x −biai )2 , (1) i neti = X wjxj,
j=1 outi = e−( netbi−iai )2 . where wj is the weight connection between xj and +i. The output of the node +i is obtained using the expression (1) where the parameters ai and bi are adjustable parameters. The terminal set of the tree consists of T = {x1, x2, . . . , xNt }. The instruction set S is defined as follows:
S = F ∪ T = {+2, +3, . . . , +Nf } ∪ {x1, x2, . . . , xNt }. (2) Each functional node +i has i random nodes as its inputs, which can be internal and terminal nodes. The model outputs can be computed using a depthfirst strategy for traversing the tree. Given a functional node +i with inputs {x1, . . . , xi}, the total input charge of +i is defined as: (3) (4) (5) Let i be a functional node if some of its inputs are other functional nodes, then the output of i is computed in a recursive way following the expressions (3) and (4). The algorithm complexity for traversing the tree is O(N ) algorithmic time where N is the number of unit in the tree. Figure 1 illustrates an example of an FNT. 2.2
The fitness function
In order to evaluate the accuracy of the FNT model in learning tasks an error
distance is considered. In this context, this performance measure is called fitness
function. In a supervised learning problem given a training set {(x(t), ytarget(t)), t =
1, . . . , T }, the goal is to infer a mapping ϕ(·) in order to predict ytarget, such
that the fitness function is minimized. Most often is used the Mean Square Error
(MSE) or Root Mean Square Error (RMSE) [
X(y(t) − ytarget(t))2.
T t=1 Another important parameter of the model performance is the number of nodes in the tree. Obviously, between two trees with equal accuracy the smaller one is preferred. 2.3
The parameter optimization using meta-heuristic techniques
Several strategies have been used to find “good” parameters and topology [
+7 x3 x4 +3 x1 x5 +5 x7 x3 x5 +3 x2
x1 x6 x2 x1 x2 +4 x4 x6 +2
x3 x6 x3 x4 neurons in the tree and we use the PSO method to find the embedded tree parameters.
The GP theory was intensely developed in the 90s [
There are two kinds of adjustable parameters on the tree: the activation
function parameters ai and bi for each functional node i presented in the
expression (1), another one refers to the weight connections between the nodes in the
tree. In this paper, we use Particle Swarm Optimization (PSO) [
In order to use PSO for estimating the embedded tree parameters, the position p(i) of the particle i is associated with the embedded parameters in one flexible tree (the weights and aj , bj , ∀j ∈ F). The PSO algorithm return the global optimal position according to the fitness function, presented in the expression (5). The relationship between the tree parameters and the particle position is given by: (p(1i), . . . , p(Ni)x ) = (a1, . . . , aNf , b1, . . . , bNf , w), (8) where w is a vector with the tree weights.
We present in the Algorithm 2 the PSO method used as meta-heuristic technique for finding the tree parameters.
Algorithm 2: Specification of the Particle Swarm Optimization used for finding the embedded tree parameters.
Inputs : Nx, number of particles, S, training data set, algorithmic parameters
Outputs: Tree parameters 1 t = 0; 2 Random initialization of p(i)(t) and v(i)(t) for all i; 3 Compute the fitness value associated with i using (8) and the fitness function; 4 Set p∗(t) and p(i)(t) for all i; 5 while (Termination criterion is not satisfied) do 6 for (Each particle i) do 7 Compute v(i)(t + 1) using the expression (6); 8 Compute x(i)(t + 1) using the expression (7); 9 Compute the fitness value associated with i using (8) and the fitness function;
Compute p(i)(t + 1); 10 13 Return the parameters using p∗(t) and the expression (8); 3
In this paper we analyze the impact of other neuron activation function in the
performance of the FNT model. The original model uses a Gaussian function
presented in the expression (4). This kind of function has been widely used in
Support Vector Machine (SVM) where the model uses the radial basis function
(RBF) kernel [
The Fermi activation function is 1 f (x) = 1 + e−x . g(x) = aif (x) + bi, (9) (10) (11) In this paper we test the FNT model using the tanh(·) and Fermi function. Moreover, we analyze a parametric variation of these functions, given by: where f (x) is the function of expression (9) and (10) and the parameters ai and bi are specifics for each tree node i. We use PSO in order to find the parameters ai and bi. 4 4.1
Description of the benchmarks
We use two benchmarks, a simulated data set which has been widely used in the
forecasting literature and a real data set about the Internet traffic data.
(1) Fixed kth order NARMA data set. This data set presents a high
nonlinearity, for this reason has been extensively analyzed in the literature [
Results We can see in the two graphics of Figure 2 the performance of the FNT prediction for the NARMA test data set using Gaussian activation function and Fermi activation function. Figure 3 shows the prediction of the model for the NARMA data set using the Gaussian activation function. In this last case we present the estimation on the last 200 samples. A example of estimation of the FNT using Gaussian is showed in the left figure of 4. The right figure of 4 shows the estimation of the model when the activation function is Fermi function. A example of prediction of the FNT model using the Fermi activation function for the Internet traffic data set is illustrated in Figure 5. We can see in Figure 4 the FNT estimation with the Fermi activation and the Gaussian activation function are very close. The Fermi function is not parametric, then the PSO algorithm is used only to estimate the weights in the tree. As a consequence the FNT model with Fermi function is faster in the training process than the FNT with exponential parametric function. The accuracy for the Internet Traffic data using the FNT technique and Fermi activation function was better than when we used the Gaussian activation function. Table 1 shows a comparison of the accuracy of the model using the different kinds of activation function in the nodes.
Function Narma Gaussian 2.25646 × 10−3 tanh(·) 3.47218 × 10−3 Fermi function 2.37082 × 10−3 Linear Fermi 3.23204 × 10−3
Ten years ago, the Flexible Neural Tree (FNT), a specific type of Neural Network was presented. The method has proved to be a very powerful tool for time series processing and forecasting problems. The model uses meta-heuristic techniques for defining the topology of the tree and for finding “good” parameters in learning tasks. The FNT uses activation function with exponential form in the nodes. In this paper we analyze the performance of the model with another family of function in its nodes, we studied the hyperbolic tangent and the Fermi functions. We tested the performance in two widely used benchmark data: a simulated and a real data set. The results are promising, specifically for the FNT model that uses Fermi function its the nodes. In future works, we will use statistical tests for comparing the different approaches presented here, as well as we will compare our results with other forecasting techniques.
This article has been elaborated in the framework of the project New creative teams in priorities of scientific research, reg. no. CZ.1.07/2.3.00/30.0055, supported by Operational Program Education for Competitiveness and co-financed by the European Social Fund and the state budget of the Czech Republic. Additionally, this work was partially supported by the Grant of SGS No. SP2014/110, VŠB - Technical University of Ostrava, Czech Republic, and was supported by the European Regional Development Fund in the IT4Innovations Centre of Excellence project (CZ.1.05/1.1.00/02.0070) and by the Bio-Inspired Methods: research, development and knowledge transfer project, reg. no. CZ.1.07/2.3.00/20.0073 funded by Operational Programme Education for Competitiveness, co-financed by ESF and state budget of the Czech Republic. 4 . 0 2 . 0 2 . 0