-the report discusses the structures of modules composition and the problems associated with their learning. The optimal algorithm for modules learning for the classification problem is considered. Examples of specific structures are given. The structural-parametric synthesis of an ensemble of modules of neural networks are described. The results of the training of the modules and ensembles are presented, as well as a comparison with the results of the training of individual neural networks.
INTRODUCTION
The classification problem is one of the most frequent
tasks that arises in the field of machine learning. It is common
in many areas of life. Researchers from around the world are
developing tools and algorithms to solve this problem
[
For more difficult tasks, the new development branch of NN has become to combine them into one module: one large network, which consists of several base networks. In the future, to increase accuracy, it is possible to combine such modules into ensembles. This will compensate for the shortcomings of each module by others, which will certainly have a positive impact on the final result of learning.
II.
The goal of this report is to research the possible structures of the module, the topologies of the networks included in the module and the machine learning of the module for solving the classification problem. The further task of using the NN modules is to combine them into an ensemble.
III.
The module topology involves the sequential combination of several different neural network architectures. In general, the module operates in the same way as an individual NN. Its advantage is the combination of various data transformations, which allows to obtain more accurate results. The module topology is presented in Fig. 1.
Fig. 1. Module topology
Modern technologies allow to operate with huge networks as with simple elements for building something larger, like LEGO bricks. The abstraction level of modern software is very high, everything is at an intuitive and understandable level, so the practical implementation of NN modules is a fairly easy task.
The main problem of modules composition is their learning algorithm. Inside the module there are several networks, therefore, we can train all these networks in different ways: • all networks are trained together; Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) • some networks are trained together, some are trained
separately; • each network is trained separately on the training sets;
then they are combined into a module.
The simplest way to train networks in a module is to use a
genetic algorithm [
As you can see, this training algorithm belongs to the first type: all networks are trained together. It has several disadvantages. Consider, for example, a module of three networks, each of which has 100 parameters. For each parameter, we will allocate 4 bits. The chromosome will contain 3 * 100 * 4 = 1200 genes. The convergence of this algorithm will require a tremendous amount of time and resources, so, in this case it is inefficient.
To maintain a balance of learning speed and accuracy, this report proposes using the network that is based on unsupervised learning as the first network. Its output goes to the input of the base network, which is trained separately under supervised learning. After the base network another network can be placed to refine the result.
The advantage of the structure presented above is that the first network is trained without a teacher very quickly, compared to the large base networks. It performs preprocessing (clustering, dimensionality reduction) of the input data, which ultimately has a positive effect on the following base networks. This preprocessing reduces the number of layers and neurons in the layers of the base network, so that it will be trained much faster and more accurately.
In some simple cases, a network that is based on unsupervised learning will already produce fairly accurate results, so the subsequent small base network will only refine them. In sum, the training of such two small networks will be much less than the training of one large base network.
In this report, the use of the Kohonen network [
As a base network, various networks can be used (e.g. perceptron, radial-basic function network, NEFClassM, etc.).
The full topology of the proposed module is presented in Fig. 2.
To refine the classification result, bidirectional associative
memory [
In simple cases, the construction of a single module may be sufficient to achieve the required accuracy, but to solve complex problems it is necessary to use several modules combined into one ensemble.
The construction of the ensemble allows you to look at the problem from points of view of different modules. The usage of elements of the ensemble as modules presented in Fig. 2 instead of simple neural networks has a set of advantages. It requires less memory, takes less time to train and, as will be shown in the next part, has greater accuracy than an ensemble consisting of individual neural networks.
IV.
For the experiment, modules were used, which consist of
two networks. The first is the Kohonen network and the
second is the base network. As the base networks were used:
perceptron, radial-basis function network, counter
propagation network, probabilistic neural network,
NEFClassM, Naïve Bayes Classifier. In order to show a clear
advantage of modular topology over individual NN, a
comparison was made with the results obtained in [
1. Kohonen network. At the beginning of the study, the Kohonen network was trained. The number of output neurons in the network is 3. In this way, after preprocessing by this network, the data dimension decreased by more than 4 times. Network training time is 3.6 ms. On large datasets, obviously, time will be longer, but it is not comparable with dozens of minutes, hours, days of training large networks with complex architectures.
2. Perceptron. Reducing the number of input neurons
to 3 reduced the number of neurons in the hidden layer from
48, as in [
3. Radial-basis function network (RBFN). The number
of neurons in the hidden layer is 3, decreased by half,
compared with [
4. Counter propagation network (CPN). The number of neurons in the input layer is 3, in the hidden layer is 3. Before the start of training, input vectors were normalized. The weights of the Kohonen layer were initialized with random values from the interval (0, 1) and normalized.
5. Probabilistic neural network (PNN). The number of neurons in the input layer is 3, in the first hidden layer is 142, in the second hidden layer is 3.
6. NEFClassM. Number of input neurons is 3. For
each feature three initial fuzzy sets were defined with the
names “small”, “medium”, “large”. In the rule layer there are
3 neurons. From the trained rule base, one best rule was
obtained for each class. Number of output neurons is 3. The
maximum number of generated rules is 50 instead of 40 as in
[
7. Naïve Bayes Classifier. Distribution functions are normal distributions. Priors is a ratio of the number of samples in the class to the total number of samples.
The learning results of the modules and all the networks described above are presented in Table 1.
As you can see, the simplification of the architecture and
the lack of preliminary standardization significantly affected
the accuracy of the underlying networks. At the same time,
the preprocessing by the Kohonen network gave very good
results that exceed the results presented in [
Subsequently, individual contributions of each network
[
NUMBER OF MISSCLASSIFIED SAMPLES WITH PRUNED
ENSEMBLE OF INDIVIDUAL NN AND MODULES
Pruned NN ensemble Naïve Bayes, Train NEFClassM, 7
PNN
As the results showed, the accuracy of the pruned
ensemble of modules is higher than the accuracy of an
ensemble of individual networks without preprocessing. It
also exceeded the accuracy values of the ensemble from [
The results showed that the use of the neural networks module, the first element of which is the Kohonen network, and the second is the basic network, allows to obtain accuracy indicators that exceed the corresponding indicators of individual networks. At the same time, to achieve a given accuracy, the total training time of the module is much lower than the training of a separate network with a complex architecture. Simplification of the topology of the basic networks in the module allowed to reduce the memory occupied by them.
Due to these advantages, the construction of an ensemble
of modules of neural networks is a more efficient and fast
solution. As the results of the study showed, the pruned
ensemble of the modules presented in this report has
accuracy indicators that exceed those of the individual
networks from [