Nowadays, the Multiple Classification Systems (MCS) (also called as ensemble of classifiers, committee of learners and mixture of experts) constitutes a well-established research field in Pattern Recognition and Machine Learning. The MCS consists in dividing the whole problem with resampling methods, or using different models for constructing the system over a single data set. A similar approach is studied in the Neural Network context, with the Modular Neural Network. The main difference between these approaches is the processing cost associate to the training step of the Modular Neural Network (in its classical form), due to each module requires to be learned with the whole data set. In this paper, we analyze the performance of a Modular Neural Network and a Multiple Classifier System integrated by small Modular Neural Networks as individual member, in order to identity the convenience of each one. The experiments here were carried out on datasets from real problems showing the effectiveness of the Multiple Classifier System in terms of overall accuracy and processing time respect to uses a single Modular Neural Network.
The Modular Neural Networks (MNN) presents a new trend in Neural Network (NN)
architectural designs. It has been motivated by the highly-modular nature in biological
networks and based on the “divide and conquer” approach [
There exist several implementations of the MNN, although the most important
difference among them refers to the nature of the gating network. In some cases, this
corresponds to a single neuron evaluating the performance of the other expert
modules [
On the other hand, currently, the multiple classifier system (MCS) (also known as ensemble of classifiers, committee of learners, etc.) is a set of individual classifiers whose decisions are combined when classifying new patterns. Some reasons for combining multiple classifiers to solve a given learning problem are: First, MCS tries to exploit the local different behavior of the individual classifiers to improve the accuracy of the overall system. Second, in some cases MCS might not be better than the single best classifier but can diminish or eliminate the risk of picking an inadequate single classifier. Finally, the limited representational capability of learning algorithms, it is possible that the classifier space considered for the problem does not contain the optimal classifier
To ensure a high performance of the MCS it is necessary to have enough diversity in the individual decisions, and consider an acceptable individual accuracy of each membership, which constitutes the MCS.
Some aspects of the MCS aim to overcome in comparison when a single classifier
is used, are [
In this paper, a comparative study that aims to display the advantages of both methods, the MNN and a MCS are used for classification task. In the first method (MNN) each member corresponds to a linear perceptron, and in the MCS each individual classifier corresponds to a single MNN, that is to say, the MCS is a neural network made with MNN. 2.
MNN called as systems committee, Hierarchical Mixture of Experts or Hybrid
Systems [
In its most basic implementation, all modules are of the same type [
In the learning process, the network uses the stochastic gradient function: − ln⎜⎜⎛⎝ ∑=r1 i gi ∗ exp⎜⎛⎝ − 12 s − Zi 2 ⎟⎞⎠ ⎟⎟⎞⎠ (eq.1) where s is the desired output for the input x, and Zi is the output vector of the j’th expert network, gj is the output of the gating network, ui is the total weighted input received by output neuron j of the gating network.
Given a pattern x n-dimensional as input, the overall learning process of the MNN considers the following steps: 1. Random initialization of the synaptic weights for the different networks with small values uniformly distributed. Henceforth, we will consider wji as weights of the expert network and wti as the integrating network. 2. The pattern x is presented to each and every one of the networks (experts and integrating network) so, the output of the knowledge network is given by:
Z m = x ∗ w mji
i where x is the input vector, and the superscript m is indicative of module. Similarly, the output of the gating network is obtained by, where ui = x * wti: exp(ui ) gi = r ∑ exp(u j ) j=1 Adjusting the weights of the expert networks and the gating network: To adjust weights, two criteria are taken into account. (eq.3) (eq.2)
From expert networks: wmji (I +1) = wim (I ) +η ∗ hi (s − Zim )x For the gating network: wti (I +1) = wti (I ) +η (hi (I ) − gi (I )) x (eq.4) (eq.5) where: hi = r ∑ g ∗ exp⎜⎛ − 1 s − Zim 2 ⎟⎞ j=1 ⎝ 2 ⎠ 4. Finally, the network decides how the modules outputs will be combines to obtain the final output of the MNN by: = !!!! g ∗ (eq.7)
Let D = {D1,..., Dh} be a set of h classifiers. Each classifier Di (i = 1,..., h) gets as
input a feature vector x ∈ Rn, and assigns it to one of the c problem classes. The
output of the MCS is an h-dimensional vector [D1(x), . . . , Dh(x)]T containing the
decisions of each h individual classifiers. After that the individual decisions are combined
by some strategy [
For constructing a MCS it is based on two aspects: the diversity in the individual
decisions and the accuracy of the single classifiers. The methods used to achieve
diversity can be described in five groups [
To integrate the MCS, in this study we use subsamples which consider patterns
manipulation, such that the resulting subsets have a proportional size to the number of
classifiers that integrate the MCS. Thus, in the experiments here reported the MCS
was integrated with 7 and 9 classifier each one, according to [
To obtain the subsamples, we use the random selection without replacement of
patterns [
When the subsamples are integrated using some resampling method, each one is presented to the MCS (Fig. 2). After that, for combining the individual classifier decisions in the literature two strategies are proposed: Fusion and selection. In classifier selection, each individual classifier is supposed as an expert in a part of the feature space and correspondingly, only one classifier is selected to label the input vector. In classifier fusion, each component is supposed to have knowledge of the whole feature space and thus, all individual classifiers are taken into account to decide the label for the input vector.
The results correspond to the experiments carried out over 12 real data sets taken from the UCI Machine Learning Database Repository (http://archive.ics.uci.edu/ml/).
For each database, we estimate the average predictive accuracy and processing time by 5-fold cross-validation, considering the 80% as the training set and the remaining as the test set (20%). According to the scheme of MNN and the MCS, some specifications are as follows: 1. Topology. Each expert in the MNN corresponds to a linear perceptron, in which the number of nodes in the input layer corresponds to the number of attributes in the input pattern. For the experts network, the number of neurons in the output layer is equal to the number of categories in the problem, while for the integrating network is equal to the number of experts used. 2. Connection weights. The connection weights where initialized to random values in the range between -0.5 and 0.5. 3. Each MNN consists of 5 modules and an gating network. 4. For the final decision on the MCS, simple majority voting was used.
Only the result of the best technique on each database has been presented. Analogously, for each database, related to the number of subsamples to induce the individual classifiers, that is, the number of classifiers in the MCS, we have experimented with 7 and 9 elements, and the best results have been finally included in Table 2. Besides, single MNN classification accuracy for each original training set is also reported as the baseline classifier.
Since the accuracies are very different for the distinct data sets, using these results across the data sets will be inadequate. Instead we calculate ranks for the methods. For each data set, the method with the best accuracy receives rank 1, and the worst receives rank 5. If there is a tie, the ranks are shared. Thus the overall rank of a method is the averaged rank of this method across the 12 data sets. The smaller rank indicates the better method.
In Table 2 are two sections; the first one includes the MNN results. The second section shows the results when the MCS is used with 7 and 9 classifiers. In this case, the corresponding capital letter identifies the resampling method used for obtaining the subsamples: randomly without replacement (A) and Bagging (B). The results correspond to the overall accuracy and the standard deviation included in parentheses and values in bold type indicate the highest accuracy for each database.
88.4 (4.6) 73.7 (8.6) 63.5 (5.4) 66.5 (1.6) 65.9 (6.2) 80.7 (11.4) 36.4 (7.1) 61.8 (18.0) 67.9 (4.5) 77.2 (2.7) 78.2 (5.6) 46.5
From results shown in Table 2, some comments may be drawn. First, except with Cancer and Segment data set, it is clear that some MCS schemes leads to better performance than the MNN. This is confirmed by the general basis of the MNN, which clearly corresponds to the poorer. Second, comparing the MCS using 7 or 9 classifiers, it is possible to observe that when we use a MCS with 7 classifiers we can find some results with a precision greater than (or equivalent) rating when nine classifiers are used. Finally, to compare different resampling methods, the A method (random selection without replacemnt) behave generally better performance than the B method (Bagging), using MCS with 7 classifiers on 5 datasets. In fact, for best results, the details are still very close to the winner.
The Vehicle data set is a special case due to the poor performance, regardless of the scheme used. In this case, a thorough analysis of the data distribution is necessary in order to identify the reason why the MNN and the MCS are not able to recognize the kinds of problem which is required.
Another aspect to be analyzed is the computational cost associated with each model. To this end, Table 3 shows the time required in minutes during the training and the classification process by each classifier model.
Results in the Table 3 clearly show large differences between the processing times
obtained by the three models used. It is interesting to note that in the majority of
cases, the time required by the MNN is almost two times more than the required by any
MCS. For example, with Sonar dataset the MNN requires nine times more than the
MCS. These differences could be because the MCS uses small subsamples in the
training process m/L, where m, is the number of patterns of training and L the number
of subsamples [
Finally, regarding the performance of the schemes used, we can note that the best classification results was obtained with an MCS with 7 classifiers requiring less time processing respect to the single MNN and short differences respect to an MCS with 9 members. 5
Designing a MCS with MNN as individual classifiers has been here analyzed. Two MCS were used, with 7 and whit 9 classifiers. For the single MNN architecture, we have employed five network experts and one gating network. The experimental results allow comparing these models, in terms of processing time and predictive accuracy. From this, it has been possible to corroborate that in general, the MCS clearly outperforms the classifier obtained with the MNN.
In addition, when comparing the behavior of the resampling methods, it has been empirically demonstrated, that to use the random selection without replacement offers the best performance: with greater precision and lower computational cost.
Finally, by comparing the results of the classification and the processing time required for each model, the use of the MCS provides the best performance, being the best option to improve the binomial time-accuracy.
As a future work to expand this research, aimed mainly at the improvement the single MNN performance. In this context, other architectures with different parameters and possible mechanisms such as regularization/Cross-validation must be analyzed. Also, it should be further investigated the relationship between the individual classifiers and the resampling methods in order to determine the “optimal” scenario. Acknowledgements. This work has partially 3834/2014/CIA project, from the Mexican UAEM. been supported grants