The generalization capabilities of deep neural networks are not well understood, and in particular, the influence of activation functions on generalization has received little theoretical attention. Phenomena such as vanishing gradients, node saturation and network sparsity have been identified as possible factors when comparing different activation functions [1]. We investigate these factors using fully connected feedforward networks on two standard benchmark problems, and find that the most salient differences between networks with sigmoidal and ReLU activations relate to the way that class-distinctive information is propagated through a network.
The simplest class of neural networks are the multilayer perceptrons (MLPs) [
Although several researchers have compared the performance of non-linear
activation functions in deep models [
Prior to the introduction of ReLUs, most DNNs used activation functions
called logistic sigmoid activations or hyperbolic tangent activations. Sigmoidal
units saturate across most of their domain: they saturate to a value of 1 when
the input is large and positive, and saturate to a value of 0 when the input is
large and negative [
Thus, each of the popular activation functions faces certain difficulties
during training, and remedies have been developed to cope with these challenges.
The general consensus is that ReLU activations are empirically preferable to
sigmoidal units [
We compare the performance of DNNs of varying width and varying depth on
several tasks. The two main activation functions of interest are the rectified linear
unit (ReLU) and the sigmoidal unit, as motivated in Section 1. We investigate
fully connected feed-forward neural networks on the MNIST [
We optimize the hyper-parameters that most strongly affect the model
convergence and generalization error, namely the learning rate and training seed.
Values for hyper-parameters such as the scale of initial random weights and
layer biases are selected after performing several initial experiments to
determine the combinations of hyper-parameters that give good convergence with
high validation accuracy. The optimizer used to train the neural networks is
Adam [
We let all models train for 300 epochs. To regularize the network training,
early stopping is used. No other explicit regularization is added to the networks,
such as L1 and L2 norm penalties. We do however acknowledge the regularizing
effect of batch normalization [
Figure 1 shows an example of the average learning and validation curves over three seeds with standard error; results shown here are for networks trained on the MNIST dataset. We observe that for different architecture configurations, the training curve of the ReLU and sigmoid networks are similar while the validation curves of the ReLU networks show better performance overall.
The learning curves from Figure 1 are obtained on the training and validation sets. The models performing best on the validation set are evaluated on the test set. Evaluation accuracy on the test set is reported over different experiments in Figure 2. From this figure we see that the ReLU networks generally perform better than the sigmoid networks on the evaluation set. However, when the network is wide enough, the performances of sigmoid and ReLU networks become more similar when not using batch normalization. An interesting observation is that when the sigmoid network has a constant width of 200 nodes, the application of batch normalization results in an increase in performance with an increase in depth. When not using batch normalization, the generalization performance of the sigmoid networks decreases with an increase in depth.
In the case of CIFAR10 from Figure 2(b), the ReLU networks only reliably generalize better than the sigmoid networks when trained with batch normalization. When not trained with batch normalization, the ReLU and sigmoid networks generalize less well with an increase in depth. The sigmoid networks perform similarly to ReLU networks in all configurations where batch normalization is not used, except for the 6- and 8-layer networks that are 200 nodes wide. The sigmoid networks generalize relatively poorly with these two configurations compared to the other configurations. This poor generalization could be attributed to the vanishing gradient problem since these two network architectures struggle to fit the training set. In contrast, we see a general increase in evaluation accuracy when increasing the number of hidden layers while training with batch normalization.
In summary, then, we observe that optimal generalization is obtained when
batch normalization is used to train wide ReLU networks; for CIFAR10, network
depth provides a small additional benefit. In contrast to [
To better understand the effect of the activation functions on training and generalization performance, we investigate the behavior of nodes after the activation function is applied. We specifically look at the activation values for each class at each node to investigate how class information is separated and propagated throughout the network. The term “activation distribution” is used to refer to the distribution of activation values for the samples of a class at a hidden node. We choose to show results only for the 4-layer networks as they have moderate depth and similar trends are observed in deeper and wider networks. At each hidden node hi,j we calculate the class activation distributions (ac,i,j )xxcc,,1M for each class c after applying the activation function T so that for a single sample-node pair we get: acm,i,j = T ( s(i−1) k=0 wi,j,khi−1,k) 1 ≤ i ≤ N with sample xc,m as input, and the distribution as: (ac,i,j )xxcc,,1M = {ac1,i,j , ac2,i,j , acm,i,j , ..., acM ,i,j } 1 ≤ i ≤ N 1 ≤ c ≤ C where c is the class index, i the layer index, j the node index and s the number of nodes in a layer. xc,1 is the first sample in a class c and xc,M the last sample (1) (2) with C the number of classes. For each class activation distribution (ac,i,j )xxcc,,1M at each node we calculate the median and standard deviation of the distribution.
When training a DNN with non-linear activation functions, there exists points where the learning process is saturated and gradients reach zero or nearzero values, depending on the activation function. For ReLU this saturation point is at 0.0, where values equal to or below this point have zero gradient and learning subsides for that specific sample at that node. The sigmoid function has two of these saturation points, at 0.0 and 1.0 respectively. For the sigmoid function, gradients never reach zero when nearing these saturation points, but they become very small and any weight update with this gradient has almost no effect. The position of the median with regard to the saturation points, and with regard to medians from activation distributions of other classes at a node provides an indication of how the class information is separated among different classes. The standard deviation gives an indication of the variability of the activation values and and the extent to which the majority of activation values approach the saturation points.
Figure 4 shows the typical output of specific hidden nodes in the trained network after the ReLU activation function has been applied at the node. It is clear that the activation distributions are cut off at 0.0 since activation values below 0.0 are rectified while the positive values retain their magnitude. The distributions in the positive domain retain some of the same uniform shape, depending on how many samples have negative activation values. In the positive domain, the activation distributions are not saturated and do not overlap as heavily for different classes such as the output of nodes in Figure 3. Not only do the nodes in deeper layers become more specialized towards one class, they activate for fewer samples than the nodes in earlier layers. This confirms the ability of ReLU trained networks to be sparse.
To understand how class information is separated and propagated, we plot the median of the activation distributions of each class at each node. Figures 5 through 7 show the median values for the distributions after the activation function has been applied. The x-axis indexes the nodes in the networks. For a 4x200 network, this means that nodes 1-200 make up layer one, nodes 201400 layer two, etc. Since the activation distributions of the sigmoid network are highly non-uniform, we track the position of activation distributions by plotting the median of the distributions. By using the median values, we regard the distribution as a whole and not only the location of the distribution mass.
For the untrained model, Figure 5 shows that the medians are mostly clustered around 0.5 for the first hidden layer. The medians in the second hidden layer (nodes 200-400) are slightly more saturated towards 1.0 and 0.0 and the median values are starting to overlap more for classes at the same node. From nodes 400 to 600 it is observed that the medians of the distributions overlap significantly, meaning that with the untrained weights the nodes are not able to distinguish activation values of one class from the others. In the last hidden layer (nodes 600-800) it is observed that the activation distributions lay almost directly on top of each other and class information cannot be separated. From Figure 5 we see that when the model is sufficiently trained, class information is separated by saturating medians towards 0.0 and 1.0. This effect is more clear at deeper hidden layers (nodes 400-599 and 600-800), while more nodes in earlier layers have activation distributions with median values slightly closer to 0.5.
For ReLU networks the mean and the median are almost identical due to the uniform shape of their activation distributions. Figure 6 shows the untrained model for the network trained with ReLU activation functions. The median values of node distributions are less saturated in earlier layers and saturate more towards 0.0 for deeper layers. It is important to note that the activation distributions in deeper layers do not overlap as strongly compared to the sigmoid network in Figure 5. The ability of the ReLU network to completely suppress class information and not activate for specific samples allows the network to still separate distributions of different classes in deeper layers. Even when distributions are not separated in a meaningful/learned way, the inherent “sparse” structure that the ReLU activations introduce suggests better separation of class information.
From Figure 6 it is observed that when a ReLU network is sufficiently well trained, the activation distributions in earlier layers have lower activation values and class information is suppressed with overlap of activation distributions of classes. This observation with the behavior seen in Figure 4 indicates that nodes in earlier layers remain relatively agnostic towards classes. The nodes in deeper layers have less overlap and nodes become more specialized towards specific classes.
Figure 7 shows the median values of activation distributions of networks trained on the CIFAR10 dataset. If the network cannot effectively fit the more complex data, class information cannot be effectively separated and the median values look more similar to the untrained models in Figure 5 and Figure 6.
Since the beneficial effects of sparsity in deep rectifier models have been
proposed as a mechanism for their performance [
In this study we compared the performance of different neural network architectures trained with ReLU and sigmoid activation functions. We observed that when networks are wide enough, sigmoid networks have comparable generalization performance to ReLU networks. The ReLU networks benefit more from the use of batch normalization than networks trained with sigmoid activations.
We investigated and compared the behavior of nodes trained with these two activation functions by looking at how class information is separated. It was observed that while ReLU and sigmoid networks both are able to effectively separate class information, they do this in fundamentally different ways. The sigmoid networks saturate class information towards 0.0 and 1.0 where there is plenty of overlap between activation distributions. The ReLU networks saturate class information towards 0.0 for earlier layers with moderate overlap of class distributions, making earlier layers more conservative towards class discrimination while nodes in later layers become more specialized towards single classes. We also show that when training a ReLU network with batch normalization, the hidden layers have lower average sparsity but superior generalization performance compared to ReLU networks trained without batch normalization.
Overall we find that sparsity and saturation seem less pertinent than the way in which class-distinctive information is propagated through the networks, and how node behavior differs. We plan to relate the node behavior of DNNs trained with different activation functions to their generalization performance in future studies.
This work was partially supported by the National Research Foundation (NRF, Grant Number 109243). Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and the NRF does not accept any liability in this regard.