Artificial neural networks (ANNs) are becoming increasingly more relevant. Although the idea of ANNs spans multiple decades, various ANN architectures are still widely being developed to this day. One of the most important components of ANNs is activation functions. They are often used for introducing non-linearity, and in turn, allow ANNs to understand intricate features in the data. Although different activation functions have been developed and studied, there exists no body of work in which the choice of activation functions would be considered in the case of solar power generation forecasts. In this paper, we propose a new approach for improving the results of ANN predictions via changing the activation functions in the ANN. We have chosen to test our approach on a range of different machine learning tasks, with the goal of introducing a new, alternative hyper-parameter that would work for different ANN architectures.
Activation functions in an ANN are used to introduce non-linear relations to the data, so that the network would better fit the results and improve the accuracy of a given task. It is a very common part of ANNs and often omitted from neural network structure diagrams. Many mathematical functions have been introduced to achieve non-linearity, such as ReLU, Tanh, Sigmoid and others, each tailored to specific tasks. In this paper we entertain the idea of using no one activation function per layer or network, but multiple, assigning a different one for each neuron.
The importance of activation functions is discussed in many recent works. Their importance is based on their wide-spread usage in ANN architectures. Dubey has published a comprehensive overview of the most common activation functions, along with their characteristics and a performance comparison between them [1]. They have found that different activations functions are more suited for certain machine learning tasks, and that in certain cases, alternative choices must be considered. Although there are some common choices, new activation functions are constantly being developed [2,3,4,5,6]. Yu has created a modified activation function based on ReLU, with the goal of increasing the accuracy of classification tasks [2]. Wang developed a activation function as a better alternative to other commonly used activation functions [3]. The developed activation function, Smish, performed better than other common activation functions in classification tasks on open datasets. Wuraola has developed a family of activation functions that are to be used in embedded systems [4]. The proposed activation functions were shown to be computationally faster, and their use resulted in higher accuracy results than other common activation functions in recurrent neural networks and logistic regression models. Kaytan has introduced a new non-monotonic activation function capable of achieving higher results than other activation functions like Swish, Mish and others for image classification tasks [5]. Chai developed a new model based on LSTM capable of achieving higher accuracy for short-term PV generation forecasts [6]. The model uses a newly proposed activation function that helps solve the gradient disappearance problem and ensures a high accuracy of the prediction results for the task of short-term PV generation. There are also works in which the activation functions of the default implementation of model architectures are switched with other, alternative activation functions. Anami had performed experiments in which they had tried to compare prediction results by switching the default activation function with other different, common activation functions [7]. Wang has performed experiments in which they tried to use alternative activation functions in VGG16, ResNet50 and LeNet architectures, achieving superior results [8]. Essai Ali has tried to modify a LSTM by changing its' Tanh functions to different activation functions [9]. The author has achieved his aim of increasing the classification accuracy from 86% to 88% using the Weather Reports dataset, and from 93% to 97 % using the Japanese Vowels dataset. Let's review the concept displayed in Figure 1. In this example we have an input layer, hidden layer of 2 neurons and one output layer. Each neuron has a different function applied to it. Calculations for such a network is as follows:
where β -hidden layers, π€ -weights, π₯ -inputs, π -bias, π§ -activation function results and πoutputs. In an artificial convolutional neural network activations play a similar role, but because there are no actual neurons in a convolutional layer, different application is required. For the convolution layer 2 approaches were introduced. For linear layers it is also possible to have a complete list of activation functions assigned. This idea is later experimented in this paper. Combinations of this list can be calculated as such. In this case 2 activation functions (ReLU, Tanh) power by 4 neurons equal to 16 variations:
where π£ -variations, π -elected activations and π -number of neurons. It must also be noted that various activation functions can be used, and it is not limited to the most used activation functions such as ReLU, Tanh, Sigmoid, etc. The range of activation functions that were tested in this work are detailed in the experiments section.
There has been a vast selection of CNN models proposed for image classification, a lot of those have complex implementations and long training hours. The models chosen for this paper are a low to mid-range complexity to test out the theory. Starting with SimpleNN, a simple neural network with one hidden layer of N neurons. TuNet -a CNN with 2 convolutions, 2 polling layers and 3 linear layers [10]. AlexNet is a convolutional neural network (CNN) architecture that consists of five convolutional layers, three fully connected layers, and two pooling layers [10]. The convolutional layers extract features from the input images, while the pooling layers reduce the dimensionality of the feature maps. The fully connected layers learn a mapping from the extracted features to the output classes. Some of the key innovations introduced by AlexNet include the use of rectified linear unit (ReLU) activation functions, dropout regularization, and data augmentation techniques.
ResNet50 derives its name from its depth, incorporating 50 layers [11]. Notably, ResNet50 addresses the challenge of training deep networks by introducing residual connections that enable the direct flow of information across layers. This innovation mitigates the vanishing gradient problem, allowing for the successful training of extremely deep networks.
The architecture comprises building blocks known as residual blocks, each containing skip connections that bypass one or more layers. These skip connections facilitate the smooth propagation of gradients during backpropagation, enhancing the model's ability to capture intricate features. Additionally, ResNet50 employs batch normalization to accelerate training convergence and improve generalization performance.
UNet was used for image segmentation tasks [12]. It is a popular model with several modifications over the years [13,14,15]. The model has improved on the results of previous image segmentation models by its' architecture consisting of a contracting path used for capturing context and a symmetric expanding path used that enables precise localization [12]. The resulting architecture consists of 23 convolutional layers and the architecture utilizes the ReLU activation function. The model also heavily utilizes image augmentation, which enables it to achieve high accuracy without relying on many training images.
Several image datasets are popular for testing performance of CNN models. The CIFAR-100 is a dataset containing 60 000 32x32 color images with 100 classes (600 images per class). It is a subset of the Tiny Images dataset and is commonly used for fine-grained image classification [16]. The dataset contains a wide variety of images of objects, animals, and textures. The images are labeled with both fine-grained and coarse labels. The fine-grained labels correspond to the specific object or scene in the image, while the coarse labels correspond to the superclass of the object or scene.
The German Traffic Sign Benchmark is a multi-class, single-image classification challenge held at the International Joint Conference on Neural Networks (IJCNN) 2011 [17]. The following dataset includes 43 classes of traffic signs and more than 50,000 images.
Cityscapes dataset is a popular image segmentation dataset that consists of 25 000 such images captured from a moving vehicle [13,14,15]. The images were taken in different cities in Germany during different weather conditions. The dataset consists of 50 different classes. Each dataset item consists of a horizontally joined image, in which the left image is the original photograph, meanwhile the right image is the semantically segmented version of the image.
Two tabular datasets were incorporated in this paper: breast cancer and iris flower classification. Breast cancer dataset features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass [18]. They describe characteristics of the cell nuclei present in the image. A few of the images can be found at http://www.cs.wisc.edu/~street/images/.
Iris flowers dataset is one of the earliest datasets used in literature on classification methods and widely used in statistics and machine learning [19]. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable from the other 2; the latter are not linearly separable from each other. When performing experiments, Obaid's work was used as a benchmark for the comparison of results [20].
Timeseries data for amazon stocks with stock price, closing price and other attributes was used [21]. Additionally, a custom photovoltaic (PV) panel generation dataset was used. The data consists of about a year of meteorological and PV generation data. The PV generation data was retrieved from a PV station in Kaunas, Lithuania, meanwhile the publicly available meteorological data was retrieved from Oikolab and from the Lithuanian Hydrometeorological Service. It was also attempted to include METAR data on cloud conditions at different altitudes, but utilizing this data did not provide any improvement to the results, so it was left out from the dataset. Based on the observed linear relationships between different meteorological features and PV generation, certain meteorological features were chosen to be used in the experiments (see Figure 4). As can be seen from the relationships between different features, a strong linear relationship between PV generation and air temperature, surface solar radiation has been observed. It was noted that using other meteorological data improved the results, although these features did not seem to have a linear relationship with the PV generation data. In total, the dataset consists of the following 11 features (see Table 1). As it can be seen from the table, a wide range of different meteorological variables were used.
Google Collab environment with a single NVIDIA Tesla T4 GPU was used for experimentations of AlexNet and ResNet50 on CIFAR100. For GTSRB, UNet and LSTM experiments, the models were trained on two Tesla T4 GPU setup. Amazon stock close predictions were performed on a Kaggle provided CPU.
Inspired by Sharma's work [22], we choose AlexNet as the primary target. Main reasons for choosing this architecture were that it had linear layers aside convolution blocks. We began experiments with the OriginalAlexNet implementation as a baseline with Tanh. Next, we experimented with changing only linear layers -changing one layer then changing both. The change was that instead of applying a single activation function, we applied 2 or 3 in cyclic order. The best results were with Tanh and Softmax combination of functions -1.14% improvement in testing accuracy compared to the ReLU baseline, however, Tanh baseline was still more superior.
Later, we expanded experimentation with modifying Convolution Neural Network layers (CNN). Here implementation consisted of changing activation functions per channel. This showed marginally better results than the OriginalAlexNet with ReLU -0.36% improvement.
For experimentation, hyper parameters were the following: learning rate -0.0001, batch size -256 and number of epochs -40.
We have also investigated Residual networks block, using ResNet50 architecture (see Table 3). Hyperparameters used for the experiment: learning rate -0.0001, batch size -256 and number of epochs -12. As can be seen from results, only a combination of three functions -Tanh, Softmax and ReLUmanaged to outperform baseline model with ReLU by 0.66% margin. Other combinations were below.
Classifying images are pre-processed in the same manner and on the same training parameters as in the previous experiments, meanwhile the fixed size image is 32 by 32 pixels. The training parameters for TuNet are as follows: optimizer -Adam, learning rate -0.001, loss function -cross entropy and batch size -32. As can be seen in Table 4, the results of the TuNet baseline are generally worse than of the modified architecture: In the table, several different models can be seen: ο· TuNet -baseline model.
ο· TuNetOnlyNN -a model, where convolution has one activation function and neuron linear layers have specific activation function for each neuron.
ο· TuNetPerNeuronAndChannel -a model, where convolution layers have a specific activation function for each channel and a specific activation for each neuron in linear layer.
We can see a very slight improvement when different activations are applied to only the linear layer.
For the image segmentation task, the popular Cityscapes dataset was chosen alongside the UNet model. The following parameters were the same for all the experiments using UNet: Adam optimizer with a learning rate of 0.001, the mean-squared error as the loss function, a batch size of 4 and 20 as the number of epochs for training.
As it can be seen from the results of the experiments, a significant Dice metric increase of about 10% was achieved by various activation function combinations (see Table 5). As can be seen from the table, using almost any combinations of activation functions can result in better prediction results in the case of UNet. It is also observed that even changing the activation in the baseline model from ReLU to Tanh has improved the results by a significant amount as well.
Experiments were performed on Amazon stock timeseries data predict the closing price for the next day. An architecture named SimpleNN was used. It is a neural network with 1 input cells, 14 hidden layer cells and 1 output. The following parameters were used in the experiment: optimizer -Adam, learning rate -0.001, loss function -mean-squared error, batch size -16, lag values -7 and number of training epochs -5.
The experiment compares the same model and its architecture, the only difference is activations per neuron and one activation for the whole network (see Table 6). As can be seen from the results, there is an increase in accuracy in certain cases, and it can also be observed that finding the best possible set of activation functions yielded the best results out of the experiments.
Experiments were performed using a time-series dataset for forecasting PV generation. An LSTM model was used, as it is often utilized for solving PV generation forecast tasks [23,24,25,26,27]. For performing the forecasts, the output of the previous step is used as the input of the following training step. The following parameters were used for the experiments: Adam optimizer with a learning rate of 0.001, mean-squared error for the error metric, a batch size of 8, 12 lag values for the PV data, and 20 training epochs.
The parameters for the experiments were chosen based on experiments performed using different sets of parameters. The batch size refers to the number of predictions retrieved from the model output and the lag values refers to the number of previous predictions to use as input of the next prediction. Based on tests using different lag values, a value of 12 was noticed to be one of the best values for this parameter, although this parameter did not seem to have much impact on the accuracy of predictions. Regarding transformations of data, the training data has been standardized so that the ranges of values would be the same for all features. As can be seen from Table 7, there is no significant improvement based on testing RMSLE. Although many experiments yielded similar results to the baseline, there was not a single experiment which yielded better results than the baseline. It can also be observed that an increase in the number of different activation functions used does not improve the forecast results either.
Tabular data is still widely used in machine learning tasks. In this paper we choose two datasets to experiment with the changes on Iris flowers and Breast cancer classifications. Both experiments have the following training parameters: optimizer -SGD, learning rate -0.01, loss function -cross entropy loss and number of training epochs -200.
From results displayed in Table 8 comparing one activation versus multiple for this Iris flowers classification task, there is no improvement compared to best suited activation function. Additionally, a activation function set from a large number of combinations was selected and the accuracy using it is better compared to one activation function (see Table 10). It should also be noted that better results were achieved than from the SVM described in Obaid's work. As can be seen from the results, there is a significant accuracy increase for the PerNeuron models, whilst the most significant increase can be seen when finding the best activation function list from all possible combinations.
The research paper explores the concept of using multiple activation functions in artificial neural networks. It discusses the role of activation functions in introducing non-linear relations to improve the accuracy of tasks. The paper investigates different approaches to incorporating multiple activation functions, including assigning a different function to each neuron or channel.
The experiments included using models such as AlexNet, ResNet50, TuNet, and SimpleNN. In the AlexNet experiment, different activation function combinations were tested in both linear layers and convolutional neural network (CNN) layers. The results showed that using OriginalAlexNet with Tanh activation function yielded the best overall performance. The ResNet50 experiments resulted in one combination performing marginally better than any of single function baselines. The TuNet and SimpleNN experiments aimed to evaluate the performance of these specific architectures on their respective datasets. Overall, the experiments provided insights into the impact of activation function combinations on model performance, with modest improvements observed compared to using a single activation function. The datasets used in the experiments included CIFAR-100, GTSRB, Breast Cancer Wisconsin (Diagnostic), Iris flowers, and Amazon stocks. In image segmentation tasks, modifying the UNet architecture with different activation function combinations leads to significant improvements in the Dice metric. Even changing the activation function in the baseline model from ReLU to Tanh shows improved results. For time series regression/forecasting tasks, the experiments show that using multiple activation functions does not significantly improve the accuracy of predictions. This paper also hints into an idea of full list of activation functions, which would learn relation with the specific data neuron is receiving. An idea which requires further analysis.
Overall, the paper concludes that while using multiple activation functions can have some benefits in certain scenarios, the improvements are not substantial compared to using a single activation function. The choice of activation function should be based on the specific task, dataset and its features.