Extreme learning machine (ELM) is a special single-hidden layer feed-forward neural network (SLFN), with only one hidden layer and randomly chosen weights between the input layer and the hidden layer. The advantage of ELM is that only the weights between hidden layer and output layer need to be trained, therefore, the computational costs are much lower, resulting in moderate training time. In this paper, we compare ELMs with different regularization strategies (no regularization, L1, L2) in context of a binary classification task related to gene expression data. As L1 regularization is known to lead to sparse structures (i.e., many of the learned weights are zero) in case of various models, we examine the distribution of the learned weights and the sparsity of the resulting structure in case of ELM.
Recent advances in neural networks lead to breakthroughs
in many applications in various domains, such as games,
finance, medicine and engineering, see e.g. [
For the above reason, other training approaches were
proposed, such as subset selection [
Extreme Learning Machines (ELM) were introduced
by Huang et al. [
While neural networks are powerful, due to their complexity, in the lack of appropriate regularization, they tend to overfit the data. In the era of deep learning, L1 regularization became popular due to various reasons: on one hand, sparse structures resemble the brain, on the other hand, they lead to computationally cheap models as the resulting zero-weights correspond to the lack of connections, thus they may be omitted.
Regularized ELMs have been shown to outperform
nonregularized ELMs [
In our study, we compare various regularization techniques – in particular: L1 and L2 regularization as well as the lack of regularization – in context of classification of gene expression data using ELM. 2
First, we define the classification problem and introduce the basic notation which is used in this paper. We are given a set X = fx(1); x(2); : : : ; x(m)g of training data containing instances x(i) = (x1(i); x2(i); :::; xn(i)) 2 IRn. For each instance x(i), its label y(i) is also given. The set of labels is denoted by Y = fy(1); y(2); : : : ; y(m)g. Each label y(i) 2 f0; 1g, 0 denotes a negative instance and 1 denotes a positive instance.
We use x1; x2; :::xn to denote the input nodes. H is the only hidden layer and the number of units in the hidden layer is denoted by L. We use hi to denote the ith hidden node. The activation value of ith hidden node for an instance x is hi[x] 2 IR, bi 2 IR is the bias of ith hidden node, ai; j 2 IR is the randomly initialized weight from xi to jth hidden node.
The output layer contains only one single unit, bi 2 IR is the weight from hi to the output unit and bo 2 IR is the bias of output node. ELM(x) 2 IR is the activation of the output unit for an input instance x. The structure of ELM is shown in Fig. 1. In this chapter we will introduce the fundamentals required to understand our work such as ELM, logistic regression and regularization of logistic regression. 3.1
ELM
Extreme Learning Machine is a special kind of single layer
feed forwarded network. The network has only one hidden
layer. The weights between the input layer and the hidden
layer (input weights, for short) are initialized once and not
trained, i.e., they remain unchanged. The output weights
between the hidden layer and the output layer are trained
iteratively. As the input weights remain in their initial state
and only the output weights are trained, the training time
of an ELM is much lower than that of a comparable single
layer feed-forward neural network (SLFN) [
The output of an ELM is the value of the activation function applied to the weighted sum of the activation values of hidden nodes:
L ELM(x) = g( å b jh j[x] + bo)
j=1 where g is the activation function. Since stucking into local minima needs to be avoided, a convex activation, in particular, the sigmoid function was chosen as activation function: The value of ith hidden node for x input is: g(z) =
1 1 + e z
: n hi[x] = g( å a j;ix j + b j): j=1 1 m
å Cost(ELM(x(i)); y(i)) m i=1 where:
J(b ) = + (1 1 m
å(y(i)log(ELM(x(i))) m i=1 y(i))log(1
ELM(x(i)))): The partial derivative of the cost function w.r.t. the kth parameter (bk) is: ¶ ¶ bk
J(b ) = 1 m
å(ELM(x(i)) y(i))hk[x(i)] m i=1 Logistic regression can be trained with gradient descent. That is: after initializing the parameters bk, in every iteration, all bk-s are updated simultaneously according to the following rule: bk = bk a ¶ ¶ bk
J(b ) where a is the learning rate. 3.3
In logistic regression and generally in all regression models, it is a common goal to keep the model as simple as possible. Regularization punishes a complex model, in particular, a penalty term is added to the cost function. Ridge regression (L2) adds squared magnitude of the coefficients as penalty term to the loss function. LASSO (Least Absolute Shrinkage and Selection Operator) regression adds absolute value of the coefficients as penalty term to the loss function. The key difference between these techniques is that LASSO shrinks the less important features’ coefficients to zero, thus, leads to a model with less complex structure.
In our case, the L1 regularized cost function is: its partial derivative w.r.t. the kth parameter (bk) is: ¶ bk 1 m
å Cost(ELM(x(i)); y(i)) + m i=1 l åL b j2; m j=1 and its partial derivative w.r.t. the kth parameter (bk) is: ¶ bk where l is the regularization coefficient which shows the weight of the penalty term in connection with the average cost.
Using the above partial derivatives, similarly to the case
of logistic regression, gradient descent can be used to train
L1 and L2-regularized ELMs.
Classification of gene expression data is a challenging
task with prominent applications in the medical domain,
such as the diagnosis of different subtypes of cancer, see
e.g. [
The Mice Protein Expression Dataset1 is available from the UCI repository The main properties of the dataset are summarized in Tab. 1.
Mice Protein Expression Dataset consists of
measurements of gene expression levels in mice. In total, the
expression levels of 77 genes were measured for 72 mice,
out of which 34 were trisomic (trisomy in mice may be
seen as a model of Down syndrome in human), while 38
belonged to the control group (i.e., mice that are not
affected by the disease). The expression levels of each gene
were measured 15-times for each mouse, resulting in a
total of 72 15 = 1080 instances, each of them
containing 77 gene expression features, see also [
The data contained high number of missing values for some of the gene expression features (in particular for 1https://archive.ics.uci.edu/ml/datasets/Mice+Protein+Expression BAD_N, BCL2_N, pCFOS_N, H3AcK18_N, EGR1_N, H3MeK4_N genes). We ignored these features.
Some of the instances of the remaining dataset contained missing values in other features, these instances were also ignored resulting in a dataset of 1047 instances and 71 gene expression features.
We split the data into train and test sets as follows: the test set contains 346 randomly selected instances, while the remaining 701 instances are assigned to the training set. 5
We compared three ELMs that differ in terms of the applied regularization technique: in the first model we did not use any regularization at all, in the second and third models we used L1 and L2 regularization, respectively. All three models were initialized with the same parameters.
For the Mice Dataset, the input weights, hidden biases and the output bias bo were randomly sampled from uniform distributions between 0:1 and 0:1, 0:75 and 0:75, as well as 1 and 1. The initial value of each output weight bk was set to zero.
Settings of the hyperparameters of ELMs, such as number of hidden nodes, learning rate, regularization coefficient and the number of training iterations are summarized in Tab. 2. 6
We use the area under receiver-operator characteristic
curve (AUC) [
L1 regularization is known to lead to sparse structures. Especially in case of highly-correlated features with similar predictive power, L1 regularization tends to prefer the best out of the slightly different features in the sense that a relatively high weight will be assigned to this ”best” feature, while zero weights will be assigned to the other ones. In contrast, L2 regularization distributes the weights more ”fairly” in the sense that highly correlated features will receive approximately the same weights.
For the above reasons, in case of L1 regularization, we expect many of the bk weights being approximately zero. In accordance with these expectations, we observed that more than two-third of all the bk weights were less than the learning rate after 125 million iterations. In this respect, the L1-regularized model is substantially different from the other two models as it can be seen in Fig. 3. The distribution of bk-s can be seen in Fig. 4.
The sparsity of the L1-regularized model, i.e., the high number of bk-s being (close to) zero, leads to a computationally simpler model: only the activation values of those hidden units need to be calculated for which the corresponding bk is different from zero. This makes L1regularized ELMs better suitable for scenarios in which the computational power is limited, such as embedded systems in case of wearable medical devices or self-driving cars. In this paper, we compared regularization approaches in context of classification of gene expression data with extreme learning machines. We observed that L1 regularization leads to sparse models that are computationally simpler than the comparable models without regularization or with L2 regularization. Therefore, L1-regularized models may be better suitable for embedded systems.
We plan to perform similar experiments on further gene expression datasets as part of our future work.
This work was supported by the project no. 204603/2018/FEKUTSTRAT within the Institutional Excellence Program in Higher Education of the Hungarian Ministry of Human Capacities. This work was also supported by Telekom Innovation Laboratories (T-Labs), the Research and Development unit of Deutsche Telekom.