Issues related to the application of multithreshold neural units in machine learning are considered in the paper. A labeled modification of the model of binary-valued multithreshold neuron is proposed. The 2-layer feedforward network architecture with the hidden layer consisting of multithreshold neurons is treated. The multicategory classifier is designed on the base of such architecture. The supervised algorithm is proposed for the synthesis of multithreshold neural networks. Simulation results are presented related to the performance of multithreshold classifier on real-world dataset compared to some popular classifiers. The impact of algorithm hyperparameters on the classifier performance, as well as on its representational capacity is discussed.
Neural networks play extremely important role in machine learning [
During few past decades we were witnesses of numerous rises and falls of the interest in neural
computations. Early neural-like models (e.g., a model of McCulloch and Pitts artificial neuron)
used linear threshold units in which binary-valued Heaviside step function [
In order to overcome these drawbacks of early models of neurons, many improvements were
proposed [
The models with multithreshold activation function were one of first in the second category [17]. A multithreshold neuron has weight vector w w1, , wn Rn associated with inputs x1, , xn and threshold vector t t1, ,tk Rk instead of a single threshold used in McCulloch and Pitts neuron. The use of multiple thresholds allows to operate with neuron in two modes: binary-valued and multi-valued, respectively [18]. The first mode uses the following activation: (1) (2) 1, if t2 j1 x t2 j , j 1, k / 2, ft2 x
0, if t2 j x t2 j1, j 0,1, k / 2.
Note that bipolar output values are also acceptable for such function, if 0 is replaced by –1 in the last row in braces in equation (1).
The second mode can be preferable for application in multiclass classification [19, 20]. In this mode the following (k 1) -valued counterpart of (1) is used 0, if x t1, 1, if t1 x t2 , ftk 1 x ........................
k 1, if tk 1 x tk , k, if tk x.
Notice that both activations (1) and (2) can be used in multiclass classification [18]. But in the case of the binary-valued activation (1) the output layer of special shape is required as well as the employing of the “one versus all” approach for the learning of classifier [21].
The main goal of the paper is the study of the model of 2-layer neural network whose hidden layer consists of binary-valued multithreshold neurons and examination whether this model is suitable for the multiclass classification, as well as the development of the synthesis algorithm for such networks.
As it was mentioned in introduction, multithreshold neural units were proposed in the early
studies in artificial intelligence [17, 21, 22]. The first models employed the binary-valued
activation in the form (1). The reason for the use of additional thresholds was the assumption that it
can improve the capability of basic (single-threshold) linear threshold units [23, 24]. The
theoretical reasons were studied in [17, 21, 23, 25] confirming this assumption and giving quantitative
expressions for it. These works dealt with the estimation of capability of multithreshold elements
to produce dichotomies of finite sets in n-dimensional real space [
Papers [
The above observation was confirmed in [
Notice that the mentioned advances in the use of multithreshold activation are actually related
to only networks that use bithreshold nodes in their hidden layers [
Let us consider again a model of k-threshold binary-valued neuron with the weight vector w and (ordered) threshold vector t, which output is given by (1). It is possible to enhance this model a bit by using a binary starting label. The performance of such unit can be described as follows: , if t2 j1 w x t2 j , j 1, k / 2, y , if t2 j w x t2 j1, j 0,1, k / 2, (3) where x x1, , xn is an input vector, w x w1x1 ... wn xn , 0,1 is a binary label of the neuron, is its negation, and y is the output value of the neuron. Further, we shall consider only binary-valued neurons that obey (3) and their attribute “binary-valued” will be omitted. The corresponding labeled activation function will be denoted as ft,2 .
The labeled k-threshold neuron is completely defined by the triplet w, t, . As well as in [
Two sets A Rn and A Rn are called k-separable [18] if there exists a k-threshold neuron with structure w, t, such that
for all x A ft,2 w x 1 and for every x A ft,2 w x 0 .
In this case labeled k-threshold neuron with the structure w, t, produces the k-dichotomy
A , A of the set A A A in two disjoint sets A and A . Therefore, k-threshold neurons
can be used as binary classifiers. But, as stated in [18], the computational power provided by a
single multithreshold unit is insufficient for the design of efficient multiclass classifiers [
Model (3) has a simple geometrical interpretation [18]. The family of parallel hyperplanes H j : w x t j , j 1, ..., k divides the space Rn by k 1 parts, which can be successively labeled by numbers 0, 1, …, k. In the case 1 all points belonging to parts with even indices are attributed as “negative”, whereas “odd” parts are considered as “positive” [18]. If 0 , then the order of parts should be changed to opposite. The illustration is shown in Figure 1, where the case n 2, k 4, 1 is considered, and the sets of blue (negative) and yellow (positive) points are 4-separable.
Figure 1 can also illustrate the nature of difficulties related to the application of a binaryvalued k-threshold neuron. Its value can alternate k times. It ensures the greater capability of the neuron on the one hand, but results in the hardness of its training on the other hand.
We can overcome the limitation of a single multithreshold neuron by using a NN in which multithreshold neurons are connected layer-wise. The simplest architecture has only two computation layers and its hidden layer consists of labeled multithreshold neurons, whereas outputs layer contains single-threshold nodes each of them corresponds to a separate class of patterns. The architecture of a 2-layer feedforward fully-connected multithreshold neural network is shown in Figure 2.
The NN has n nodes in the input layer, h multithreshold neurons in the hidden layer and l threshold neurons in the output layer, where l is a predefined number of classes. Note that the hidden layer can be a bit heterogeneous in the sense of the number of thresholds and labels of neurons. Namely, the same number of thresholds is not required for neurons in this layer. This assumption can make network synthesis easier and may reduce the size of memory that is necessary to store all network parameters.
Consider how multithreshold NN can be used to perform the desired partition of patterns in ndimensional space. The example of the network performance in two dimensions is presented in Figure 3.
Circles are members of the first class C1. The second class C2 is formed by triangles, and squares belong to the third class C3. Note that both C1 and C3 are formed by 2 compact clusters, which are denoted using multiple occurrences of class names. The presented partition can be produced by 2-layer multithreshold NN, which have 2-4-3 topology. The hidden layer of the network consists of 4 multithreshold neurons each of them is responsible of the recognition of patterns belonging to one of 3 classes. For the proper classification of patterns from class C1 a single neuron is sufficient. The same is true for C2. The first class C1 requires the use of multithreshold neuron with 3 thresholds, whereas a single bithreshold neuron suffices for the second class. The proper classification of patterns belonging to the third class C3 requires the use of 2 bithreshold neurons (one of them can be even replaced by a single-threshold neuron). The hyperplanes that form bounding surfaces of multithreshold neurons are drawn in solid lines.
Each hidden neuron makes its own primary “slice” whose width depends on neuron two thresholds, which in turn are defined by the relative position of patterns. Every multithreshold neuron recognizes at least two patterns of corresponding class, with possibly some additional points. All primary slices are equidistant from their pivot lines depicted in Figure 3 in dashed line. For the neuron corresponding to the first class one more (unbounded) slice is required, which is performed using an additional third threshold. The corresponding line is marked in Figure 3 using an arrow. The label 0 should be assigned to this neuron, whereas all other neurons in the hidden layer would be labeled with 1. Every single-threshold neuron in the output level can be connected only with neurons corresponding to its own class and performs only the disjunction of its inputs.
Consider the network performance in the prediction mode. Let P1 and P2 are new instances presented to the network. Using (3) and the rule of the action of the output neurons we can conclude that network classifies P1 and P2 as members of C1 and C3, respectively. It should be noted that classification of patterns P3 and P4 seems less clear. We shall return to this issue later.
Consider the task of the synthesis of multithreshold NN. Essentially, multithreshold NN with l outputs computes a vector mapping F : Rn 0,1l . The question arises how to design of a computationally efficient algorithm for synthesis a network, which is able to produce a given partition C1, ,Cl of the finite set C into l disjoint nonempty sets C , ,Cl , i.e., for all x C 1 Fi x 1 x Ci , where i 1, ,l .
In the algorithm design the approach to NN synthesis shall be used that is the generalization
of similar techniques used in [
Let a collection of pairs x1, y1 , ,xm , ym be our training set, where m m n is its size, xi Rn is a training pattern provided with its label yi 1, ,l and yi j implies that the corresponding pattern xi belongs to the class C j , i 1, , m . Suppose that all patterns in the sequence X x1, , xm are in general position in n-dimensional space—that is, every subset of n or fewer patterns consists of linearly independent elements. Assume also that X i X Ci , i 1, , m, y y1,..., ym . Consider the following pseudocode of the network synthesis algorithm:
MultithresholdSynthesis(X, y, α, p) Stage 1. Synthesis of the hidden layer 1.1 Z X1, , X l 1.2 h 0 1.3 while Z : 1.4 select a random set X i from Z 1.5 h h 1 1.6 ch i 1.7 r min n, Xi 1.8 Move r randomly chosen patterns from Xi into the matrix A 1.9 Solve linear system w AT 1 1.10 min | w x 1| x X \ Xi 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 1.20 1.21 1.22 1.23
Xi Xi \ x Xi | w x 1| t 1 , 1 1 foreach a X i : s w a 1 min s w x x X \ Xi , w x s 2 min w x s x X \ Xi , w x s B x x Xi , 1 w x s 2 if B p :
X i X i \ B if 1 is well-defined: insert s 1 in the ordered vector t else: 0 if 2 is well-defined: insert s 2 in the ordered vector t
if X i : remove Xi from S if
ch i : vij 2 else: vij 0
The presented algorithm has 2 parameters X—the sequence of training patterns, y—the target vector of class labels for all patterns (is used only in step 1.1), and two hyperparameters α and p. The synthesis process is divided by 2 stages. The main first stage defines the creation of the network hidden layer consisting of multithreshold neurons with variable number of thresholds. The second stage is very simple and defines the output layer consisting of single-threshold neurons.
In the above algorithm h is the counter responsible for the current number of hidden multithreshold neurons. After the completion of the first stage h equals the size of the hidden layer. Inside a while loop the portion of unhandled patterns is selected, which belong to the same class. Then r patterns are moved into rows of auxiliary matrix A. In step 1.9 the vector of length r is denoted by 1, which consists solely of 1’s. Steps 1.8–1.12 are intended to determine the parameter of the “main slice” of multithreshold neuron which is defined by the weight vector w from step 1.9—a normal vector of the two parallel hyperplanes that perform this slice using thresholds 1 and 1 , respectively. Such slices are crucial for MultithresholdSynthesis, because each of them properly classifies at least n patterns of ith class (when patterns are in general position). The nested loop in steps 1.14–1.23 is intended to try candidates for secondary slices. Each such slice is taken into account only if it detects at least p new properly classified patterns, where p is the hyperparameter of the synthesis algorithm.
Note also that 1 and 2 determine the width of corresponding secondary slice and can be undefined. In this case at most a single threshold will be added into the threshold vector. It can result in multithreshold neurons with odd number of thresholds. If 1 is undefined, then 0 is assigned to the neuron label. It is useful in the case when all remaining class patterns are only training instances lying in the “negative” half-space produced by corresponding hyperplane w x s .
As mentioned in [
In step 2.5 we denote by (vi, 1) the ith single-threshold neuron with the weight vector vi vi1, , vih and the unit threshold.
The following proposition estimates the complexity of the considered algorithm.
Proposition. An arbitrary partition of m-set X into l disjoint nonempty sets X1, …, Xl of ndimensional vectors can be performed by a 2-layer multithreshold NN with at most m / n l nodes in the hidden layer and l neurons in the output layer, whereas the network can be synthesized using MultithresholdSynthesis(X, y, α, p) in O m3 time.
Let us consider the capability of MultithresholdSynthesis to synthesis a multithreshold multiclass classifier to solve the classification problems compared to classical classifiers, such as k-nearest neighbor, decision tree, random forest, RBF SVC (support vector classifier with RBF kernel), MLPClassifier (multilayer perceptron or fully-connected feed-forward NN, using softmax function for estimates the probability of the target class over all possible target classes and crossentropy loss function). Classifiers were tested on the “Wine Quality” dataset [31], related to red and white variants of the Portuguese "Vinho Verde" wine provided by “UC Irvine Machine Learning Repository” [32], as well as other popular repositories, e.g., [33]. This medium-sized dataset contains 6497 instances, 4898 of which are related to white wine, and other 1599—to red one [31].
The data includes only 12 features, 11 physicochemical properties as inputs (see [34]), wine color and one sensory quality score as the output from 0 to 10 (actually, for red wine the quality is between 3 and 8, and for white wine—between 3 and 9). The distribution of qualities is shown in Figure 4. Note that only 9 instances have the quality value 9 and they are almost invisible in Figure 4. The most frequent quality value is 6 with 2836 representatives. Thus, the classes are very imbalanced. Moreover, not all input features are supposed to be quite relevant [31]. Thus, the training task for this dataset can be considered hard enough to estimate classifier ability [34].
Dataset contains only numerical data (except the color value). Thus, the feature extraction was
very simple. Data were standardized by removing the mean and scaling to unit variance using
standard scaler from Scikit Learn [
For every classifier stratified 5-fold cross-validation was used in order to evaluate the classifier performance and to tune its hyperparameters. The average performance measure over all 5 splits and grid search was employed for this purpose.
For k-nearest neighbor classifier only the number of neighbors was explored. For multilayer perceptron Adam optimizer was used and the number of hidden layer as well as their sizes, the kind of activation function and batch size were investigated. For support vector classifier with RBF kernel different combinations of values were tried of regularization parameter C, degree of the polynomial kernel function and kernel coefficient gamma [35]. In the case of random forest classification, the number of trees in the forest and the maximum depth of the tree were chosen. The last parameter also was studied for decision tree classifier.
Both hyperparameters α and p were explored for MultithresholdSynthesis(X, y, α, p). {0.7, 0.8, 0.85, 0.9, 0.95, 1, 1.05, 1.1, 1.15, 1.2} and {2, 3, 4, 5, 6, 7} ranges were used for α and p, respectively. During the experiments we used implementations of 5 above-mentioned popular classifiers provided by Scikit-Learn library [35].
Two classification metrics were used during simulation. The first was the accuracy, the second—top-2 score (successful when the target class is within the top 2 predictions) [35].
The following two tables contain results of experiments in the case of first and second metrics, respectively. For first 5 classifiers best between all specified combinations of hyperparameter values are not shown for sake of brevity (and because their study is beyond the main scope of the paper). Note that all classifiers were retrained one more time on full training set after the finish of cross-validation and grid search. That is why the values of metrics on the test set are often higher than average across all splits during cross-validation.
Results for two versions of multithreshold 2-layer NN are presented in last two rows in both tables. In the synthesis of NN1 α = 1 was used. NN2 was synthesized using the best value α* = 1.05 found by the grid search. In both networks the best value p* = 4 was used.
By analyzing the performance results shown in Table 1 and Table 2, we can conclude that:
The basic version of multithreshold network NN1 was good memorizer and performed perfectly on the training set.
The generalization ability of NN1 was very poor. Its performance on the test set was worst for both considered metrics.
The optimized version NN2 lost a bit in the accuracy on the training set, but performed considerable better on new patterns.
The proper choice of the α allow us to avoid the great overfitting.
The test error during cross-validation varies significantly.
The use of value of lying inside the half-interval (0, 1] results in the degradation of the generalization capability of the classifier.
The most typical number of thresholds of the neuron in the hidden layer of NN2 was 3. The growth of the value of hyperparameter p results in almost bithreshold hidden layer. Using the modified software implementation and additional memory it is possible to decrease the time complexity of MultithresholdSynthesis(X, y, α, p) in Proposition from O m3 to O m2 n log m .
The estimated hidden layer size m / n l from Proposition is too pessimistic. The actual size in the experiment was 231. But it seems plausible that the number of nodes in the hidden layer growths as m / n .
The proposed model of 2-layer multithreshold NN and corresponding synthesis algorithm can be
considered as the first step towards an application of multilayer NN based on the multithreshold
paradigm in machine learning in the case of an arbitrary number of thresholds. Even the first
applications of such networks in classification on the real-word datasets showed that they have
the similar pros and cons that bithreshold neural networks considered in [
The principal consequences that follow from concrete results of simulation concerning the pattern classification on synthetic [35] as well as real-world datasets [32, 33] using multithreshold 2-layer NN suggest that networks yielded by MultithresholdSynthesis or its more complicated modifications have good performance on the design set but are rather inefficient on new instances. More precisely, as it was observed that for “Wine Quality” and other datasets:
Making prediction, classifier can attribute new patterns to a class whose all representatives in the training set are very distant from this pattern (“nonlocality” of classifier [24]).
The size of the hidden layer of the network strongly depends on the order in which the training patterns were presented to learner. Moreover, during the cross-validation the NN synthesized on different splits demonstrated the significant diversity of the size of the hidden layer.
The size of the hidden layer is too large in the case of the larger datasets.
This downsides of the basic multithreshold 2-layer NN classifier is illustrated in Figure 3. The network classifies pattern P3 as a member of first class despite it looks more likely be a member of one of two other classes (first drawback). Moreover, the network is undecisive when it predicts a class for new pattern P4, despite it is situated in the proximity of a compact cluster of members of the first class (second drawback).
A conjecture was stated in [
It should be noted that the more expensive part of MultithresholdSynthesis algorithm is computations in step 1.10, which have the O(m2n) contribution in its time complexity. These computations can be performed simultaneously with solely synchronization of the shared variable ε. Therefore, MultithresholdSynthesis has massively parallel structure and can be effectively realized using multicores processors and graphical accelerators.
The ways of the use of multithreshold paradigm in neural computation were treated in the paper. The new modification of the model of binary-valued multithreshold neural unit was proposed, which allows often to reduce number of thresholds. The 2-layer neural network architecture was studied whose hidden layer consists of multithreshold neurons with different number of thresholds. The synthesis algorithm was developed for such multithreshold NN. The multiclass classifier was designed on the base of such architecture.
Experiment results of the performance of multithreshold classifier on “Wine Quality” realworld dataset were presented. These results clarify pros and cons of the proposed model compared to some popular classifiers. The impact of algorithm hyperparameters on the classifier ability to make a precise prediction, as well as and the ways to improve it were discussed. The main conclusion is that there exists a need in the development of localized modifications of the multithreshold activation function and additional hyperparameters in synthesis algorithm in order to regulate the network performance more precisely.
[16] Z.-G. Zhang, Y.-L. Xiao, J. Zhong, Unitary learning in conditional models for deep optics neural networks, in: Proceedings of SPIE – The International Society for Optical Engineering, volume 12565, 2023, no. 1256543. [17] R. Takiyama, Multiple threshold perceptron, Pattern Recognition 10.1 (1978): 27–30. [18] V. Kotsovsky, A. Batyuk, Multithreshold neural units and networks, in: Proceedings of IEEE 18th International Conference on Computer Sciences and Information Technologies, CSIT 2023, Lviv, Ukraine, 2023, pp. 1-5, doi: 10.1109/CSIT61576.2023.10324129. [19] N. Jiang, Z. Zhang, X. Ma, J. Wang, Y. Yang, Analysis of nonseparable property of multivalued multi-threshold neuron, in: Proceedings of 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China, 2008, pp. 413-419, doi: 10.1109/IJCNN.2008.4633825. [20] I. Prokíc, Characterization of multiple-valued threshold functions in the Vilenkin-Chrestenson basis, Journal of Multiple-Valued Logic and Soft Computing 34.3-4 (2020): 223–238. [21] R. Takiyama, The separating capacity of a multithreshold threshold element, IEEE Transactions on Pattern Analysis and Machine Intelligence. PAMI-7.1 (1985): 112–116. [22] K. Ashenayi, J. Vogh, M.R. Sayeh, B. Karimi, T. Baradaran, Multiple threshold perceptron using sinusoidal function, International Journal of Modelling and Simulation 12.1 (1992): 22–26. [23] S. Olafsson, Y. S. Abu-Mostafa, The capacity of multilevel threshold function, IEEE Transactions on Pattern Analysis and Machine Intelligence 10.2 (1988): 277–281. [24] V. Kotsovsky, A. Batyuk, Feed-forward neural network classifiers with bithreshold-like activations, in: Proceedings of IEEE 17th International Scientific and Technical Conference on Computer Sciences and Information Technologies, CSIT 2022, Lviv, Ukraine, 2022, pp. 9–12. [25] N. Jiang, Y. X. Yang, X. M. Ma, and Z. Z. Zhang, Using three layer neural network to compute multi-valued functions, in 2007 Fourth International Symposium on Neural Networks, June 3-7, 2007, Nanjing, P.R. China, Part III, LNCS 4493, 2007, pp. 1-8. [26] V.K. Venkatesan, I. Izonin, J. Periyasamy, A. Indirajithu, A. Batyuk, M.T. Ramakrishna, Incorporation of energy efficient computational strategies for clustering and routing in heterogeneous networks of smart city, Energies 15.20 (2022): 7524. [27] Y. Andrashko et al., A method for assessing the productivity trends of collective scientific subjects based on the modified PageRank algorithm, Eastern-European Journal of Enterprise Technologies, 1.4 (121) (2023): 41–47. [28] V. Kotsovsky, A. Batyuk, V. Voityshyn, On the size of weights for bithreshold neurons and networks, in: Proceedings of IEEE 16th International Conference on Computer Sciences and Information Technologies, CSIT 2021, Lviv, Ukrain, 2021, volume 1, pp. 13–16. [29] E. B. Baum, On the capabilities of multilayer perceptrons, Journal of Complexity 4.3 (1988): 193–215. [30] V. Kotsovsky, “Hybrid 4-layer bithreshold neural network for multiclass classification,” in
CEUR Workshop Proceedings, volume 3387, 2023, pp. 212–223. [31] P. Cortez, A.L. Cerdeira, F. Almeida, T. Matos, J. Reis., Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems 47 (2009): 547–553. [32] M. Kelly, R. Longjohn, K. Nottingham, The UCI machine learning repository, 2023. URL: http://archive.ics.uci.edu. [33] OpenML: A worldwide machine learning lab, 2024. URL: https://openml.org. [34] D. C. Angus, Modeling wine quality from physicochemical properties, (2019). URL: https://api.semanticscholar.org/CorpusID:209521363. [35] Scikit-learn: Machine Learning in Python, (2024). URL: https://scikit-learn.org.