Hopfield networks, renowned for their associative memory capabilities, have been extensively studied since their inception. Recent advancements in activation functions have sparked interest in exploring their applicability within the framework of these networks. This paper investigates the integration of chaotic activation functions into Hopfield networks and their implications on network dynamics and performance.
with a symmetric matrix of connections. In the course of operation, the dynamics of such networks converge to one of the equilibrium positions. These equilibrium positions are local minima of a functional called the network energy (in the simplest case, local minima of a negatively defined quadratic form on an n-dimensional cube). Such a network can be used as an auto-associative memory, as a filter, and to solve some optimization problems. Unlike many neural networks that work until a result is obtained after a certain number of cycles, Hopfield networks work until an equilibrium is reached, when the next state of the network is equal to the previous one. This model describes the process of training and subsequent pattern recognition.
In Hopfield models, activation functions are used to model the behavior of neurons in
the brain. Although linear or sigmoidal activation functions are commonly used, some
studies show that the use of chaotic activation functions can lead to interesting results [
The use of chaotic activation functions can also improve the performance of Hopfield
networks in solving problems with a large number of local minima. Chaotic functions can
help in finding global optima due to their more complex and random nature [
An additional advantage of using chaotic activation functions in Hopfield networks is
their ability to create more complex dynamic dependencies between neurons. This can
lead to an improvement in the network's ability to adapt to changes in input data or to
different environmental conditions. Such chaotic dependencies can help the model
reproduce complex dynamic processes such as memory or bias learning [
It is important to keep in mind that the use of chaotic activation functions requires additional careful tuning of model parameters and selection of appropriate training methods. Incorrect tuning can lead to unpredictable network behavior or failures in the learning process. It should also be borne in mind that chaotic activation functions may be less effective in tasks requiring high accuracy or stability, as their nature may lead to greater variability in results.
The need for a deeper study of the use of chaotic activation functions in Hopfield
networks is becoming increasingly important due to the growing interest in neural
networks and their potential applications in complex information processing tasks.
Although some studies have already shown the promise of this approach, there is
insufficient coverage of this topic in the scientific literature. It is important to conduct
more in-depth experimental and theoretical research to explore the potential advantages
and limitations of using chaotic activation functions in Hopfield networks [
Understanding the dynamics of such networks with chaotic activation functions can lead to important discoveries in the areas of information storage and recovery, as well as in working with large data sets.
Let the variable x(t) denote the activity of the neuron at time t. The state x(t) = 1 corresponds to the "excitation" state, and the state x(t) = -1 corresponds to the "inhibition" state. Let h(t) be an external influence on the neuron, for example, from other neurons. For convenience, we assume that time is discrete (t = 0, 1, 2,...). In [12], it was proposed that the evolution of a neuron is determined by a dynamical system, which is a piecewise linear, piecewise continuous one-dimensional function:
( + ) = ( ( ), ℎ( )) (1)
This family of functions is defined by the function K(h) of the slope angle. This dependence must satisfy the following conditions: 1) parity K(-h) = K(h); 2) monotonic decrease at h > 0 (and, therefore, monotonic increase at h < 0); 3) K(0) = 2. All these requirements are met for the function
2 (ℎ) =
1 + |ℎ|/ The dynamics of a formal neuron is determined by a ratio: (2) ( + 1) = {−11,, ℎℎ(())><00 (3)
That is, without any transient process, x(t) takes the value +1 or - 1 regardless of the neuron's state at the previous moment.
It is known that a real neuron is an extremely complex system of inertia [13, 14, 15, 16]. Its processes have characteristic times ranging from units to hundreds (or more) of milliseconds [17, 18, 19, 20]. In [12], it was assumed that at a large value of external influence h, the neuron "quickly" enters the state of excitation, while at lower values, the speed of transition to the state of excitation decreases. This is taken into account in the above chaotic Izhikevich activation function in (1), because as h decreases, K(h) increases and the transition time increases in (2), although the only special point x = 1 is still stable. When K(h) passes through 1, x = 1 loses stability and the neuron's dynamics becomes chaotic.
The above scenario of the transition from order to chaos in (3) is observed in many discontinuous or sharp-vertex functions and is called a homoclinic explosion [21]. Previous studies have shown that the results are weakly dependent on the specific type of the function K(h) if this scenario is chaos. The most important parameter to display is only the value of h = p, in which K(h) = 1, i.e., at which the transition to chaos occurs.
Differentiable activation functions in neural networks have a number of significant advantages over their undifferentiable counterparts. First of all, they allow efficient use of optimization algorithms, such as back-propagation, to train the network. The smooth nature of differentiable functions allows you to find optimal parameters by searching for gradients, which simplifies the training process and improves the convergence rate.
It is important to keep in mind that differentiable activation functions allow the use of regularization methods that reduce network overtraining and improve its overall generalization ability. For example, you can apply L1 or L2 regularization to the model weights, which helps to control their complexity and avoid overfitting.
In addition, differentiable activation functions provide a smooth and continuous relationship between the network's inputs and outputs, which facilitates smooth transitions between values and eases optimization processes. They also allow the use of more powerful optimization algorithms, such as adaptive gradient descent methods, which contributes to faster and more efficient network convergence.
The chaotic Izhikevich activation function described in the previous section has a drawback, it is not differentiable for small values of h(t). This drawback can be solved by replacing the linear function with a parametric Bézier function (check Figure 1). Such a replacement allows to ensure the differentiability of activation functions by fulfilling the condition on the bijectionality of the equation x(p), where x is a vector of input values and p is the parameters of the Bézier function.
The following experiment was conducted to compare the activation functions of the Hopfield network - a signum, a chaotic Izhikevich activation function, and its differentiable modification. The input data were samples from normal distributions ( , 2) of two classes. The parameters μ and σ2 were chosen to cover a variety of cases of intersection of the distributions. Training images were built on the training data according to each category of binary classification, and the quality of the models was evaluated on the test data using Leave-One-Out validation, with the F1-score as the target metric.
The null hypothesis is that the activation functions do not affect the model results. The alternative hypothesis is that chaotic activation functions produce better results.
Consider the case when the distributions intersect, 1 = 0.4, 2 = 0.6, 12 = 22 = 0.05. After 100 experiments, using the signum as an activation function, an average F1-score of 0.81 was obtained. Using statistical power, the required number of experiments for the incremental increase to the mean value of the F1-score to be significant can be obtained. Accordingly, at least 23664 experiments are needed for each group for the z-test to be significant for an increase of at least 1%.
24000 experiments were conducted for each of the activation functions compared (signum, chaotic Izhikevich activation function and its differentiable modification), and calculated F1-scores for each experiment. The average value of the F1-score for the experiments with the signum was 0.9, with the chaotic Izhikevich activation function was 0.91, and with the differentiable modification was 0.911. After that, z-tests were performed for each pair of experiments (check Table 1).
For the experiments with different mathematical expectation, the previous tendency has been preserved: chaotic functions show a statistically significant improvement over signum, but the differentiable modification shows better results.
Both samples with chaotic functions are statistically significantly different from the sample of experiments with signum, but do not have a statistically significant difference between themselves, although the modified function gave better results, as can be seen from the z-statistics.
Similarly, experiments with other parameters of normal distributions were conducted, calculated F1-scores for each sample, and conducted paired z-tests (check Table 2).
In the experiments with the same mathematical expectation, all functions showed similar results and have no statistically significant difference.
Now let's consider the effectiveness of chaotic activation functions in an experiment with real data.
Some of the first reports of malignant changes appeared in the 1960s, when the content of X-chromatin in somatic cells was widely studied and its instability became apparent in various functional changes in the body and general pathology of somatic cells. In the presence of tumors in the body, there are obvious changes in the content of X-chromatin in the buccal epithelium and peripheral blood neutrophils. It has also been shown that changes in the number of cells with X chromosomes are caused by a defect in the functional state of the heterozygous X-chromosome [22].
Studies demonstrating changes in buccal epithelial cells in patients with tumors are of great interest: in the 1960s, H. Nieburgs and co-authors [23] reported a characteristic redistribution of chromatin mass in somatic cells in 77% of cancer patients and called these changes tumorigenic. These changes were characterized by an increase in the size of epithelial cell nuclei and an increase in the size of "restricted" chromatin regions surrounded by light areas [24]. The same changes were observed in cells of the liver, kidneys and other organs. In patients with breast cancer, an increase in DNA content in the buccal epithelium and the size of interphase nuclei was found. However, some authors did not find a significant difference in this indicator between patients and practically healthy men when the DNA content of buccal epithelial cells in men with bronchial epithelioma was measured by cellular spectrophotometry [25].
Study [26] examined a control group (29 patients), a group of patients with stage II breast cancer (68 patients), and a group of patients with fibroadenomatosis (33 patients). All diagnoses were confirmed by histology. The morphologic dataset consisted of 20256 images of interphase nuclei of the buccal epithelium (6752 nuclei were scanned in three versions: without filtering, with a yellow filter, and with a purple filter).
The morphological material was smears of epithelial cells of the oral mucosa in the middle depth of the spinous layer. On average, each preparation consisted of 52 cells. The content of DNA-fuchsin in the nuclei of epithelial cells was calculated as the product of optical density and area. At the first stage of the analysis, an image of the chromatin distribution was obtained in the form of a 128 x 128 pixel matrix [26].
To reflect the fractal nature of the chromatin distribution and to ensure invariance to image rotation, a spatial fill curve was created along each pixel of the image, and the RGB color values of the image pixels were read sequentially rather than line by line. As a result, the pixel matrix could be mapped to three vectors corresponding to the three channels of the RGB color model. The Sierpinski curves [26] were used as space filling curves.
In order to apply the fractal image analysis method, the image must be pre-processed. For this purpose, the Otsu method was applied [26]. This method is used for threshold binarization of halftone images. The algorithm assumes that there are two classes of pixels in the image (main and background), and searches for the optimal threshold value that divides the pixels into two classes so that the intra-class variance is minimal.
There are several methods for calculating the fractal dimension of an image. In [26], the Hurst index was chosen because it is very suitable for sequential analysis. The Hurst index is related to the fractal dimension D by the formula H = 2 - D.
The input data for the model were three-channel (RGB) samples of fractal kernel dimensions for each patient. The samples differed significantly in the number of elements. Therefore, when preparing the data before training the neural network, n quantiles were calculated for each sample, where n is the number of elements in the smallest sample (check Figure 4).
Additional datasets were created from the input three-channel RGB data. The following input data samples were used in the experiments: RGB (three channels at the same time), R (red only), G (green only), B (blue only), Gray (gray channel calculated using the formula 0.299 ∙ R + 0.587 ∙ G + 0.114 ∙ B), Mean (arithmetic mean of three channels).
To build training images for the Hopfield network, two methods of quantile aggregation were used: arithmetic mean and median.
At this stage, the training images were sequences of real numbers. The neurons of the Hopfield network can have two values: 1 or -1. Therefore, additional binarization was applied to the images: real values were rounded to the ninth decimal place, converted to binary and concatenated into the final binary portrait (check Figure 5).
A total of 144 experiments were set up and divided into 4 categories: cancer patients vs. healthy controls, cancer patients vs. healthy controls and fibroadenomatosis patients, cancer patients vs. fibroadenomatosis patients, fibroadenomatosis patients vs. healthy controls.
Training images were built on the training data according to each category of binary classification, the quality of the models was evaluated on the test data using Leave-OneOut validation, and the target metrics were precision, recall, and F1-score.
In the comparison "cancer patients vs. fibroadenomatosis patients", the differentiable modification showed better results in the experiments with the best results, the median being a better aggregation function than the arithmetic mean. The highest accuracy was obtained with input data in RGB format and single-channel inputs R, G, and B (check Table 3).
In the comparison "cancer patients vs. healthy controls", the differentiable modification showed similar results to the chaotic Izhikevich function. Both chaotic functions gave better results than signum in most experiments. The median and arithmetic mean gave similar results in similar experiments. The most accurate results were obtained with input data in B and RGB formats (check Table 4). N Experiment Input Aggregation F. activation 1 BC_vs_C+FAM B avg bezier 2 BC_vs_C+FAM B avg izh 3 BC_vs_C+FAM B avg sign 4 BC_vs_C+FAM B median bezier 5 BC_vs_C+FAM B median izh 6 BC_vs_C+FAM B median sign 7 BC_vs_C+FAM G avg bezier 8 BC_vs_C+FAM G avg izh 9 BC_vs_C+FAM G avg sign 10 BC_vs_C+FAM G median bezier 11 BC_vs_C+FAM G median izh 12 BC_vs_C+FAM G median sign 13 BC_vs_C+FAM gray avg bezier 14 BC_vs_C+FAM gray avg izh 15 BC_vs_C+FAM gray avg sign 16 BC_vs_C+FAM gray median bezier 17 BC_vs_C+FAM gray median izh sign bezier izh sign bezier izh sign bezier izh sign bezier izh sign bezier izh sign bezier izh sign N Experiment 1 FAM_vs_C 2 FAM_vs_C 3 FAM_vs_C 4 FAM_vs_C 5 FAM_vs_C 6 FAM_vs_C 7 FAM_vs_C 8 FAM_vs_C 9 FAM_vs_C 10 FAM_vs_C 11 FAM_vs_C 12 FAM_vs_C 13 FAM_vs_C 14 FAM_vs_C 15 FAM_vs_C 16 FAM_vs_C 17 FAM_vs_C 18 FAM_vs_C
In the comparison "cancer patients vs. healthy controls and fibroadenomatosis patients", all three activation functions showed similar results. Similarly, there are no overperforming alternatives among the aggregations and input data formats (check Table 5).
In deep learning, especially in the context of artificial neural networks, the use of chaos in activation functions plays a significant role in ensuring the efficiency and flexibility of models. Chaotic activation functions allow to enrich the feature distribution space, which allows the model to adapt to various conditions and maintain resistance to noise in the data. This approach increases the robustness of models to changes in training data and can improve their overall ability to generalize to new data. The use of chaotic activation functions can also be important in cases where the model needs to adapt to unpredictable or nonlinear relationships in the data, making it more versatile and powerful in solving complex machine learning problems.
Also, activation functions that can be differentiated in neural networks have several important advantages over their undifferentiable counterparts. First of all, they allow for the efficient use of optimization algorithms, such as back-propagation of errors, to train the network. The smooth nature of differentiable functions allows finding optimal parameters using gradient descent, which simplifies the training process and improves the convergence rate.
The paper substantiates and demonstrates the advantages of chaotic activation functions of the Hopfield model on the example of experiments with artificial samples from normal distributions and on real data in predicting breast cancer.
This area still requires further research and experimentation, and the development and implementation of new chaotic activation functions may be a promising way to improve deep learning not only in Hopfield models but also in alternative algorithms and neural network architectures and improve results in various tasks.
Authors would like to express their sincere gratitude to K. Golubeva and N. Borodai for generously providing the data used for training the models in this study. [11] X. Liang, Y. Yang, R. Wang, J. Chen, Synchronization of delayed stochastic reactiondiffusion Hopfield neural networks via sliding mode control, Nonlinear Analysis Modelling and Control 29 (2024) 1-19. doi:10.15388/namc.2024.29.34884. [12] E. Lieberman-Aiden, Comprehensive mapping of long-range interactions reveals folding principles of the human Genome, Science 326 (2009) 289-193. doi:10.1126/science.1181369. [13] A. Babloyantz, S. Nicolis, M. Salazor, Evidence of chaotic dynamics of brain activity during the sleep cycle, Physics Letters A. 111 (1985) 152-156. doi:10.1016/03759601(85)90444-X. [14] W. Freeman, Simulation of Chaotic EEG Patterns with a Dynamic Model of the Olfactory System, Biological Cybernetics 56 (1987) 139-150. doi:10.1007/BF00317988. [15] M. Hirsch, Convergent Activation Dynamics in Continuous Time Neural Networks,
Neural Networks 2 (1989) 331-351. doi:10.1016/0893-6080(89)90018-X. [16] J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc Natl Acad Sci U S A. Apr; 79 (1982) 2554-2558. doi:10.1073/pnas.79.8.2554. [17] G. Shepherd, The synaptic organization of the brain, Oxford University Press (1990) doi:10.1093/acprof:oso/9780195159561.001.1. [18] C. Skarda, W. Freeman, How brains make chaos in order to make sense of the world, Behavioral and Brain Sciences 10 (1987) 161-195. doi:10.1017/S0140525X00047336. [19] I. Tsuda, Dynamic link of memory - Chaotic memory map in nonequilibrium neural networks, Neural Networks 5 (1992) 857-893. doi:10.1016/S0893-6080(05)80029-2. [20] Y. Yao, W. Freeman, Model of Biological Pattern Recognition with Spatially Chaotic Dynamics, Neural Networks 3 (1990) 153 - 170. doi:10.1016/0893-6080(90)90086Z. [21] S. Sparrow, The Lorenz Equations. Bifurcations, Chaos, and Strange Attractors,
Applied Mathematical Sciences 41 (1982) 26-50. doi:10.1007/978-1-4612-5767-7. [22] M. Radak, H. Lafta, H. Fallahi, Machine learning and deep learning techniques for breast cancer diagnosis and classification: a comprehensive review of medical imaging studies, Journal of Cancer Research and Clinical Oncology 49 (2023) 1047310491. doi:10.1007/s00432-023-04956-z. [23] H. Nieburgs, Recent progress in the interpretation of malignancy associated changes (MAC), Acta Cytologica 12 (1968) 445-453. doi:10.1155/238921. [24] N. Boroday, V. Chekhun, E. Golubeva, D. Klyushin, In vitro and in vivo densitometric analysis of DNA content and chromatin texture in tumor cell nuclei under the influence of a nano composite and magnetic field, Advances in Cancer Research & Treatment 2016 (2016) 1-11. doi:10.5171/2016.706183. [25] G. Ogden, J. Cowpe, M. Green, The effect of distant malignancy upon quantitative cytologic assessment of normal oral mucosa, Cancer 65 (1990) 477-480. doi:10.1002/1097-0142(19900201)65:3<477::AID-CNCR2820650317>3.0.CO;2-G. [26] D. Klyushin, K. Golubeva, N. Boroday, D. Shervarly, Breast Cancer Diagnosis Using Machine Learning and Fractal Analysis of Malignancy-Associated Changes in Buccal Epithelium, Artificial Intelligence, Machine Learning, and Data Science Technologies Future Impact and Well- Being for Society 5 (2021) 1-18. doi:10.1201/9781003153405-1.