From year to year information processing algorithms based on neural networks(NN) are gaining more and more popularity. Such networks are characterized by a large number of hidden layers and huge amounts of training data, requiring a specialized high-performance device. In the past few years the main platforms for implementing hardware neural networks have been FPGAs and GPUs. In all power-limited scenarios FPGAs are the natural choice. Developers of specialized computing systems are interested in porting these algorithms to embedded computing platforms. However, there are limitations that prevent the use these algorithms for managing technical systems and objects. One of the limitations is the lack of a mathematical apparatus that would formally evaluate the compliance of real-time constraints for arti cial neural networks(ANN). To solve this global problem, a particular problem is solved in this paper. There was proposed and tested a method for validating the temporal characteristics of hardware arti cial neural networks implemented on FPGA. This method is suitable for use on neural networks transferred to a hardware platform with pre-selected coe cients, and for neural networks trained directly on the target platform. The method is based on the apparatus of queuing networks (QN), which allow the validation of the temporal characteristics of arti cial neural networks according with the hardware features of the target computing platform. The presented method has an applied focus. This paper presents the results of experimental testing of the proposed method and proved the feasibility of using the queuing network apparatus for validating the temporal characteristics of arti cial neural networks.
Arti cial neural networks applications are becoming more popular. They are
using for control of technical and cyberphysical systems. High performance and
energy e ciency are very important for such systems. It can be achieved due to
hardware implementation of calculators. In recent years FPGAs and GPUs are
main platforms for hardware neural networks. Not only functional predictability
of the behavior, but also temporal predictability are critical condition of using
neural networks in control systems. Both of these aspects of predictability are
not resolved issues. The article is focused on the second aspect - temporary
predictability of the behavior. These days there are no established methodologies for
the con rmation of real-time requirements implementation. The article discusses
the opportunities for modeling of hardware neural networks which are based on
FPGAs using queueing networks.The work shows that [
In this paper it is proposed to consider consider a way of neural networks
modeling in order to con rm real-time requirements. In the article [
Arti cial neuron consists of multi input adder, unit of activation and set of coe cients (synaptic weights, o set, activation function parameters) that are stored in memory. The memory is also used for hardware tabled realization of function`s activation. Among modern technologies, only FPGA is able to provide a lot of independent memory units with superfast access. Thus, it caused the popularity of FPGAs for ANN development and this also means that the memory can be the bottleneck during realization ANN based on FPGAs.
The structure of NN can be portable on structure of open queueing network (OQN). Each neuron is a node of OQN, which can be presented as a service device without queue. The time of service is determined by speed of FPGA. Despite of visible "hard" digital circuit synchronization, the time of service is not xed and has Gaussian distribution. It is explained by the jitter of the clock signal which is accumulated while sequential calculation of ANN layers. Input data stream (requests) is random, evenly distributed. As mentioned, internal memory is the bottleneck in ANN development based on FPGAs. To save memory it is possible to use one unit of function`s activation by several neurons. Unfortunately, because of parallel calls of neurons to memory units there are collisions and delays. Moreover, there is an issue of NN topologies optimization using performance criterion. The paper describes the developed OQN models, performed simulation, calculated characteristics for ANN-FPGAs with di erent ways of neuron combinations. The main ways to combine neurons are combine inside layers and between layers. These options are presented on Fig. 1, each marked as A, B, C. OQN devices numbers are marked by digits. These devices perform the neurons functions.
As properties of OQN are determined by properties of network nodes, it is necessary to calculate characteristics of each node individually. Nodes characteristics calculation is made considering temporal characteristics for tabled FPGA realization of neuron with average working time 90 microseconds. These characteristics are presented in Table 1.
Simulation proved dependence of temporal characteristics on node-neuron combination (Table 1 and 2). Using simulation tools it was revealed that some requests in node 2 stayed in a queue, thereby node 2 has become the bottleneck while increasing ow request. It is impractical to have a large amount of memory for other nodes in case of described request ow. Given earlier QN model option (FPGA-ANN) showed the potential evaluation of temporal network characteristics. Futhermore, due to simulation tools this option de ned some systems
bottlenecks and pointed how memory resource optimization important is.Thanks to OQN network-wide characteristics there ia s possibility to identify the maximum allowable intensity ot the requests input stream, thereby test the technical parameters of the developed system.
n W = X aj wj 1.187
j=1 n U = X aj uj 381.929 380.917 383.867 j=1 n L = X lj 0.97 1.023
1.417 0.94 2.83 0.99 2.87
Often there is a necessity to provide the training of neural networks directly on target computing platform. It is possible to determine three main ways how to provide the training of neural networks: { training with a teacher - The training with a teacher assumes the availability of the complete set of labeled data for model training at all development stages. The availability of complete marked dataset means that every example in training set must comply with the answer that the algorithm should receive. Generally the training with a teacher is used to solve classi cation and regression problems. In classi cation problems the algorithm o ers discrete values that correspond to class numbers to which the objects belong.
Regression problems are related to continuous data. { training without a teacher - In the training without a teacher the model has the data set and there is no exact directions of what to do with this data. Neural network itself tries to nd data correlations by extracting useful features and analyzing them. Typically, such training method is used by neural networks the purpose of which is to group data according to certain parameters. { training with a partial involvement of a teacher - Training data set contains labeled data as well as unlabeled. { training with a support | It is one of machine learning methods during which the test system (agent) is trained, interacting with some environment.
In this paper the method of training with a teacher will be considered. Usually the backpropagation method is used for neural networks training. The main purpose of this method is propagation of error sygnals from the network output to its input. The propagation occures in the opposite direction to the direct propagation of signals in normal operation. Therefore, to realize this method on hardware computing platform is necessary to add the error computing unit to developed neural network. There are three main ways to error calculation: Mean Squared Error (MSE), Root MSE and Arctan. These days there is no uni ed algorithm for error calculation, so the neural network developer should choose himself the method that is the most suitable for target goal. To calculate the error using method Arctan is necessary to use the following formula: arctan2(i1 a1) + arctan2(i2 a2) + :::: + arctan2(in an) i=1 = Xn ( arctan2(ii n n ai) )
ArtanError = where: { i - predictions (expected values or unknown results), { a - observed values (known results). The error always will be greater when using the method Arctan. This is due to the method working principle: the larger the di erence, the greater the error. To calculate the error using method Root MSE is necessary to use the following formula: RM seError = r (i1 a1)2 + (i2 a2)2 + :::: + (in n an)2
n = X(( (ii i=1 n The Root MSE method has smallest error. To calculate the error using method MSE is necessary to use the following formula:
M seError = (i1 a1)2 + (i2 a2)2 + :::: + (in n an)2
n = X( (ii i=1 n ai)2 ) When using MSE method the error value will be average compared to Arctan and Root MSE methods. Therefore, the MSE method is used more often. It provides balance in error calculation. The MSE method will be will be considered for error computation in this paper. Structural scheme of error computation using MSE is presented in the Fig. 2.
This structural scheme consists of di erent units. Firstly, it consists of 2 multiplexers that choose input value for predicted signal and observed signal. Multiplexer control counter is also included in this scheme. Moreover, the scheme consists of subtraction unit that provides subtracting the observed signal value from the programmable signal value. The di erence squaring unit is included in this scheme. It contains the memory unit and multiplier. The scheme consists of adder that provides addition of the value obtained at the output of the squaring unit with the previous value. The output value of the adder goes to the input of memory unit which is reset at the end of the error calculation.
In oder to implement the backpropagation method, it is necessary to calculate the error for each neuron separately starting from the output neuron. The following formula is provided for output neuron error calculation: o = (i
a) f 0 (in) { i - predictions (expected values or unknown results), { a - observed values (known results), { f 0 (in) - derivative activation function To implement the backpropagation method, it is possible to use only those activation functions that can be di erentiate. Derivative function can be represented as follows: f 0 (in) = fsugmoid = (1 a) a This function is provided the neuron error calculation for the inner layer: h = n X (wi (i=1) i) f 0 (in) where where { w - the weight of the output value, { d - the error value of the neighboring neuron.
To calculate the new value of the neuron weight, the following formula is presented: 4wi = E
GRADw + a 4wi 1 { E - training speed, { a - moment - training step size.
To calculate the new value of the neuron weight, it is necessary to obtain the gradient value for the given function. The following formula is provided the gradient value calculation:
GRAD = b outa
For arti cial neural networks (ANN) shown in Fig. 1, the simulation modeling of the training process was performed. To perform the simulation modeling, developed model should be supplemented by a device that imitates the error calculation process. Moreover, functions that imitate the neuron work should be replaced by functions that calculate the new neuron weight. Calculation of the nodal characteristics should be performed according to time characteristics for target platform. Average working time is 40 microseconds and equals function of new neuron value calculation. Average time of ANN error calculation is 5 microseconds. All these characteristics are presented in Table 5.
Dependence of time characteristics on the option of combining neuron nodes was also con rmed for the neural networks simulation training process ( Table 5 and Table 4 ). The simulation modeling results show that in contrast to the process of calculating the time characteristics of the output signal the best way to combine neurons for training process is presented in Fig. 1.C. This is due to the fact that in this case there is a multiplexing of two unconnected neurons, thus the backpropagation method is calculated without delay.
A B C n X aj wj 0.516 0.616 0.44 j=1 n X aj uj 169.141 170.55 168.952
j=1 n The number of requests in the network M = X mj 0.42 1.27 0.43
0.39 1.31 1.28
The correctness of simulation results with using OQN was checked full-scale experiment. The stand consists of: { ANN realizes on FPGA by Lattice MachXo2-1200ZE. { Input-output data system for neural networks. Input data are pre-bu ered as FIFO and synchronously extracted to ANN inputs. There is a requests stream with a certain, adjustable intensity. The data trigger on the network output allows to measure the delay in data processing relative to the input clock signal. { The monitor which con gures input FIFO while system start. It checks validity (availability) of the output data and determines the temporal NN characteristics.
The experiment consisted of 1000 sequentially applied input impacts with xing the input signal installation time and observable system`s response time. ANN average time reaction data are presented in Table 5, and histogram of the distribution density of the ANN reaction for network topologies type A is presented in Fig. 3.A.
As presented in Table 5, ANN reaction time, which is obtained in experiment, coincides (up to 5%) with average time of staying request in the network, calculated for OQN model. Calculation error is related to the fact that time limitations of hardware interfaces between individual network neurons were not taken into account in the OQN model. Thus, ANN simulation results based on OQN were con rmed. Furthermore, because of the information transfer between individual neurons it is necessary to include in the OQN model the delay. This delay can be implemented as separate service device which simulates operation of a separate communication interface. As a result changes in data transfer interfaces between individual neurons can lead to changes in the OQN model. Identi ed instability of calculated reaction time is due to the fact that functioning of ANN hardware unit in FPGA is not absolutely synchronous. As the tacts number for ANN output calculation is constant in this realization, instability as caused by clock signal jitter. It is assumed that the factor causing the ANN clock signal jitter is the primary source of clock pulses. Generator signal frequency measurements and distribution histogram presented in Fig. 3.B were made. The nature of the frequency values distribution of the clock generator ( Fig. 3.B) coincides with the nature of the ANN reaction time distribution (Fig. 3.A), that allows to x direct connection of the second fact with the rst. Therefore, developing ANNs based on FPGA, special attention should be paid to the stability parameters of the used clock signal generator.
Previously described experiment was also performed for neuron networks training process. The Table 6 shows that ANN training time coincides with the calculated values obtained in the OQN model simulation. Coincidence accuracy is 5%. Network topology type C provides the best reaction time result for simulation modeling too. Reaction density distribution graph that was created for OQN training process is presented in Fig. 4. It revealed that the nature of the distribution of the frequency values of the clock generator (Fig. 3.B) coincides with the nature of the distribution of the OQN reaction time (Fig. 4). The study concluded that applying OQN models for hardware neural networks simulation based on FPGA is expedient and using this mathematical apparatus it is possible to evaluate the temporal characteristics of hardware NN. The results of this method were con rmed by eld tests. It was revealed that one of the source of unstable operation of hardware NN is associated with a jitter synchrosignal. The correlation of the distribution characteristics of the frequency values of the clock generator with the distribution characteristics of the ANN reaction time was also proved experimentally. This experiment was carried out on ANN implemented on a single crystal and with constant environmental characteristics. In further studies it is planned to test this theory for multi-chip ANNs implementation.