The softmax operation is used in the last layer of deep learning classifiers. This function is characterized by complex arithmetic operations as exponentials and divisions. In hardware implementations, such complexity negatively impacts on hardware resources. In this paper, we present an eficient hardware implementation of the softmax function. The proposed architecture has been designed and simulated in Simulink, coded in VHDL, and finally synthesized using the Xilinx Vivado toolchain. Implementation results are presented in terms of area and power.
In the last few years, we assisted to a significant
increase in Machine Learning based applications and
re1. The availability of a great amount of data thanks
to internet difusion.
tems that makes possible the eficient parallel
computations as for example GPUs and FPGAs.
3. The introduction of new technologies that allow
the implementation of artificial neural networks
more and more similar to natural ones [
New technologies can range from sensors [
In Machine learning systems, the softmax [
Vaccaro) © 2020 Copyright for this paper by its authors. Use permitted under Creative ∶ ℝ →
∈ ℝ | > 0, ∑ = 1 (z) = { ∑ =1 =1 = 1, … , .
}
As shown in eq. 2 the Softmax function replaces the activation function of the output layer of the neural network.
( = |x) = (
w x+ ) ∑ ∈ ( w x+ )
The block diagram of the softmax function is shown in Fig. 1 and works as follows: • The system accepts a k-dimensional vector "y"
as input; • The vector is input to a logic block that produces k results, the result of the exponential function of the input values. This result is input to two logic blocks; (1) (2) 2. Softmax hardware implementation • The “summation” block compresses a k-dimensional Because the exponential relationship between the vector into a numerical value equal to the sum number of the bit of the input and the location number of the k values; of the Look-Up table used for the exponential function computation, it is necessary to introduce a technique • The last block divides each element produced by able to reduce the location number of the Look-Up tathe exponential by the sum of the exponentials ble in order to not negatively impact on the hardware themselves, producing a k-dimensional vector resources used for the implementation. For this purcontaining the percentages. pose we use the linear interpolation.
Let’s consider the function ( ) = we define the function g as linear interpolation function in the interval 0 < < 0 + ℎ, with h defined as the interpolation step. The function g is defined in eq. 3.
The proposed architecture for the hardware softmax implementation is shown in Fig. 2. It is composed of three main blocks: • Look Up tables for the exponential computation
of e • An adders tree for the summation computation • Dividers for the division computation (, 0) = 0 + (, 0) = ( 0) + ⋅ ( ( 0 + ℎ) − ( 0))
− 0 − 0 = = 0 + ℎ − 0 ℎ 0
⋅ ( ℎ − 1) ⋅ ( − 0) ℎ
It is now possible to define the absolute error as shown in eq. 4 and the relative error shown in eq. 5
(3) (, 0) = (, 0) − ( ) = 0 ℎ = 0 +
⋅ ( ℎ − 1) ⋅ ( − 0) − (, 0) = 0 = 1 + ℎ − 1
ℎ = 1 − − 0 + ⋅ ( − 0) − ℎ − 1 ℎ 0 = ⋅ ( − 0)
Choosing h = 0.25 and = 0.125 it is possible to obtain maximum relative errors less than 0.8% with rms = 0.566% and less of 0.2% with rms = 0.14%.
In fig. 2 is shown a comparison between the exponential function and an interpolated exponential function.
In order to characterize the performance of the proposed system in terms of maximum frequency, hardexperiments on several implementations varying the ware resources, and power consumption, we performed device. range of the exponent and the size of the LUTs.
As regards the range, simulations were carried out with x in the range [-16; 15] and [-2; 1].
The device used for our experiments is the Zynq Ultrascale + ZCU 104, which allows you to manage the input to 8.
For x in the range [-15;15] we choose 5 bit for the integer part and 11 bit for the fractional part. The linear interpolation has been implemented with h=0.5. Using this approach the total amount of locations is equal (4) to 64. The minimum relative error in terms of rms is 2.22%. The values stored in the Look-up table are in the range [ −16 = 1.1254 ∗ 10−7; 15.5 = 5.3897 ∗ 10−6].
For x in the range [-2;1] we choose 2 bit for the integer part and 7 for the fractional part. The linear interpolation has been implemented with h=0.125. Using this approach the total amount of locations is equal to 8. The minimum relative error in terms of rms is 0.25%. (5) The values stored in the Look-up table are in the range [ −2 = 0.1353; 2 = 7.3891]. 3. Experimental results After a floating point and a fixed point Simulink simulation, the proposed softmax architectures were coded in VHDL and implemented using the XILINX Vivado toolchain. Finally, they have been characterized in terms of hardware resources speed and power consumption.
For our experiments we use a Xilinx Zynq Ultrascale+
Implementation results are shown in Fig. 3 and in Fig. 3 Implementations have been performed with a clock constraint of 20 ns. Experimental results show a very low use of hardware resources available in the device For what concerns power consumption, the results show a reduced dissipation. This is aspect is very power is not provided directly by the power line but it is provided by batteries or energy harvesting systems.
This is the case for example of IoT systems. Results show a power consumption of 0.120w and 0.107w for the two analyzed cases. In our experiment we fix the value of the number of numerous inputs and outputs of 16-bit architectures. important especially in all those applications where
In this paper, we presented the main features and potential of Xilinx RFSoCs. Thanks to RF converters (until 6 Gsps) they are very suitable for direct sampling of RF signals with bandwidth up to 4 GHz. As well as being fully supported from Vivado tool, these devices are also supported by other tools such us Mathworks Simulink.