Convolutional Neural Networks are commonly employed in applications involving Computer Vision tasks like image/video classi cation/recognition/segmentation. The increasing focus of the community on this topic, has generated a wide scope of approaches that use di erent kernel shapes and techniques for executing convolutions with respect to the classic one, such as for example separable convolutions, deformable convolutions or deconvolutions ([4, 5]), frequently used in semantic segmentation tasks ([23, 13]). While it is common knowledge that FPGAs can be used to accelerate classic Convolutional layers in CNNs, there is limited literature about FPGA-based accelerators supporting less regular and common processing kernels ([20]). In our research, starting from the previous experience acquired developing NEURAghe, we plan to improve exibility of CNN accelerators and to study new methodologies to improve e ciency on the previously mentioned use-cases. As a rst experiment we focus on layered approaches based on 1D convolutions, that, as indicated by several recent research results, can be e ectively used to classify and segment time series and sequences, as well as in tasks involving sequence modeling. In multiple scenarios a convolution approach applied on the time dimension, hereafter called Temporal Convolution Network (TCN) can outperform classic strategies relying on recurrent networks in terms of accuracy and training time. We modi ed NEURAghe to support TCN and validate results on an ECG-classi cation benchmark, achieving up to 95% e ciency in terms of GOPS/s with respect to the accelerator peak performance.
Recent studies demonstrate the e ectiveness of CNNs, already extensively used for
computer vision applications, over tasks like audio synthesis ([
Nowadays also, a broad range of mono-dimensional CNNs are used for human
signals analysis tasks like ECG classi cation ([
The ubiquitous success of CNNs and their high demands in terms of computing power have motivated during past years a huge out ow of research aimed at developing hardware accelerators for CNN inference. Among other solution, one of the most adopted has been exploiting the cooperation between general purpose processors and FPGAs in modern SoCs, like Xilinx Zynq, providing an e cient implementation of MAC operations on the large amount of DSP Slices available.
Previous mentioned areas of application together with the di erent convolutional schemes suggest that these kind of hardware accelerators need to be as exible as possible, supporting multiple CNN's features.
Looking to this objective, and motivated by recent claims about the e
ectiveness of TCNs in di erent application domains, this work explores the capabilities
of a CNN inference accelerator, NEURAghe [
We report a performance analysis and propose an optimization method relying on batch processing to improve e ciency.
The TCN under test with its parameters variability over layers represents a good benchmark for the architecture's exibility capabilities. 2
Convolutional Neural Networks have become the state of the art solutions in elds
which concern computer vision tasks like image recognition [
They are also used in applications requiring a mono-dimensional elaboration
of data like sentence classi cation [
More recent applications involves machine translation [
On the other hand Recurrent Neural Network (RNN) are considered the go-to
solution for sequence modeling tasks [
Nevertheless, in a recent study, Bai et al. ([
Another type of application, recently targeted by TCNs, involves human signal
analysis like ECG classi cation ([
With respect to CNN applications for Computer Vision, during years, many
hardware solutions have been proposed in order to accelerate the inference task.
Among di erent solutions adopted, one common approach exploits modern SoCs
integrating both a general purpose processor and a programmable logic [
Other works proposed FPGA based accelerators for LSTM RNN, like [
To our knowledge there are no FPGA-based architectures that speci cally tackled the problem of hardware acceleration for sequences processed by Temporal Convolutional Neural Networks. 3
TCN model net’s receptive field input layer 1: kernel_size = 2
dilation = 1 layer 2: kernel_size = 3
dilation = 2 layer 3: kernel_size = 4
dilation = 3 output
The input of the network is a continuous sequence of samples from a set of source channels. While in some use-cases input processing can be executed o -line, multiple applications require continuous and near-real-time analysis of the input aimed at the identi cation of speci c events and/or at promptly taking decision on speci c actions. In this case, the TCN must analyze as soon as possible any new input sequence sampled by the system and update at every new sample time. At every time step the network processes a sliding window whose minimal size is known as receptive eld of the network. This is the smallest amount of samples needed to produce an output and depends on convolutional layer parameters such as the kernel size and the dilation: receptivef ield = 1 +
L X [kernel size(l) l=1 dilation(l) (1) where l 2 1; 2:::L is a layer of the network.
Figure 1 represents a computational step of a TCN with di erent layer parameters. Each input or output dot must be considered as a vector with its own dimension in terms of number of channels. The gure shows how many input samples are needed by each layer to perform a valuable convolution and how kernel size and dilation a ect the receptive eld. Moreover, especially with di erent dilation rates, the network can have longer memory without increasing too much the depth and the number of parameters to train. 4
The starting point for this NEURAghe architectural template has been the one
described in Meloni et al. [
The programmable logic hosts the Convolution Speci c Processor (CSP) while the processing system acts as a General Purpose Processor (GPP) dealing with tasks hardly to accelerate in the programmable logic, like fully connected layers execution. In this work, the CSP has been enhanced with respect to the previous
DDR 64 bit 32 bit
High-speed domain Low-speed domain PS – PL interface L2 BRAM Memory
WDMA0 rx load WDMA1 rx load WEIGHT MEMORY 0 1 2 3 … 45 46 47 t c e n n o c itr e n d e s a b – I X A
ArxDMtxA ad re lo tso 12 ports x_in … XBAR
4 ports y_in 4ports y_out XBAR
TCDM 0 1 2 3 … 16 18 19 uC
PRIVATE MEM version to improve the accelerator's exibility towards various networks characteristics. In particular there has been a substantial modi cation of the Convolution Engine, previously characterized by the so called Line Bu er and a complete different SoP module model. The former, in charge to supply convolutional windows to the SoP matrix, has been removed thanks to the new pixel fetching method, while the latter, composed by a double trellis of pipelined DSPs, has been completely redesign. Both changes let the architecture to be more exible. Finally the transfers capabilities are improved by doubling the Weight DMA. The Convolution Engine is the computational core of the accelerator. It is designed to execute a high number of Multiply and Accumulate (MAC) operations in parallel, to relief the host processor from the most computational-intensive tasks of a CNN. It is composed by a matrix of M columns by N rows of Sum of Product (SoP) units in charge to calculate the contribution of M input features to N output features. Partial result from SoPs in each row are summed together by means of N Shift Adder modules.
kernelsize dilation address kernel size address
SoP 0 SoP C SoP 18 SoP 24
SoP 1 SoP D SoP 19 SoP 25
SoP 2 SoP E SoP 1A SoP 26
Activation source SoP 3 SoP F SoP 1B SoP 27
SoP 4 SoP 10 SoP 1C SoP 28
SoP 5 SoP 11 SoP 1D SoP 29
SoP 6 SoP 12 SoP 1E SoP 2A
SoP 7 SoP 13 SoP 1F SoP 2B
SoP 8 SoP 14 SoP 20 SoP 2C
Weights source
In more detail, each SoP reads 4 samples=cycle and executes 4 M ACs=cycle, one for each of the 4 consecutive kernel windows applied to an IF. In this way a SoP module produces 4 new output samples after kernel size cycles. Every output sample produced can be sent to the Shift Adder module.
In this con guration the C.E. has a 12 4 SoP matrix, therefore, every clock cycle, 4 samples of 12 input features are fetched to SoPs from the C.E. internal memory (TCDM), by means of the Activation Source modules. Samples of every IF are sent to the 4 SoPs of a column. Whereas 12 4 weight kernels are independently fetched from Weight Memory banks by means of the Weights source module.
SoP modules are built using 4 Xilinx DSP48E1 primitives con gured to perform a MAC operation per cycle using the internal loop to iterate among consecutive partial result given by a weight kernel application. This design allows the accelerator: { to be kernel size agnostic, { to execute convolutions with multiple stride values without performance overhead.
As the Convolution Engine works on 16-bit sample data, every SoP reads 64-bit activation data per cycle and multiplies all four pixels for the same kernel weight. Thus each SoP needs to read only 16-bit weight data per cycle.
Shift Adder modules read 64-bit data that are 4 16-bit values resulting from the previous 12 IF contribution and produce 64-bit data output summing together these inputs and those given by the actual 12 IF contribution. 4.2
Scheduling As NEURAghe exploits a double bu ered memory policy, local memory banks are subdivided in two parts, one dedicated to the data needed for the actual computational step, while the other allows to perform data transfers that will be used for the next phase. This allows the accelerator to overlap transfers with computational phases aiming to reduce idles.
From the e ciency point of view, the best situation is when all transfers are overlapped by execution phases because it is exploited all the computational power of the accelerator. 4.3
TCN model on NEURAghe Temporal CNN characteristics allow NEURAghe to handle samples of di erent layers in a speci c way that slightly di ers from regular convolutions. As mentioned above, kernel size and dilation a ect the receptive eld of the network that de nes the number of input samples that must be processed to gain a new output. Furthermore, for every layer a speci c receptive eld can be considered, given by: Layer N • Kernel size = 2 • Dilation = 3 • Batch size = 1 Layer N+1 • Kernel size = 2 • Dilation = 2 • Batch size = 1
Receptive field N that is the minimum number of input samples per channel needed to obtain an output sample from a layer. Figure 4 shows the memory status in di erent moments for two layers of an example network. In this example, layers N and N + 1 are characterized by the same kernel size of 2 and dilation of 3 and 2 respectively. For the sake of simplicity, it is represented a single sample update instead of the 4 processed by the accelerator.
As it can be seen, for every computational step, it is necessary to retain in memory only the samples equal to the speci c receptive eld of each layer. As a consequence, every transfer concerns only this minimum amount of samples per channel. 4.4
Improving through batch processing The previously mentioned approach minimizes the classi cation/recognition latency. It produces outputs as soon as possible and repeats network execution every time that a new sample is available to update the input sliding window.
However, this can determine the performance of the system to be easily bandwidth limited. All the network parameters/weights have to be loaded on the accelerator local memory and are used only for the production of one single output. This decreases signi cantly the operational intensity of the application.
As an example we show a roo ine model of our system in Figure 5. The leftmost Batch size = 348 300 100 s / SP10 O G
Batch size = 1 1 3 30
OPS/byte red symbol indicates the performance achieved when using the sample-by-sample processing on the use-case that will be presented in the following.
As it may be noticed, in this case, the system performance are de nitely limited by input/output bandwidth.
If the application allows to trade-o some increase in latency to improve performance, a solution to this problem may rely on batch processing. We can pre-bu er input samples and process longer sample sequences producing more outputs with Batch size = 8
C.S.P. Roofline Model
Peak Convolutional Engine Performance (30,72 GOPS/s)
Layer N • Kernel size = 2 • Dilation = 3 • Batch size = B Layer N+1 • Kernel size = 2 • Dilation = 2 • Batch size = B
New Samples = B Receptive field N Receptive field N+1
DMAs
CE TCDM every TCN execution. Figure 6 shows the transfer scheduling when batch size is increased.
Increasing the batch size, with the same weight transfers from DDR to local memory, we perform more computations. In this way, it is possible to increase the operational complexity and to gain e ciency, see other red symbols in Figure 5. 4.5
Resources utilization on target board The NEURAghe architecture is scalable and can be implemented in di erent devices like those which are part to the Xilinx Zynq-7000 SoC family. In particular, the con guration described above is highly suitable for boards, like the Zedboard, integrating a Xilinx Zynq Z-7020. Table 1 shows the resource occupation of the recon gurable logic of the device.
It is worth noticing that the architecture uses 192 out of the 220 DSP blocks available in the device, so the processing power utilization is very high. Also the Block RAM primitives are extensively used due to the particular Weights Memory implementation. The accelerator on Zedboard is clocked at 80 MHz. 5
Use case network: ECG Classi cation
The use case that we have chosen as benchmark is a network for ECG
monitoring and classi cation (Goodfellow et al. [
In Table 2 are shown the characteristics of convolutional layers of the network along with minimum input size required. The latency constrained con guration provides best performance from the point of view of total elaboration time for a new output value of the network. On the other hand this con guration lacks in terms of e ciency with respect to peak GOP S=s reachable by NEURAghe ( rst column of Table 3).
Low e ciency for this con guration is due to the huge weight transfer overhead with respect to the very short activation load time and execution time. So, despite the double bu ered scheduling strategy, transfer and computation phases hardly overlap.
If the target application has no latency constraint it is possible to work with much more activation samples as input for every layer. In particular when the execution time reaches and surpasses transfer times the architecture can get performances very close to the peak, as it is shown in third column of Table 3.
As a third case it can be considered that for which the latency constraint is not as tightening as for the rst case and it is possible to increase the e ciency without compromise performance from the point of view of the execution time.
In particular it is possible to do interesting assumption about a real time operating mode by referring to the sample rate mentioned in the paper, that is 300 Hz.
Second column of table 3 also shows that by bu ering a small amount of samples, that is increasing the batch size parameter by 8 samples, before feeding the accelerator, it is possible to substantially gain in e ciency without loss in performance with respect to the latency constrained con guration. Moreover, having an initial bu ering of 8 samples means that the accelerator has a new batch every 26:67 ms which is enough to complete an end-to-end computation for this network. 6
Conclusions and future work In this work has been presented an application towards Temporal Convolutional Networks of NEURAghe, an FPGA-based hardware accelerator enhanced to be kernel and dilation rate agnostic and also to process inputs with multiple stride values, without overhead. Motivated by recent claim about TCNs implementation over various applications it has been made an explorations of the architecture's performances with respect to convolutional layers of an use case TCN. Results showed that the architecture has a good exibility over various layer characteristics. It has also been showed how performances improve by changing the computational paradigm, going from a latency constrained approach to a batched approach, by agreeing with a certain latency.
The next step can be exploring di erent architectural con gurations suitable both for di erent target devices and di erent TCNs in order to nd the solution that adapt best.