Research on memory hierarchies regarding the non-functional requirements in embedded multicore systems demands for a framework to support the prototyping and evaluation of new methods. In current multicore processors, accesses on shared resources by arbitrary tasks lead to interferences, which can result in timing violations of high-priority tasks. However, incorporating all potential interferences in the schedulability analysis leads to an enormous overestimation of the task execution times, and requires a full analysis of all tasks running on the system. Enhancements in the memory hierarchy can provide isolation to restrict potential interferences, thus improving the worst-case performance. To research on modifications in the memory hierarchy of a multicore processor, a prototyping and evaluation framework is required. This paper describes the design of such a framework, and outlines the individual parts and their interconnections.
The performance of multicore processors is strongly desired in various domains of embedded systems to satisfy the increasing demand for computational power. Complex algorithms and software systems, e. g. in autonomous driving, benefit from high-performance general-purpose shared-memory multicores. However, these processors do not meet the typical requirements on real-time and safety, and thus cannot be used without performance-degrading and laborious software mechanisms. Elaborate methods in such systems have been developed to further improve the average-case performance of the processor, for example the increasing depth of the memory hierarchy. These and the shared resources, like last-level caches, buses, and main memory, result in the ultimate challenge of calculating tight WCET bounds for the tasks in a time-critical system.
The crucial problem is the missing guaranteed freedom of interferences between tasks that run
on separate cores. Thus, an arbitrary low-priority task is able to influence the timing behaviour
of another, potentially high-priority task on a diferent core. This can happen through accesses
on shared resources, for example shared caches or the main memory [
The general objective of research on this topic is to facilitate predictable performance, with
minimal over-estimation of timing bounds, by reducing the sources of potential interferences on
shared resources. Existing software-based approaches, e. g. performance counter monitors [
Previous work focused mostly on fault tolerance of parallel systems, but the research always
involved shared-memory systems. Diferent systems have been used to evaluate the proposed
methods, from software-only approaches on typical desktop and server hardware, over the
Gem5 simulator, to FPGA prototypes. As a side efect of the conducted implementations and
evaluations, some experience with diverse platforms has been collected. The work on a
softwareonly fault tolerance mechanism [
This paper describes a prototyping and evaluation framework for embedded multicore systems, and outlines the assembly of the individual parts into a synthesizable design for both simulation and prototyping on an FPGA. The framework is based on ChipYard, which supports design and evaluation of full-system hardware, using the Rocket Chip generator and its in-order RISC-V CPUs. The main benefit of ChipYard is the configurability and customizability of the involved modules. The interconnects could also be replaced with a NoC to research on manycore systems, or a combination of both with shared-memory clusters connected through a NoC. Based on the proposed framework, the research on elements of the memory hierarchy will be facilitated to improve the applicability of multicore processors in embedded systems.
To approach the objective of calculating tight WCET bounds for time-sensitive tasks in sharedmemory multicore systems, the potential interferences on shared resources have to be identified and measured first. While such evaluations can be performed on existing hardware, potential new methods to prevent or restrict interferences require customisable hardware components.
A system that enables the modification and enhancement of individual elements in the memory hierarchy should fulfil the following requirements: • Customisable hardware to extend or modify elements of the memory hierarchy • Measurement of the overall performance and counting individual accesses on shared resources • Independence of CPU architectures • Scalable number of processor cores • Hardware cost estimation of extensions and customisations • Fast response on functional correctness of the implementation • Fast and approximate evaluation of the simulated model • Accurate full-system evaluation on an FPGA These requirements are satisfied by our proposed framework, for which the Chipyard project provides a promising foundation. It is the predestined choice, since it is built around the open RISC-V ecosystem, and allows to customize or replace individual elements of the memory hierarchy. It further supports simulation and FPGA synthesis based on the same and identical code.
The evaluation framework builds upon existing open-source projects that have been developed in recent years around the prevalent RISC-V architecture.
Chipyard [
The generated Verilog code can be further compiled with Verilator for a simulation of the overall system, or with FireSim, which additionally allows to simulate DDR3 main memory.
Rocket Chip Generator
Peripherals FPGA shells
Verilator / FireSim FPGA prototype
Simulation
Alternatively, individual FPGA shells wrap the code with a harness to connect the units of the SoC to I/O pins of a concrete FPGA, to build a prototype running on an FPGA evaluation board. Such prototype is able to communicate with the built-in peripheral devices like UART and JTAG, as well as the of-chip DDR memory.
The Rocket Chip Generator [
A common objective of research on memory hierarchies for real-time systems is to reduce interferences on shared resources. From this, the main elements of the system under evaluation are derived: All units that control access to shared resources, like the peripheral bus, or the L2 cache, are of interest, as well as the private L1 caches that are connected to the shared system bus. In Fig. 3, these elements are shown below the processor cores, which are not of special interest for interference analysis. All accesses to shared resources that originate in the cores have to pass through the L1 instruction or data caches, which can control the communication.
The prototyping flow from implementing a design of one or more specific parts of the memory hierarchy to code generation and simulation or evaluation is depicted in Fig. 4. Unit tests can Rocket Chip
Rocket Tile
Rocket Tile
Rocket Tile
Rocket Tile
Core L1I L1D
Core L1I L1D
Core L1I L1D
Core
L1I L1D System Bus
Periph. Bus L2 Bank
L2 Bank
Ctrl. Bus
Front Bus Memory Bus
Boot ROM, Intrs., . . .
DMA provide fast checks of the functional correctness of the implemented or modified mechanisms. After passing these tests, Verilog code is generated, which can be simulated with Verilator to test the design with a set of benchmarks. The simulation provides fast feedback on the behaviour of the system, to compare diferent potential implementations before running the full evaluation of the synthesised bitstream on the FPGA. The evaluation of the design on the FPGA provides accurate timing measurements of the individual tasks, and a trace log of accesses on shared resources. These results allow to quantify the improvements of the implemented memory hierarchy modifications, and enable the detection of timing violations or forbidden Core
Core L1I L1D
L1I L1D
System Bus L2 Cache
Periph. Bus interferences that should not occur.
The possibility to connect a debugger to the simulation, as well as to the system on the FPGA, facilitates the detection of implementation faults, and provides detailed insight into the behaviour of the system under specific circumstances when needed.
With the feedback loop between the design and the simulation, available computational capabilities can be leveraged to compare numerous diferent design variations, to select a few designs of interest for the full evaluation of the FPGA.
This paper described the design of a prototyping and evaluation framework to research on memory hierarchies, for getting closer to the overall objective of enabling high-performance multicore processors in embedded real-time systems. The framework is built upon existing open-source projects around the RISC-V architecture, connecting the diferent tools together. It integrates all the required steps to automatically generate the Verilog code, compile and run the simulation, to synthesise the bitstream and program the FPGA with it, and to run the evaluation.
The next step is to implement the basic tool chain for automatic unit tests, code generation, simulation, and synthesis. Afterwards, measurement facilities in the individual components of the memory hierarchy have to be added to evaluate the behaviour of the system under parallel workloads. Such workloads first have to be identified based on use-cases from diferent industries, and reconstructed by a set of diferent benchmarks.
Based upon the proposed framework, research on new approaches for controlling interferences on shared resources within shared-memory multicores can take of.
This work is partially supported by the CERCIRAS COST Action no. CA19135 funded by COST.