The compliance of modern computing platforms with Moore's Law requires developers to use new architectural approaches during development. One of these is a variety of systems on a chip (SoC). Among the wide variety of SoCs for real-time systems, SoC FPGAs (Field-Programmable Gate Array) have become widely used [1]. SoC FPGAs are integrated circuits that combine the functionality of a microprocessor core (CPU) with the capabilities of a programmable logic gate array (FPGA) on a single chip. This is an important class of microelectronics characterized by high programming flexibility, high-performance computing, and efficient use of resources [2][3][4]. In real-time embedded systems, FPGAs play a key role due to their versatility and ability to solve tasks in signal processing, controllers, image processing, and more [5]. Even though FPGAs provide outstanding characteristics (Flexible programming, Integration, Performance, Scalability), using them in their pure form requires high costs and qualifications in the development of modern applications.
Accordingly, to reduce the complexity of development, all real-time embedded applications must interact with the embedded operating system [6]. This is because it is difficult to manually create an optimized system without an OS. After all, when tested, such a system will not work better than similar solutions that use an OS [7]. Also, this approach allows you to better focus on system-level optimization when developing applications based on SoC FPGA. Because the operating system will be maximally optimized for the architecture of the microprocessor. The hardware is often fully optimized for the FPGA architecture. At the same time, communication between the software and hardware parts will be carried out through optimized bridges [8].
An embedded OS is a specialized operating system for computer systems embedded in other systems. These OS are computer systems designed for a specific purpose, to increase functionality and reliability for a specific task [9].
An important difference between most embedded operating systems and desktop operating systems is that the application, including the operating system, is usually statically compiled into a single package. Speaking of the Linux kernel, it is worth noting that embedded OSes based on it have features that positively distinguish them from specialized OSes. Resource efficiency is achieved at the cost of losing some functionality or granularity that larger computer operating systems provide, including features that may not be used by the specialized systems on which they run. Depending on the multitasking method, this type of OS is often thought of as a real-time operating system, or RTOS. Embedded systems most often use real-time operating systems. QNX, FreeRTOS, and VxWorks are the most widely used embedded operating systems today [10][11][12].
Most embedded operating systems are proprietary and closed to modification. The field of embedded development is very complex and highly demanding for engineers. Open Source Software (OSS) allows the creation of task and platform-oriented OSes without the need to implement low-level layers from scratch. In the field of SoC FPGA and embedded OSes, two systems for assembly and construction are attracting the most attention Yocto Project + Poky [13] and Buildroot [14]. These tools are representatives of the FPGA supported and used by large companies (Intel, AMD). Using them allows the use of the entire FPGA library and the experience of professionals from open-source communities [15].
Accordingly, the work aims to compare the existing approaches to building an embedded OS using open-source tools using the example of the DE10-Nano board [16] with SoC FPGA Cyclone V. Since the process of synthesizing an embedded OS, except for some points, is vendorindependent and will be similar for other SoC FPGAs.
The paper is organized as follows. A comparison of different OS development approaches and their impact on the efficiency of the FPGA SoC system is presented in section 2. The step-by-step process of building an embedded OS with hardware-specific extensions for FPGA SoC is presented in section 3. The development of a user space system and conclusions are presented in sections 4 and 5 respectively.
As already mentioned above, the two most widely used tools for developing an embedded OS distribution are Yocto Project + Poky and Buildroot. A third alternative is development using custom scripting.
A Linux distribution is a set of software packages and the ways they interact. There are hundreds of Linux distributions available. Most of them are not designed for embedded systems, lack the flexibility needed to achieve target sizes and change functionality, and are not suitable for systems with limited resources.
The Yocto project is a Linux distribution factory that uses several other open-source projects [15]. Yocto Project, on the other hand, is a distribution creation tool in its own right. It allows you to create a Linux distribution designed for a specific system.
Unlike Yocto, BuildRoot uses more general technologies, which negatively affects its flexibility. BuildRoot is a set of make files and patch aggregation that simplifies and automates the process of creating a complete and bootable Linux environment for an embedded system while using cross-compilation to create multiple target platforms on a single Linux-based host machine. Buildroot can automatically assemble the necessary cross-compilation tools, create the root file system, compile the Linux kernel image, and generate the bootloader for the target system, or perform any independent combination of these actions. For example, an already installed crosscompilation tool can be used independently, and Buildroot only creates the root file system [17].
Buildroot is primarily designed for use in small or embedded systems based on a variety of computer and instruction set architectures (ISAs), including x86, ARM, MIPS, and PowerPC.
Unlike the options described above, the Full Custom approach allows for a multifunctional system that can be updated and modified on the fly and is much more flexible in the development and prototyping process [18]. When building the OS itself, we can make changes to the software package itself, which will be standard in it, and can choose the type of Linux wrapper depending on the needs of the project (Arch Linux, Debian, etc.) [19]. This option is also capable of providing access to external resources and packages for on-the-fly installation, which greatly simplifies the process of iterating in development. Full Custom, unlike the previous approaches, is not characterized by the convenience of initial development and requires the entire image generation process. We need to assemble all parts of the OS ourselves, starting with the kernel and ending with the file system configuration.
It is a more comprehensive option that gives us the ability to control every element of the resulting OS and assemble the software and tool packages needed to get the system up and running, and if necessary, easily adapt to updating these packages from a remote source. Another significant positive factor is the ability to control what exactly will be updated in the next iteration of the distribution.
However, it should be kept in mind that that approach's adoption has higher requirements for the engineer, as the process itself needs to be fully controlled.
Overall, both frameworks can produce the same resulting distribution: a root file system image for an embedded device.
Buildroot trying to be as simple to use as possible. Its main tool is made to be as compact and user-friendly as possible, so engineers can catch up with all the tools and start developing as fast as possible. Specific features are built by extensions, allowing to create products using standard tools that are included in Buildroot distribution. The OS image created by Buildroot is usually oriented to be minimalistic, making builds quick and allowing you to create a generic system rather than one tailored to a specific use case.
In contrast, Yocto is much more robust and supports a wide range of embedded systems. To make that possible, configuration is defined in scripts (called 'recipes') that declare what software to build and how to build it, while allowing to gather recipes into layers, which can be used are collections of recipes written and maintained by the development community or by the distribution author himself [20].
Full custom is based on the step-by-step creation of a distribution, with full control of each element. Initially, this is not a fully automated process that relies heavily on the knowledge of an engineer. However, this process can be easily adapted to automation and allows you to build a process that is necessary specifically for a particular task.
The result of Buildroot is an image of the root file system, nothing more. Every time you need to update the system, the whole product will be rebuilt from scratch to avoid discrepancies.
The result of Yocto is a "distribution". More precisely a fully done OS with specific tools, configuration and packages. It is worth mentioning that integration with external package providers so the system can be modified while booted, requires development and is not pre-built. Yocto supports development on the board by providing SDK, to avoid continuous rebuilds of distribution for each modification.
Builds created with Buildroot and Yocto are also different in configuration. Buildroot stores configuration in a main file that can be edited using the kernel's kconfig tool (e.g. xconfig or menuconfig). That file basically contains all the configuration for built OS, including user space part.
In the case of the Full Custom approach, the result is both the final image and all its components, which can be easily modified depending on the needs of the project.
As was mentioned above, the process of building a Linux-based distribution consists of a couple of steps (see Fig. 1).
The boot-loader is a crucial one because all the next parts of the OS are started by it and U-Boot handles all basic system configuration. We are using U-Boot because it is the most used and proven one, and so it usually is familiar to engineers while having an extensive knowledge base available. To start with, we can get publicly available source code and choose a specific build version, as shown in Fig. 2. Generally, the default configuration is enough to start with, but if we do need to customize something deeply, all configuration headers are open. For example, we can modify data and variables that are stored in the U-Boot environment or add new ones. When U-Boot starts running the kernel, it reads most of the configuration from the environment and runs scripts/code based on it.
For example, Fig. 3 demonstrates how we can anchor our OS to only a specific MAC address by modifying the header file.
Most of the settings for U-Boot are stored in those header files, and they are used in compilation, which means that specific releases are hard-linked to those header files. We can change some of those variables (e.g. we can set the ethernet address while U-Boot is starting). Compilation for U-Boot is straightforward (see Fig. 4) after the modification of headers is done.
Once compilation completes we can find out the U-Boot distribution by searching for file `uboot-with-spl.sfp`. We will use that binary file when all the OS will be compressed in one image, as the launching point for the whole system.
When building an OS, we need to go through several steps:
1. Get Preloader is a binary file provided by the manufacturer, in our case Intel/Altera. It performs some basic configuration before passing it to the bootloader. 2. Compile Bootloader -software that performs hardware initialization before passing it to the kernel for OS initialization. 3. Configure Kernel -The kernel is the heart of the OS and contains all the information about the hardware on the board. 4. Build RootFS -Create a root file system that contains the basic packages and dependencies. When developing embedded systems, one of the key steps is to select the configuration and configure the Linux kernel options. This process includes choosing a basic RootFS configuration and selecting the kernel that best suits our needs [21].
In our case, the basic configuration of RootFS is based on Arch Linux, and the kernel chosen is altera-opensource (as shown in Fig. 5). The reason for choosing the Altera-opensource kernel was the ability to program the FPGA at startup or directly from the operating system, which is supported only by this Altera kernel modification. Therefore, we will use this particular fork in the future, after compiling it. Configuring Linux kernel parameters is an important step in the process of developing embedded systems. For this purpose, special tools, such as menuconfig, as it specified in Fig. 6, are used to configure various parameters that determine the behavior and functionality of the kernel. Here are some basic steps to configure Linux kernel settings To do this, we need to enable the `Overlay filesystem support` and `Userspace-driven configuration filesystem` in the kernel during configuration (see Fig. 7).
In the future, we can change the configuration depending on our needs, but as an initial configuration, our version is sufficient.
After we`ve done with configuration we can build the image of said kernel using the steps from Fig. 8.
As a result of compilation, we will get a binary zImage file, which will be used as a main Kernel executable. That binary file can be replaced separately from all other elements of system, allowing engineers to freely modify only the needed part of the distribution.
The flexibility and extensibility of a system based on an embedded operating system are important aspects, as they determine the ability to adapt and expand the system's functionality in the future. Let's take a closer look at these aspects:
• System flexibility:
1. Configuration: Embedded operating systems typically have a flexible architecture that allows for customization of parameters and options to meet the requirements of a particular application. b. Modularity:
The use of a modular architecture makes it easy to add and remove functional modules, which makes the system more flexible in changing requirements and project development.
• System extensibility:
1. Support for new devices: Embedded operating systems have mechanisms to dynamically add and support new devices without the need to recompile or flash the kernel. b. Extensibility of functionality:
The system can easily add new features and services without significant changes to the source code or architecture.
• Means for expansion:
1. Dynamic libraries and modules: Using dynamic libraries and modules allows you to add new features without rebooting your system or recompiling your applications. b. Network services:
Implementation of network services that allow dynamic application loading and system configuration over the network.
The use of an embedded operating system (OS) such as Linux in FPGA system-on-chips (SoC FPGA) significantly increases the flexibility and extensibility of projects. Due to the configurability of Linux, developers can customize the system to meet specific project requirements, including the choice of supported devices and functionality. The modular architecture of Linux allows you to dynamically add and remove features through the kernel or load modules while the system is running, which allows you to quickly expand functionality without a complete system reboot. In addition, Linux supports FPGA software upgrades without the need to shut down the system, which makes it easy to adapt to changes and implement new features and capabilities without much effort.
In the case of Full Custom Linux, we have access to many external modules, which means we can quickly reconfigure the entire system or its specific parts.
Since the basic distribution includes a set of tools for working with FPGA, you can make changes to the firmware directly from the OS, as well as read the FPGA configuration and perform basic control over it. Similarly, we can read and write from the HPS-to-FPGA registers.
To increase the flexibility of the system and simplify the development process, a separate set of packages has been built in. Using these tools allows you to automate interaction, create interfaces, and reduce the time required to update the FPGA circuitry and structure. Writing FPGA general purpose registers (GPOs)
In addition to the above package, it is also worth adding the ability of the OS to read .rbf files when the kernel is running and update the FPGA part with a new version of the schematic. This set of functionality has a positive impact on the flexibility of the entire system, simplifies development, and allows you to implement dynamic processes while iterating on a project.
Access to external resources and repositories allows you to expand the system with the help of specific packages, both those provided by vendors and those published by the system developers. Using the embedded OS allows you to flexibly respond to changes in the environment, tasks, or development vector. It also makes it possible to implement more complex systems and combine them, expand their functionality, and provide access to external resources.
To achieve maximum flexibility in an embedded system, various tools and strategies can be used:
Modular architecture: Developing a system based on a modular architecture allows you to divide the system into small modules that can be independently developed, tested, and updated. This provides flexibility to make changes and extend functionality.
Use of configuration files: configuration files allow you to change the FPGA configuration, including changing logic, I/O, and other parameters without the need to completely recompile the code.
Use of IP cores: FPGA IP cores are off-the-shelf modules that can be used to implement various functional blocks. The use of IP cores allows you to quickly and efficiently integrate the desired functionality without the need for development.
Memory Management Unit (MMU): The MMU allows you to dynamically manage memory access, which provides flexibility in the use of memory resources and allows you to use memory efficiently for different tasks.
Use of virtual devices: Virtual devices provide an interface between the FPGA and external components or systems, allowing for easier integration with other systems.
The use of tools included in the OS package allows for faster and easier interaction with the FPGA. For example, the FPGA-status and FPGA-reset utilities allow you to read the status and delete an existing configuration, making the system flexible in management and debugging. Also, FPGA-writeConfig allows you to change the configuration, which allows you to quickly adapt the system to new requirements or work scenarios. In addition, the FPGA-readBridge and FPGA-writeBridge utilities extend the capabilities of interaction between the software and hardware environment, and FPGA-gpiRead and FPGA-gpoWrite provide the ability to interact with the system through general-purpose input register signals.
This package makes the system more flexible and adaptable to changes in requirements and use cases.
Similarly, access to external repositories allows you to make changes externally, using both an "over-the-air" update and a rolling release approach, which reduces the time it takes to deliver an update to the system. The support for configuring the FPGA from the FAT partition of the embedded OS allows us to have a stable startup configuration and, if necessary, return to it.
Since the system can update the configuration during operation, we can use a modular approach and targeted development of certain modules for functionality or to solve certain tasks facing the project.
Adding to this the ability to cross-communicate between FPGA and OS parts, we can freely operate memory and utilize configurable approaches based on data exchange between system elements. Taken together, the use of the tools described above allows you to build your strategies for expanding and maintaining the designed system.
Developing the user space in a Linux-based embedded operating system (OS) for an FPGA SoC involves creating software that runs directly on the FPGA SoC. This user space can include various components such as applications, services, drivers, and other software modules that provide the required system functionality [23]. Here are some key aspects of the third point:
1. Application development: Create applications that interact with the FPGA SoC through various interfaces such as UART, SPI, I2C, Ethernet, etc. These applications can perform data processing, control devices, and provide a user interface to interact with the system.
Create services that run in the background and provide certain functions or event processing. For example, services can manage network connections, collect and analyze data, and interact with other system components.
3. Driver development: Create drivers to interact with FPGA SoC hardware components such as peripherals, storage, sensors, etc. These drivers allow the user space to access and interact with hardware resources.
4. Integration with FPGA resources:
The use of FPGA resources to implement specific functions or data processing. This can include the design and integration of logic blocks, signal processing, hardware accelerators, and other elements into the system's user space.
Once the software is developed, it is important to test and debug it to ensure that it works correctly, to identify and eliminate errors, and to optimize system performance and reliability.
To create a user space, we need to configure and enable RootFS properly. Rootfs is a special instance of ramfs always present on Linux 2.6 systems. It is used as a location inside the Linux kernel, as a list with directories of the file system the place to search. RamFs is also a pointer to the points for starting the operating system, but it is focused more on working with RAM. Most systems simply mount another file system on top of it and ignore it, because the ramfs instance takes up very little space. Rootfs is a file system. In Linux, all file systems have a mount point, which is the directory where the mounted file system connects to the root file system. This is the difference between them, RamFs are the older ancestor of RootFs, but with a focus on placing a small amount of RAM.
The peculiarity of our approach is the use of symlinks (a virtual link to a specific directory that can be located in any other directory), which allows us to split the elements of the distribution while keeping it unified for virtual access.
One of the main differences in creating RootFS for Archlinux ARM is that we need to first create an SD card image, mount it, and then create our RootFS [24]. The build process can be summarized as a step-by-step script (see Fig. 9) Following the scenario above, we will get a functional file system that can be modified depending on the needs and configured to meet the project's requirements. Such file systems do not take up many resources and do not put a significant burden on the entire scheme, since they contain minimal components and basic libraries for their extension. However, if necessary, they can be converted into a user-level system with all its pros and cons.
As a result, we will obtain the generalized OS which consists of several parts that are interconnected with each other (see Fig. 10) That composition means that we can update the specific part of the Image and the existence of the FAT32 additional component allows us to hook the default configuration that will be used as a starter, or even a recovery point. Also, we can interact with additional data part from U-Boot, if required.
Interacting with FPGA SoC system resources from the user space level of the embedded operating system (OS) can be done using various tools and utilities, including table-based utilities such as FPGA-status, FPGA-reset, FPGA-writeConfig, FPGA-readBridge, FPGA-writeBridge, FPGA-gpiRead, FPGA-gpoWrite, and the devmem tool.
FPGA-reset: Allows you to clear the FPGA configuration to the basic level FPGA-writeConfig: This utility allows you to write the FPGA configuration from the user space level.
FPGA-readBridge and FPGA-writeBridge: These tools will allow you to interact with bridges and interfaces between FPGA and other system components from the user space level.
FPGA-gpiRead and FPGA-gpoWrite: These utilities allow you to read input signals and write FPGA output signals from the user space level.
devmem: Allows you to read and write data to the device memory from the user space level. This can be useful for accessing various FPGA SoC resources, such as configuration registers, control registers, and others.
In general, the tools already in place allow integration and interaction with various aspects of the FPGA SoC from the OS user space level, making them important tools for simplifying development and interaction with FPGA. In addition to the above capabilities, the user can also use the direct memory access (DMA) option to quickly and efficiently read and write data to the FPGA SoC device memory from the user space level. This enables efficient use of memory resources and increases system performance. In addition, using the embedded Linux-based operating system, the user can easily use packages from external repositories. This extends the development capabilities as it allows the use of off-the-shelf solutions and libraries, which greatly facilitates the development, support, and expansion of the functionality of embedded systems based on FPGA SoC with a Linux environment [25]. In general, the available tools allow you to build interaction through HPS-to-FPGA and direct memory access, which simplifies the development of systems that can process and exchange data with FPGA. Knowing the addresses of the registers or the bus communication protocol, we can directly read and write binary data to this register using most programming languages or scripting tools (shell, as an example).
There are many advantages to using libraries for development on the embedded Linux for FPGA SoC with access to external repositories. Firstly, it allows us to significantly reduce development time, as we can use ready-made solutions to interact with hardware and software resources such as GPIO, SPI, UART, etc. Libraries also contribute to code optimization, as they provide an efficient interface to interact with these resources without the need to write your own code from scratch.
Furthermore, the use of external resources allows you to access a wide range of libraries and modules that extend the system's functionality. For example, we can include libraries for networking, data processing, creating a graphical interface, and much more. This helps to make the project more flexible and adaptive to changes in requirements or new features. In addition, by knowing which libraries and modules are required for your project, it is possible to create a standard package that can be included in a new release of the embedded OS. This simplifies the deployment and support of the system, as the standard package contains all the necessary components for the new version to work on the circuit.
Prototyping directly on the FPGA significantly reduces development time and makes it easier to understand how a system might behave in real-world conditions. Instead of using simulations or emulation environments, we can work directly with the FPGA, which allows us to identify and fix bugs faster and test different system scenarios more efficiently.
Thanks to our architecture, we can directly obtain test versions of the software code and check and modify them dynamically. This ensures a fast development cycle as we can work on new features and modules efficiently. We can also update the FPGA circuit to suit our project without rebuilding the entire system using the tools described above. This allows us to respond to changes in requirements or identified issues and quickly make the necessary changes to the system hardware without stopping the software.
In the summary of the Linux kernel-based OS distribution for embedded systems based on FPGA SoC compared to Yocto and Buildroot, we concluded that using the Full Custom system with a set of built-in tools significantly speeds up development and simplifies prototyping. Based on this information, we built an OS distribution focused on rapid adaptation and simplification of project development on the FPGA SoC platform.
The resulting system has the necessary components built in to work on projects and the ability to dynamically add and remove libraries and modules. This approach greatly facilitates the process of code development and optimization. In addition, the availability of utilities that allow direct interaction with the FPGA system allows you to get test versions of the software code, check and modify them dynamically, and update the FPGA circuit according to the project requirements. This ensures efficient and fast system development based on the embedded Linux OS for FPGA SoC.
The final system, although not significantly different in resource requirements from Yocto, is more adaptable and can be ported to the Yocto base if necessary. This allows us to make the most of the Full Custom approach, ensuring an optimal balance between development efficiency, responsiveness to changes, and system resource requirements.
Oleh Krulikovskyi thanks the Intel company for the equipment provided within the Intel FPGA Academic Program.