The virtualization technology is attractive for modern embedded systems in that it can ideally implement resource partitioning but also can provide transparent software development environments. Although hardware emulation overheads for virtualization have been reduced signi cantly, the network I/O performance in virtual machine is still not satisfactory. It is very critical to minimize the virtualization overheads especially in real-time embedded systems, because the overheads can change the timing behavior of real-time applications. To resolve this issue, we aim to design and implement the device driver of RTEMS for the standardized virtual network device called virtio. Our virtio device driver can be portable across di erent Virtual Machine Monitors (VMMs) because our implementation is compliant with the standard. The measurement results clearly show that our virtio can achieve comparable performance to the virtio implemented in Linux while reducing memory consumption for network bu ers.
The virtualization technology provides multiple virtual
machines on a single device, each of which can run own
operating system and applications over emulated hardware in an
isolated manner [
A drawback of virtualization, however, is the overhead for
hardware emulation, which causes higher software execution
time. Although the emulation performance of instruction
sets has been signi cantly improved, the network I/O
performance in virtual machine is still far from the ideal
performance [
In order to improve the network I/O performance, usually a paravirtualized abstraction layer is exposed to the device driver running in the virtual machine. Then the device driver explicitly uses this abstraction layer instead of accessing the original I/O space. This sacri ces the transparency of whether the software knows it runs on a real machine or a virtual machine, but can improve the network I/O performance avoiding hardware emulation. It is desirable to use the standardized abstraction layer to guarantee portability and reliability; otherwise, we would have to modify or newly implement the device driver for di erent Virtual Machine Monitors (VMMs) and have to manage di erent versions of device driver.
In this paper, we aim to design and implement the
virtio driver for RTEMS [
The rest of the paper is organized as follows: In Section 2, we give an overview of virtualization and virtio. We also discuss related work in this section. In Section 3, we describe our design and implementation of virtio for RTEMS. The performance evaluation is done in Section 4. Finally, we conclude this paper in Section 5.
In this section, we give an overview of virtualization and describe virtio, the virtualization standard for I/O devices. In addition, we discuss the state-of-the-art for network I/O virtualization.
The software that creates and runs the virtual machines is
called VMM or hypervisor. The virtualization technology is
generally classi ed into full-virtualization and
paravirtualization. The full-virtualization allows legacy operating
system to run in virtual machine without any modi cations.
To do this, VMMs of full-virtualization usually perform
binary translation and emulate every detail of physical
hardware platforms. KVM and VirtualBox are examples of
fullvirtualization VMMs. On the other hand, VMMs of
paravirtualization provide guest operating systems with
programming interfaces, which are similar to the interfaces provided
by hardware platforms but much simpler and lighter. Thus,
the paravirtualization requires modi cations of guest
operating systems and can present better performance than
fullvirtualization. Xen [
There has been signi cant research on network I/O
virtualization. The most of existing investigations are, however,
focusing on the performance optimization for general-purpose
operating systems [
In this section, we suggest the design of virtio front-end driver for RTEMS. Our design can e ciently handle hardware events generated by the back-end driver and mitigate memory consumption for network bu ers. We have implemented the suggested design on the experimental system that runs RTEMS (version 4.10.2) over the KVM hypervisor as described in Section 4.1, but it is general enough to apply to other system setups.
The virtio network device is implemented as a PCI device. Thus, the front-end driver obtains the information of the virtual network device through PCI con guration space. Once the registers of the virtio device are found in the con guration space, the driver can access the I/O memory of the virtio device by using the Base Address Register (BAR). The virtio header, which has the layout shown in Fig. 2, locates in that I/O memory region and is used for initialization. Our front-end driver initializes the virtio device through the virtio header as speci ed in the standard. For example, the driver decides the size of the virtqueues by reading the value in Queue Size region. Then the driver allocates the virtqueues in the guest memory area and lets the back-end driver know the base addresses of the virtqueues by writing these to the Queue Address region. Thus, both front-end and back-end drivers can directly access the virtqueues by means of memory referencing without expensive hardware emulation.
The front-end driver also initializes the function pointers of the general network driver layer of RTEMS with the actual network I/O functions implemented by the front-end driver. For example, the if_start pointer is initialized by the function that transmits a message through the virtio device. This function adds a send descriptor to the TX virtqueue and noti es it to the back-end driver. If the TX virtqueue is full, this function intermediately queues the descriptor to the interface queue described in Section 3.2.
The interrupt handler is responsible for hardware events. However, since the interrupt handler is expected to nish immediately relinquishing the CPU resources as soon as possible, the actual processing of hardware events usually takes place later. In general-purpose operating systems, such delayed event handling is performed by the bottom half that executes in interrupt context with a lower priority than the interrupt handler. In regard to network I/O, demultiplexing of incoming messages and handling of acknowledgment packets are the examples that the bottom half performs. However, RTOS usually do not implement a framework for bottom half; thus, we have to use a high-priority thread as a bottom half. The interrupt handler sends a signal to this thread to request the actual event handling, where there is a tradeo between signaling overhead and size of interrupt handler. If the bottom half thread handles every hardware event aiming for a small interrupt handler, the signaling overhead can increase in proportional to the number of interrupts. For example, it takes more than 70 s per interrupt in RTEMS for signaling and scheduling between a thread and an interrupt handler on our experimental system described in Section 4.1. On the other hand, if the interrupt handler takes care of most of events to reduce the signaling overhead, the system throughput can be degraded, because interrupt handlers usually disable interrupts during its execution.
In our design, the interrupt handler is only responsible for moving the send/receive descriptors between interface queues and virtqueues when the state of virtqueues changes. Fig. 3 shows the sequence of event handling, where the interface queues are provided by RTEMS and used to pass network messages between the device driver and upper-layer protocols. When a hardware interrupt is triggered by the backend driver, the interrupt handler rst checks if the TX virtqueue has available slots for more requests, and moves the send descriptor that is stored in the interface queue waiting for the TX virtqueue to become available (steps 1 and 2 in Fig. 3). Then the interrupt handler sees whether the RX virtqueue has used descriptors for incoming messages, and moves these to the interface queue (steps 3 and 4 in Fig. 3). Finally, the interrupt handler sends a signal to the bottom half thread (step 5 in Fig. 3) so that the actual processing for received messages can be processed later (step 6 in Fig. 3). It is noteworthy that the interrupt handler handles multiple descriptors at a time to reduce the number of signals. In addition, we suppress the interrupts with the aid from the back-end driver.
On the sender side, the network messages are intermediately bu ered in the kernel due to the TCP congestion and ow controls. As the TCP window moves, the bu ered messages are sent as many as the TCP window allows. Thus, a larger number of bu ered messages can easily ll the window size and can achieve higher bandwidth. On the receiver side, received messages are also bu ered in the kernel until the destination task becomes ready. A larger memory space to keep the received messages also can enhance the bandwidth, because it increases the advertised window size in ow control. Although a larger TCP bu er size is bene cial for network bandwidth, the operating system limits the total size of messages bu ered in the kernel to prevent the messages from exhausting memory resources. However, we have observed that the default TCP bu er size of 16 KByte in RTEMS is not su cient to fully utilize the bandwidth provided by Gigabit Ethernet. Therefore, in Section 4.2, we heuristically search the optimal size of the TCP bu er that promises high bandwidth without excessively wasting memory resources.
Moreover, we control the number of preallocated receive bu ers (i.e., mbuf ). The virtio front-end driver is supposed to preallocate a number of receive bu ers that matches the RX virtqueue, each of which occupies 2 KByte of memory. The descriptors of the preallocated bu ers are enqueued at the initialization phase so that the back-end driver can directly place incoming messages in those bu ers. This can improve the bandwidth by reducing the number of interrupts, but reserved memory areas can waste memory resources. Therefore, it is desirable to size the RX virtqueue based on the throughput of the front and back-end drivers. If the front-end driver can process more messages than the back-end driver, we do not need a large number of preallocated receive bu ers. However, we need a su cient number of bu ers if the front-end driver slower than the back-end driver. The back-end driver of KVM requires 256 preallocated receive bu ers, but we have discovered that 256 bu ers are excessively large for Gigabit Ethernet as discussed in Section 4.2.
Fig. 4 shows how we control the number of preallocated receive bu ers. The typical RX virtqueue has the used ring and available ring areas, which store descriptors for used and unused preallocated bu ers, respectively. Our front-end driver introduces the empty ring area to the RX virtqueue in order to limit the number of preallocated bu ers. At the initialization phase, we ll the descriptors with preallocated bu ers until desc_head_idx reaches to the threshold de ned as sizeof (virtqueue) sizeof (empty ring). Then, whenever the interrupt handler is invoked, it enqueues the descriptors of new preallocated bu ers as many as vring_used.idx used_cons_idx (i.e., size of used ring). The descriptors in the used ring are retrieved by the interrupt handler as mentioned in Section 3.2. Thus, the size of empty ring is constant. We show the detail analysis of tradeo between the RX virtqueue size and network bandwidth in Section 4.2.
In this section, we analyze the performance of our design suggested in the previous section. First, we measure the impact of network bu er size on network bandwidth. Then, we compare the bandwidth and latency of our implementation with those of Linux.
We implemented the suggested design in RTEMS version 4.10.2. The mbuf space was con gured in huge mode so that it is capable of preallocating 256 mbufs for the RX virtqueue. We measured the performance of the virtio on two nodes that were equipped with Intel i5 and i3 processors, respectively, as shown in Figure 5. The Linux version 3.13.0 was installed to the former, and we installed the Linux version 3.16.6 on the other node. The two nodes are connected directly through Gigabit Ethernet.
We ran the ttcp benchmarking tool to measure the network
bandwidth with 1448 Byte messages, which is the maximum
user message size that can t into one TCP segment over
Ethernet. We separately measured the send bandwidth and
receive bandwidth of virtio by running a virtual machine
only on the i5 node with KVM. We reported the bandwidth
on the i3 node that ran Linux without virtualization. The
intention behind such experimental setup is to measure the
performance with the real timer because the virtual timer
in virtual machines is not accurate [
As mentioned in Section 3.3, we analyzed the impact of network bu er size on bandwidth. Fig. 6 shows the variation of send bandwidth with di erent sizes of the TCP bu er. We can observe that the bandwidth increases as the kernel bu er size increases. However, the bandwidth does not increase anymore after 68 KByte of TCP bu er size, because 68 KByte is su cient to ll the network pipe of Gigabit Ethernet. Fig. 7 shows the experimental results for receive bandwidth, which also suggests 68 KByte as the minimum bu er size for the maximum receive bandwidth. Thus, we increased the TCP bu er size from 16 KByte to 68 KByte for our virtio.
We also measured the network bandwidth varying the size of the RX virtqueue as shown in Fig. 8. This gure shows that the network bandwidth increases until the virtqueue size becomes only 8. Thus, we do not need more than 8 preallocated bu ers for Gigabit Ethernet though the default virtqueue size is 256.
In summary, we increased the in-kernel send and receive TCP bu er sizes from 16 KByte to 68 KByte, which requires additional memory resources of 104 KByte (= (68 16) 2) for higher network bandwidth. However, we reduced the number of preallocated receive bu ers from 256 to 8 without sacri cing the network bandwidth, which saved 496 KByte (= 256 2 8 2) of memory, where the size of mbuf is 2 KByte as mentioned in Section 3.3. Thus, we saved 392 KByte (= 496 104) in total while achieving the maximum available bandwidth over Gigabit Ethernet.
We compared the performance of RTEMS-virtio with that of Linux-virtio to see if our virtio can achieve comparable performance to the optimized one for general-purpose operating system. As shown in Fig. 9, the unidirectional bandwidth of RTEMS-virtio is almost the same with that of Linuxvirtio, which is near the maximum bandwidth the physical hardware can provide in one direction. Thus, these results show that our implementation can provide quite reasonable performance with respect to bandwidth.
We also measured the round-trip latency in a way that two nodes send and receive the same size message in a pingpong manner repeatedly for a given number of iterations. We reported the average, maximum, and minimum latencies for 10,000 iterations. Fig. 10 shows the measurement results for 1 Byte and 1448 Byte messages. As we can see in the gure, the average and minimum latencies of RTEMSvirtio are comparable to those of Linux-virtio. However, the maximum latency of RTEMS-virtio is signi cantly smaller than that of Linux-virtio (the y-axis is log scale) meaning RTEMS-virtio has a lower jitter. We always observed the maximum latency in the rst iteration of every measurement for both RTEMS and Linux. Thus, we presume that the lower maximum latency of RTMES is due to its smaller working set.
In this paper, we have suggested the design of the virtio front-end driver for RTEMS. The suggested device driver can be portable across di erent Virtual Machine Monitors (VMMs) because our implementation is compliant with the virtio standard. The suggested design can e ciently handle hardware events generated by the back-end driver and can reduce memory consumption for network bu ers, while achieving the maximum available network bandwidth over Gigabit Ethernet. The measurement results have showed that our implementation can save 392 KByte of memory and can achieve comparable performance to the virtio implemented in Linux. Our implementation also has a smaller jitter of latency than Linux thanks to smaller working set of RTEMS. In conclusion, this study can provide insights into virtio from the viewpoint of the RTOS. Furthermore, the discussions in this paper can be extended to apply to other RTOS running in virtual machine to improve the network performance and portability.
As future work, we plan to measure the performance of our virtio on a di erent VMM, such as VirtualBox, to show that our implementation is portable across di erent VMMs. Then we will release the source code. In addition, we intend to extend our design for dynamic network bu er sizing and measure the performance on real-time Ethernet, such as AVB.
This research was partly supported by the National Space Lab Program (#2011-0020905) funded by the National Research Foundation (NRF), Korea, and the Education Program for Creative and Industrial Convergence (#N0000717) funded by the Ministry of Trade, Industry and Energy (MOTIE), Korea.