=Paper=
{{Paper
|id=Vol-1464/ewili15_13
|storemode=property
|title=Supporting Virtualization Standard for Network Devices in RTEMS Real-Time Operating System
|pdfUrl=https://ceur-ws.org/Vol-1464/ewili15_13.pdf
|volume=Vol-1464
|dblpUrl=https://dblp.org/rec/conf/ewili/KimLJ15
}}
==Supporting Virtualization Standard for Network Devices in RTEMS Real-Time Operating System==
https://ceur-ws.org/Vol-1464/ewili15_13.pdf
Supporting Virtualization Standard for Network Devices in
RTEMS Real-Time Operating System
Jin-Hyun Kim Sang-Hun Lee Hyun-Wook Jin
Department of Smart ICT Department of Computer Department of Computer
Convergence Science and Engineering Science and Engineering
Konkuk University Konkuk University Konkuk University
Seoul 143-701, Korea Seoul 143-701, Korea Seoul 143-701, Korea
jinhyun@konkuk.ac.kr shunlee@konkuk.ac.kr jinh@konkuk.ac.kr
ABSTRACT services with high system utilization and low power con-
The virtualization technology is attractive for modern em- sumption. As modern complex embedded systems are also
bedded systems in that it can ideally implement resource facing the size, weight, and power (SWaP) issues, researchers
partitioning but also can provide transparent software devel- are trying to utilize the virtualization technology for tem-
opment environments. Although hardware emulation over- poral and spatial partitioning [5, 21, 12]. In the partitioned
heads for virtualization have been reduced significantly, the systems, a partition provides an isolated run-time environ-
network I/O performance in virtual machine is still not sat- ment with respect to processor and memory resources; thus,
isfactory. It is very critical to minimize the virtualization virtual machines can be exploited to efficiently implement
overheads especially in real-time embedded systems, because partitions. Moreover, the virtualization can provide a trans-
the overheads can change the timing behavior of real-time parent and efficient development environment for embedded
applications. To resolve this issue, we aim to design and software [10]. For example, if the number of target hard-
implement the device driver of RTEMS for the standardized ware platforms is smaller than that of software developers,
virtual network device called virtio. Our virtio device driver they can work with virtual machines that emulate the target
can be portable across different Virtual Machine Monitors hardware system.
(VMMs) because our implementation is compliant with the
standard. The measurement results clearly show that our A drawback of virtualization, however, is the overhead for
virtio can achieve comparable performance to the virtio im- hardware emulation, which causes higher software execution
plemented in Linux while reducing memory consumption for time. Although the emulation performance of instruction
network buffers. sets has been significantly improved, the network I/O per-
formance in virtual machine is still far from the ideal perfor-
Categories and Subject Descriptors mance [13]. It is very critical to minimize the virtualization
D.4.7 [Operating Systems]: Organization and Design— overheads especially in real-time embedded systems, because
real-time systems and embedded systems the overheads can increase the worst-case execution time and
jitters, thus changing the timing behavior of real-time ap-
plications. Few approaches to improve the performance of
General Terms network I/O virtualization in the context of embedded sys-
Design, Performance tems have been suggested, but these are either proprietary
or hardware-dependent [7, 6].
Keywords
Network virtualization, RTEMS, Real-time operating sys- In order to improve the network I/O performance, usually a
tem, virtio, Virtualization paravirtualized abstraction layer is exposed to the device
driver running in the virtual machine. Then the device
1. INTRODUCTION driver explicitly uses this abstraction layer instead of access-
The virtualization technology provides multiple virtual ma- ing the original I/O space. This sacrifices the transparency
chines on a single device, each of which can run own operat- of whether the software knows it runs on a real machine or
ing system and applications over emulated hardware in an a virtual machine, but can improve the network I/O per-
isolated manner [19]. The virtualization has been applied to formance avoiding hardware emulation. It is desirable to
large-scale server systems to securely consolidate different use the standardized abstraction layer to guarantee porta-
bility and reliability; otherwise, we would have to modify
or newly implement the device driver for different Virtual
Machine Monitors (VMMs) and have to manage different
versions of device driver.
In this paper, we aim to design and implement the vir-
tio driver for RTEMS [2], a Real-Time Operating System
(RTOS) used in spacecrafts and satellites. virtio [17] is the
EWiLi’15, October 8th, 2015, Amsterdam, The Netherlands. standardized abstraction layer for paravirtualized I/O de-
Copyright retained by the authors.
vices and is supported by several well-known VMMs, such as
KVM [8] and VirtualBox [1]. To the best of our knowledge,
this is the first literature that presents detail design issues
of the virtio front-end driver for RTOS. Thus, our study
can provide insight into design choices of virtio for RTOS.
The measurement results clearly show that our virtio can
achieve comparable performance to the virtio implemented
in Linux. We also demonstrate that our implementation can
reduce memory consumption without sacrificing the network
bandwidth.
The rest of the paper is organized as follows: In Section
2, we give an overview of virtualization and virtio. We also
discuss related work in this section. In Section 3, we describe
our design and implementation of virtio for RTEMS. The
performance evaluation is done in Section 4. Finally, we
conclude this paper in Section 5.
2. BACKGROUND
In this section, we give an overview of virtualization and
describe virtio, the virtualization standard for I/O devices.
In addition, we discuss the state-of-the-art for network I/O Figure 1: virtio.
virtualization.
suitable for embedded systems, because their network con-
2.1 Overview of Virtualization and virtio trollers are not equipped with sufficient hardware resources
The software that creates and runs the virtual machines is
to implement multiple virtual network devices. Though
called VMM or hypervisor. The virtualization technology is
there is an architectural research on efficient network I/O
generally classified into full-virtualization and paravirtual-
virtualization in the context of embedded systems [6], it
ization. The full-virtualization allows legacy operating sys-
also highly depends on the assist from network controller.
tem to run in virtual machine without any modifications.
The software-based approach for embedded system has been
To do this, VMMs of full-virtualization usually perform bi-
studied in a very limited scope that does not consider the
nary translation and emulate every detail of physical hard-
standardized interfaces for network I/O virtualization [7].
ware platforms. KVM and VirtualBox are examples of full-
virtualization VMMs. On the other hand, VMMs of paravir-
Compared to existing research, virtio can be differentiated
tualization provide guest operating systems with program-
in that it does not require hardware support and can be
ming interfaces, which are similar to the interfaces provided
more portable [17, 18]. The studies for virtio have mainly
by hardware platforms but much simpler and lighter. Thus,
dealt with the back-end driver [11, 14]. However, there are
the paravirtualization requires modifications of guest oper-
several additional issues for the front-end driver on RTOS
ating systems and can present better performance than full-
due to inherent structural characteristics of RTOS and the
virtualization. Xen [3] and XtratuM [12] are examples of
resource constraint of embedded systems. In this paper, we
paravirtualization VMMs.
focus on the design and implementation issues of the virtio
front-end driver for RTOS.
virtio is the standard for virtual I/O devices. It was initially
suggested by IBM [17] and recently became an OASIS stan-
dard [18]. The virtio standard defines paravirtualized in- 3. VIRTIO FOR RTEMS
terfaces between front-end and back-end drivers as shown in In this section, we suggest the design of virtio front-end
Fig. 1. The paravirtualized interfaces include two virtqueues driver for RTEMS. Our design can efficiently handle hard-
to store send and receive descriptors. Because virtqueues are ware events generated by the back-end driver and mitigate
located in a shared memory between front-end and back-end memory consumption for network buffers. We have imple-
drivers, the guest operating system and VMM can directly mented the suggested design on the experimental system
communicate each other without hardware emulation. Many that runs RTEMS (version 4.10.2) over the KVM hypervi-
VMMs, such as KVM, VirtualBox, and XtratuM, support sor as described in Section 4.1, but it is general enough to
virtio or its modification. General-purpose operating sys- apply to other system setups.
tems, such as Linux and Windows, implement the virtio
front-end driver.
3.1 Initialization
The virtio network device is implemented as a PCI device.
2.2 Related Work Thus, the front-end driver obtains the information of the vir-
There has been significant research on network I/O virtual- tual network device through PCI configuration space. Once
ization. The most of existing investigations are, however, fo- the registers of the virtio device are found in the configu-
cusing on the performance optimization for general-purpose ration space, the driver can access the I/O memory of the
operating systems [20, 15, 22, 16, 4]. Especially, the ap- virtio device by using the Base Address Register (BAR). The
proaches that require supports from network devices are not virtio header, which has the layout shown in Fig. 2, locates
Figure 2: virtio header in I/O memory. Figure 3: Hardware event handling.
described in Section 4.1. On the other hand, if the interrupt
in that I/O memory region and is used for initialization.
handler takes care of most of events to reduce the signaling
overhead, the system throughput can be degraded, because
Our front-end driver initializes the virtio device through the
interrupt handlers usually disable interrupts during its exe-
virtio header as specified in the standard. For example,
cution.
the driver decides the size of the virtqueues by reading the
value in Queue Size region. Then the driver allocates the
In our design, the interrupt handler is only responsible for
virtqueues in the guest memory area and lets the back-end
moving the send/receive descriptors between interface queues
driver know the base addresses of the virtqueues by writing
and virtqueues when the state of virtqueues changes. Fig. 3
these to the Queue Address region. Thus, both front-end
shows the sequence of event handling, where the interface
and back-end drivers can directly access the virtqueues by
queues are provided by RTEMS and used to pass network
means of memory referencing without expensive hardware
messages between the device driver and upper-layer proto-
emulation.
cols. When a hardware interrupt is triggered by the back-
end driver, the interrupt handler first checks if the TX virtqu-
The front-end driver also initializes the function pointers of
eue has available slots for more requests, and moves the
the general network driver layer of RTEMS with the actual
send descriptor that is stored in the interface queue waiting
network I/O functions implemented by the front-end driver.
for the TX virtqueue to become available (steps 1 and 2 in
For example, the if_start pointer is initialized by the func-
Fig. 3). Then the interrupt handler sees whether the RX
tion that transmits a message through the virtio device.
virtqueue has used descriptors for incoming messages, and
This function adds a send descriptor to the TX virtqueue
moves these to the interface queue (steps 3 and 4 in Fig. 3).
and notifies it to the back-end driver. If the TX virtqueue
Finally, the interrupt handler sends a signal to the bottom
is full, this function intermediately queues the descriptor to
half thread (step 5 in Fig. 3) so that the actual processing for
the interface queue described in Section 3.2.
received messages can be processed later (step 6 in Fig. 3).
It is noteworthy that the interrupt handler handles multiple
3.2 Event Handling descriptors at a time to reduce the number of signals. In
The interrupt handler is responsible for hardware events. addition, we suppress the interrupts with the aid from the
However, since the interrupt handler is expected to finish back-end driver.
immediately relinquishing the CPU resources as soon as pos-
sible, the actual processing of hardware events usually takes 3.3 Network Buffer Allocation
place later. In general-purpose operating systems, such de- On the sender side, the network messages are intermedi-
layed event handling is performed by the bottom half that ately buffered in the kernel due to the TCP congestion and
executes in interrupt context with a lower priority than the flow controls. As the TCP window moves, the buffered mes-
interrupt handler. In regard to network I/O, demultiplex- sages are sent as many as the TCP window allows. Thus, a
ing of incoming messages and handling of acknowledgment larger number of buffered messages can easily fill the win-
packets are the examples that the bottom half performs. dow size and can achieve higher bandwidth. On the receiver
However, RTOS usually do not implement a framework for side, received messages are also buffered in the kernel un-
bottom half; thus, we have to use a high-priority thread as til the destination task becomes ready. A larger memory
a bottom half. The interrupt handler sends a signal to this space to keep the received messages also can enhance the
thread to request the actual event handling, where there bandwidth, because it increases the advertised window size
is a tradeoff between signaling overhead and size of inter- in flow control. Although a larger TCP buffer size is benefi-
rupt handler. If the bottom half thread handles every hard- cial for network bandwidth, the operating system limits the
ware event aiming for a small interrupt handler, the signal- total size of messages buffered in the kernel to prevent the
ing overhead can increase in proportional to the number of messages from exhausting memory resources. However, we
interrupts. For example, it takes more than 70 µs per in- have observed that the default TCP buffer size of 16 KByte
terrupt in RTEMS for signaling and scheduling between a in RTEMS is not sufficient to fully utilize the bandwidth
thread and an interrupt handler on our experimental system provided by Gigabit Ethernet. Therefore, in Section 4.2,
Figure 5: Experimental System Setup.
4.1 Experimental System Setup
Figure 4: Controlling the number of preallocated We implemented the suggested design in RTEMS version
receive buffers. 4.10.2. The mbuf space was configured in huge mode so
that it is capable of preallocating 256 mbufs for the RX
virtqueue. We measured the performance of the virtio on
we heuristically search the optimal size of the TCP buffer
two nodes that were equipped with Intel i5 and i3 proces-
that promises high bandwidth without excessively wasting
sors, respectively, as shown in Figure 5. The Linux version
memory resources.
3.13.0 was installed to the former, and we installed the Linux
version 3.16.6 on the other node. The two nodes are con-
Moreover, we control the number of preallocated receive
nected directly through Gigabit Ethernet.
buffers (i.e., mbuf ). The virtio front-end driver is supposed
to preallocate a number of receive buffers that matches the
We ran the ttcp benchmarking tool to measure the network
RX virtqueue, each of which occupies 2 KByte of memory.
bandwidth with 1448 Byte messages, which is the maximum
The descriptors of the preallocated buffers are enqueued at
user message size that can fit into one TCP segment over
the initialization phase so that the back-end driver can di-
Ethernet. We separately measured the send bandwidth and
rectly place incoming messages in those buffers. This can
receive bandwidth of virtio by running a virtual machine
improve the bandwidth by reducing the number of inter-
only on the i5 node with KVM. We reported the bandwidth
rupts, but reserved memory areas can waste memory re-
on the i3 node that ran Linux without virtualization. The
sources. Therefore, it is desirable to size the RX virtqueue
intention behind such experimental setup is to measure the
based on the throughput of the front and back-end drivers.
performance with the real timer because the virtual timer
If the front-end driver can process more messages than the
in virtual machines is not accurate [9].
back-end driver, we do not need a large number of preallo-
cated receive buffers. However, we need a sufficient number
of buffers if the front-end driver slower than the back-end 4.2 Impact of Network Buffer Size
driver. The back-end driver of KVM requires 256 preallo- As mentioned in Section 3.3, we analyzed the impact of net-
cated receive buffers, but we have discovered that 256 buffers work buffer size on bandwidth. Fig. 6 shows the variation
are excessively large for Gigabit Ethernet as discussed in of send bandwidth with different sizes of the TCP buffer.
Section 4.2. We can observe that the bandwidth increases as the kernel
buffer size increases. However, the bandwidth does not in-
Fig. 4 shows how we control the number of preallocated re- crease anymore after 68 KByte of TCP buffer size, because
ceive buffers. The typical RX virtqueue has the used ring 68 KByte is sufficient to fill the network pipe of Gigabit
and available ring areas, which store descriptors for used Ethernet. Fig. 7 shows the experimental results for receive
and unused preallocated buffers, respectively. Our front-end bandwidth, which also suggests 68 KByte as the minimum
driver introduces the empty ring area to the RX virtqueue buffer size for the maximum receive bandwidth. Thus, we
in order to limit the number of preallocated buffers. At the increased the TCP buffer size from 16 KByte to 68 KByte
initialization phase, we fill the descriptors with preallocated for our virtio.
buffers until desc_head_idx reaches to the threshold defined
as sizeof (virtqueue) − sizeof (empty ring). Then, whenever We also measured the network bandwidth varying the size
the interrupt handler is invoked, it enqueues the descriptors of the RX virtqueue as shown in Fig. 8. This figure shows
of new preallocated buffers as many as vring_used.idx − that the network bandwidth increases until the virtqueue
used_cons_idx (i.e., size of used ring). The descriptors in size becomes only 8. Thus, we do not need more than 8
the used ring are retrieved by the interrupt handler as men- preallocated buffers for Gigabit Ethernet though the default
tioned in Section 3.2. Thus, the size of empty ring is con- virtqueue size is 256.
stant. We show the detail analysis of tradeoff between the
RX virtqueue size and network bandwidth in Section 4.2. In summary, we increased the in-kernel send and receive
TCP buffer sizes from 16 KByte to 68 KByte, which requires
4. PERFORMANCE EVALUATION additional memory resources of 104 KByte (= (68 − 16) × 2)
In this section, we analyze the performance of our design for higher network bandwidth. However, we reduced the
suggested in the previous section. First, we measure the im- number of preallocated receive buffers from 256 to 8 without
pact of network buffer size on network bandwidth. Then, we sacrificing the network bandwidth, which saved 496 KByte
compare the bandwidth and latency of our implementation (= 256 × 2 − 8 × 2) of memory, where the size of mbuf is
with those of Linux. 2 KByte as mentioned in Section 3.3. Thus, we saved 392
Figure 6: Tradeoff between TCP buffer size and send Figure 8: Tradeoff between RX virtqueue size and
bandwidth. receive bandwidth.
Figure 7: Tradeoff between TCP buffer size and re-
ceive bandwidth. Figure 9: Comparison of bandwidth.
KByte (= 496 − 104) in total while achieving the maximum
maximum latency in the first iteration of every measure-
available bandwidth over Gigabit Ethernet.
ment for both RTEMS and Linux. Thus, we presume that
the lower maximum latency of RTMES is due to its smaller
4.3 Comparison with Linux working set.
We compared the performance of RTEMS-virtio with that of
Linux-virtio to see if our virtio can achieve comparable per-
formance to the optimized one for general-purpose operating 5. CONCLUSIONS AND FUTURE WORK
system. As shown in Fig. 9, the unidirectional bandwidth In this paper, we have suggested the design of the virtio
of RTEMS-virtio is almost the same with that of Linux- front-end driver for RTEMS. The suggested device driver
virtio, which is near the maximum bandwidth the physical can be portable across different Virtual Machine Monitors
hardware can provide in one direction. Thus, these results (VMMs) because our implementation is compliant with the
show that our implementation can provide quite reasonable virtio standard. The suggested design can efficiently han-
performance with respect to bandwidth. dle hardware events generated by the back-end driver and
can reduce memory consumption for network buffers, while
We also measured the round-trip latency in a way that two achieving the maximum available network bandwidth over
nodes send and receive the same size message in a ping- Gigabit Ethernet. The measurement results have showed
pong manner repeatedly for a given number of iterations. that our implementation can save 392 KByte of memory
We reported the average, maximum, and minimum laten- and can achieve comparable performance to the virtio im-
cies for 10,000 iterations. Fig. 10 shows the measurement plemented in Linux. Our implementation also has a smaller
results for 1 Byte and 1448 Byte messages. As we can see in jitter of latency than Linux thanks to smaller working set of
the figure, the average and minimum latencies of RTEMS- RTEMS. In conclusion, this study can provide insights into
virtio are comparable to those of Linux-virtio. However, the virtio from the viewpoint of the RTOS. Furthermore, the
maximum latency of RTEMS-virtio is significantly smaller discussions in this paper can be extended to apply to other
than that of Linux-virtio (the y-axis is log scale) meaning RTOS running in virtual machine to improve the network
RTEMS-virtio has a lower jitter. We always observed the performance and portability.
A. Liguori. kvm: the linux virtual machine monitor. In
Linux Symposium, pages 225–230, 2007.
[9] S.-H. Lee, J.-S. Seok, and H.-W. Jin. Barriers to
real-time network i/o virtualization: Observations on
a legacy hypervisor. In International Symposium on
Embedded Technology (ISET), 2014.
[10] P. S. Magnusson, M. Christensson, J. Eskilson,
D. Forsgren, G. Hallberg, J. Hogberg, F. Larsson,
A. Moestedt, and B. Werner. Simics: A full system
simulation platform. IEEE Computer, 35(2):50–58,
2002.
[11] M. Masmano, S. Peiró, J. Sanchez, J. Simo, and
A. Crespo. Io virtualisation in a partitioned system. In
6th Embedded Real Time Software and Systems
Congress, 2012.
[12] M. Masmano, I. Ripoll, A. Crespo, and J. Metge.
Figure 10: Comparison of latency. Xtratum: a hypervisor for safety critical embedded
systems. In 11th Real-Time Linux Workshop, pages
263–272, 2009.
As future work, we plan to measure the performance of our [13] A. Menon, A. L. Cox, and W. Zwaenepoel. Optimizing
virtio on a different VMM, such as VirtualBox, to show network virtualization in xen. In USENIX Annual
that our implementation is portable across different VMMs. Technical Conference, pages 15–28, 2006.
Then we will release the source code. In addition, we in- [14] G. Motika and S. Weiss. Virtio network
tend to extend our design for dynamic network buffer sizing paravirtualization driver: Implementation and
and measure the performance on real-time Ethernet, such performance of a de-facto standard. Computer
as AVB. Standards & Interfaces, 34(1):36–47, 2012.
[15] H. Raj and K. Schwan. High performance and scalable
6. ACKNOWLEDGMENTS i/o virtualization via self-virtualized devices. In ACM
This research was partly supported by the National Space Symposium on High-Performance Parallel and
Lab Program (#2011-0020905) funded by the National Re- Distributed Computing, June 2007.
search Foundation (NRF), Korea, and the Education Pro- [16] K. K. Ram, J. R. Santos, Y. Turner, A. L. Cox, and
gram for Creative and Industrial Convergence (#N0000717) S. Rixner. Achieving 10gbps using safe and
funded by the Ministry of Trade, Industry and Energy (MOT- transparent network interface virtualization. In ACM
IE), Korea. SIGPLAN/SIGOPS International Conference on
Virtual Execution Environments (VEE), March 2009.
7. REFERENCES [17] R. Russell. virtio: towards a de-facto standard for
[1] Oracle VM VirtualBox. http://www.virtualbox.org. virtual i/o devices. ACM SIGOPS Operating Systems
[2] RTEMS Real Time Operating System (RTOS). Review, 42(5):95–103, 2008.
http://www.rtems.org. [18] R. Russell, M. S. Tsirkin, C. Huck, and P. Moll.
[3] P. Barham, B. Dragovic, K. Fraser, S. Hand, Virtual I/O Device (VIRTIO) Version 1.0. OASIS,
T. Harris, A. Ho, R. Neugebauer, I. Pratt, and 2015.
A. Warfield. Xen and the art of virtualization. ACM [19] L. H. Seawright and R. A. MacKinnon. Vm/370 - a
SIGOPS Operating Systems Review, 37(5):164–177, study of multiplicity and usefulness. IBM Systems
2003. Journal, 18(1):4–17, 1979.
[4] Y. Dong, X. Yang, J. Li, G. Liao, K. Tian, and [20] J. Sugerman, G. Venkitachalam, and B.-H. Lim.
H. Guan. High performance network virtualization Virtualizing i/o devices on vmware workstation’s
with sr-iov. Journal of Parallel and Distributed hosted virtual machine monitor. In USENIX Annual
Computing, 72(11):1471–1480, 2012. Technical Conference, June 2001.
[5] S. Han and H.-W. Jin. Resource partitioning for [21] S. H. VanderLeest. Arinc 653 hypervisor. In 29th
integrated modular avionics: comparative study of IEEE/AIAA Digital Avionics Systems Conference
implementation alternatives. Software: Practice and (DASC), 2010.
Experience, 44(12):1441–1466, 2014. [22] L. Xia, J. Lange, and P. Dinda. Towards virtual
[6] C. Herber, A. Richter, T. Wild, and A. Herkersdorf. A passthrough i/o on commodity devices. In Workshop
network virtualization approach for performance on I/O Virtualization (WIOV), December 2008.
isolation in controller area network (can). In 20th
IEEE Real-Time and Embedded Technology and
Applications Symposium (RTAS), 2014.
[7] J.-S. Kim, S.-H. Lee, and H.-W. Jin. Fieldbus
virtualization for integrated modular avionics. In 16th
IEEE Conference on Emerging Technologies &
Factory Automation (ETFA), 2011.
[8] A. Kivity, Y. Kamay, D. Laor, U. Lublin, and