With the continuing rise of cloud technology, hypervisors play a vital role in the performance and reliability of current services. Hypervisors implement interfaces providing call-based connectivity to hosted virtualization-aware virtual machines. One of them is the hypercall interface, allowing hypervisor service requests from virtual machines. A hypercall injection tool measuring hypercall execution times must minimize internal overhead. Among other things, limiting logging to strictly required information is crucial. However, checks for values to log for every injection requires executing many branch instructions. We assess the performance diference between using and avoiding such branching, measured by the hypercall throughput of the injector tool.
Today, hypervisors are virtually omnipresent. They are widespread throughout data centers,
functioning as the backbone of cloud computing [
Virtualization allows to create virtual instances of physical devices called Virtual Machines (VMs). In a virtualized environment, governed by a hypervisor, VMs share resources. Hypervisors implement interfaces providing call-based connectivity to virtualization-aware hosted VMs. One of them is the hypercall interface, allowing for VMs to request services from the hypervisor. Hypercalls are software traps from a VM to the hypervisor. Before introducing x86 hardware virtualization in 2006, hypercalls were one solution to run virtualized OSs. Nowadays, while technically not required, hypercalls are still a common utility to improve eficiency or ofer additional features.
The crucial role that hypervisor play in today’s infrastructure requires robustness and high
performance, among other properties. We proposed a framework to help testing these
qualities [
Other works concentrate on reducing the cost of branches if that cannot be removed [
The remainder of this paper is structured as follows: Section 2 presents background knowledge about virtualization, Hyper-V, hypercalls, and branching in processors. Next, Section 3 introduces the measurement approach and environmental conditions during its realization; Section 4 then presents and discusses the measurements. Finally, Section 5 concludes the paper.
In a non-virtualized scenario, an OS manages physical hardware (i.e., processor, memory, and IO devices) and provides and schedules physical resource accesses for applications running on top. Virtualization describes the concept of introducing an abstraction layer above the hardware. That layer, called the hypervisor or Virtual Machine Monitor (VMM), provides a set of virtual resources, which can form multiple virtual machines managed by independent OSs.
One way to classify hypervisors is by whether directly control the hardware or run on top of
an OS [
Para-virtualization applies changes to the source code of OSs themselves. These
modifications allow the hypervisor and VMs to interact more eficiently, e.g., by using abstract IO
interfaces instead of emulating existing, physical devices, reducing overhead and improving
performance [
Hyper-V is an x86_64 hypervisor developed by Microsoft [
Hyper-V ofers various interfaces for VM-Hypervisor and VM-VM communication [
Hypercalls are triggered by special processor instructions that transfers execution control from VM to hypervisor. Hyper-V expects a call code present in a specified register beforehand. Also, parameters for the call can be placed in registers and memory, depending on the calling convention. After processing a hypercall, Hyper-V returns a result value, which indicates either a successful execution or an error code (e.g., missing privilege, out of memory).
Apart from virtualization terms, some background knowledge about CPU execution is
required to understand the goal of this paper. Modern processors try to execute as many
instructions as possible at a time. Branching instructions are problematic because the processor does
not know which instructions follow until the branch is fully executed. CPUs deploy branch
prediction to guess and speculatively execute further instructions, which helps to keep the
performance penalty down for well predictable branches [
The hypercall injector consists of three parts. There has to be a description of the hypercall workload, called the campaign file. It consists of binary data, describing which hypercalls with which parameters should be executed, and also if any delays should be waited between calls. A Windows kernel driver, the so-called injector module, executes the campaign. It loads the campaign into memory, executes the call and delay instructions one by one, and in between logging values if required and storing them in a file in the end. These two are connected by a desktop application. It takes the path to the campaign file as an argument, along with which values to log. Then, it prepares and loads the injector kernel module, and passes all the information required to execute the campaign correctly.
We support the logging of timestamp pairs for each action (hypercall, delay), execution times (diference between the timestamps), as well as result values and all output values. Hyper-V stores output values on a dedicated memory pages. Saving these values therefore requires storing 4MB of data, which is slow. Thus, always storing all values is not viable; the slowdown would be too big for a test campaign, e.g., a stress test ignoring log values and injecting hypercalls as fast as possible. That is why the required log values are passed to the desktop application. Yet, there are two diferent ways of handling optional logging values during execution. Firstly, the probably more natural approach is too place if statements in the main injection loop. Listing 1 shows a pseudo-C-code of this approach. The loop works through the actions of the campaign. We are interested in issuing as many calls as possible, so there must not be any delays in the workload. Accordingly, we can skip the handling of delays. Without any logging, hypercalls need their parameters prepared in memory and the processor registers, and the call must be invoked. Additionally, depending on the requested log values, the loop has to optionally take timestamps, calculate execution time, and store values to the log bufer. These if statements introduce a lot of branches. It has to be noted however that they always take the same branch as the requested log values cannot change during the campaign. The speculative execution engine of the processor should have no problem getting the branch right.
The alternative approach is the implementation of the loop for every combination of log values, which does not require any ifs. This still requires branching depending on the log values, but only once at the start. As a drawback, this approach causes lots of duplicate code, increasing driver size (not really an issue at this scale) but also drastically hurts maintainability. To test
Listing 1: Main injection loop with if statements if the chaos of copy-pasted injection loops is worth it, we want to determine the maximum hypercall throughput of both configurations.
We tried focusing the workload to spend as much time as possible injecting hypercalls in order to spot an eventually existing performance diference. By choosing an invalid hypercall code (hypervisor detects quickly and returns with an error) and passing no parameters, time for other tasks is reduced as much as possible. The measurements are performed on an Lenovo ThinkPad P1 with an Intel i7 9850H CPU and 2 × 16GB of 2667MHz DDR4 RAM. Unfortunately, the laptop form factor means that the CPU will probably thermally throttle under high load, i.e., rapid hypercall injection. Maximum throughput during short burst as well as continuously sustained high load throughput are of interest, so we chose the following methodology:
A single test campaign contains 50 million invalid hypercalls. Each configuration starts with a processor that has been idling for at least five minutes, to ensure fair starting temperatures in the heat sink. The campaign is executed 35 times in a row without breaks. The desktop application is adapted to log the execution time the kernel driver requires to perform the workload. Tested configurations are of course the branching and branchless variants, each compiled once as debug executables and once with Release optimizations by Microsoft’s C++ compiler. The next section is now going to illustrate the measurement results. iteration 30
In this work, we shared our findings regarding the cost of including if statements into the main loop of a hypercall injector tool. Results showed that even though the same branches are always taken, the penalty is high enough to make an significant diference in hypercall throughput. In conclusion however, it should be stated that working dozens of slight variations of the same loop is not practical if changes happen to the code occasionally. However, a compromise of maintenance and performance should be achievable: In performance-critical cases that require no logging or only time measurements, a dedicated, no-branch loop can be implemented. For other scenarios, which are more concerned with values that hypercalls return, performance is hindered by logging but usually not of great importance - here, a branching implementation can save lots of duplicate code.
This work was written in Cooperation with SPEC RG Security. Lukas Ifländer made his initial contributions as group leader with University of Würzburg.