Container virtualization has become the tool of choice for running isolated applications in cloud environments. Linux-Containers virtualize at the operating system level, with multiple containers running atop the operating system kernel directly. Therefore, threats to one container are potentially threats to many others. Especially for PaaS and Serverless providers, the secure execution of untrusted workloads on their platform in order to mitigate software vulnerabilities from spreading has high priority. Containers face a variety of di erent threats, vulnerabilities and historical weaknesses that need to be considered and defended against. This paper presents current approaches to securing container workloads. gVisor, Kata Containers and Firecracker are presented and compared with each other. Although sandbox containers have di erent attack surfaces such as the container daemon process, network, or storage, this paper focuses on the Linux kernel itself as a vulnerability in sandbox containers and examines how each approach implements protection.
1:2
ware virtualization. Time has shown that container virtualization and hardware virtualization are not
opposed to each other. Rather, the technologies are complementary, especially when the focus is on
security. Hardware virtualization is able to isolate computing resources while Containers focus on
isolating processes instead of resources. A fully virtualized system gets its own set of resources assigned
to it and minimal sharing takes place. Virtual Machines (VMs) offer a comparatively high degree of
isolation and thus security for workload isolation. However, the replication of a hardware environment
including the operating system leads to an overhead and the start up time of a virtual machine is
significantly slower than with container virtualization. Compared to VMs, containers offer a significantly
less isolation level but can be deployed immediately without complex installation processes, therefore
container virtualization is not suitable for implementing security concepts [
To secure container workloads, multi-layered security is necessary and issues from both approaches are needed. On the one hand, the performance of container deployment and on the other hand the isolation of the host system, which a virtual machine offers. In this paper, we have reviewed which approaches are currently being driven forward for the implementation of an additional security layer. We have addressed the question of whether these approaches can be distinguished from each other on an abstract level. First, scenarios for container isolation and currently used security mechanisms for containers are presented and the question why an additional security layer is necessary is answered. On the basis of this, gVisor, Kata Containers and Firecracker are presented. In conclusion, the approaches are compared with each other and an overview of the current development approaches is given.
This section covers backgrounds for container isolation. It starts with use cases and scenarios for container isolation. It then describes the status of containerized runtime environment security and the need for layered security.
2.1.1 Sandbox untrusted/vulnerable code. Containers are ideal for isolation on application level. There are use cases where a user wants to run an application with a number of previous vulnerabilities on a host machine. There is a risk that an untrusted attacker may be aware of these vulnerabilities and new vulnerabilities may be easily found. To protect the host system, the use of a highly isolated container environment for the application is useful. Another use case would be a scenario in which code is to be executed on a host machine that cannot be trusted. This may be relevant if access to the source code is not given. The lack of transparency can lead to a security risk. Container isolation is also suitable for protection here.
2.1.2 Defense in depth scenarios. For the processing of sensitive workloads and data, defense in depth is of high relevance. In use cases where the code is trusted, it may still be important to protect it from possible attacks. This can be especially important for medical or financial services.
2.1.3 Serverless Computing Scenarios. Serverless Computing is a cloud computing paradigm where the cloud provider provides a runtime environment for executing server-side logic. Server-side logic is implemented by means of short-lived, stateless functions called serverless functions. The execution environment for these functions is called Function-as-a-Service (FaaS) [Spillner 2018]. Serverless Computing is the ideal of an event-based application architecture - an application architecture that emerges from the current trend in container and microservices architectures [Baldini et al. 2017]. As a result, functions are executed in a serverless application in response to the triggering of certain events. If a serverless function is triggered, the cloud provider dynamically allocates the computing capacity required to execute the function. After the execution has been completed, the computing capacities are released again. In this scenario, the cloud provider does not know the code executed by users in the runtime environment provided. This code can be potentially harmful to its environment. For serverless platform providers, it is therefore of great interest to isolate the individual runtime environments of users strongly.
2.1.4 Multi-Tenancy Scenarios. There are more than one type of multi-tenancy, and multiple definitions of a tenant. Depending on the tenant scenario, the required degree of isolation may vary. Assuming a multi-tenancy model in which multiple untrusted users (Tenants) may execute untrusted code in a shared cloud environment, thus code deployed by tenants can potentially be executed on the same host machine on which code from other tenants is simultaneously executed. Compared to Serverless Computing, this means that the cloud provider makes the compute resources available to the user, but the user himself can use his own runtime environments, e.g. in the form of his own containers. The cloud provider does not know the user’s application and runtime environment in advance. Unlike multi-tenant scenarios, where users of a cloud environment trust each other and also trust the executed code, e.g. within a company with different departments as tenants sharing a compute cluster, the described model requires a strict isolation of resources belonging to different clients.
Today many mechanisms based on Linux technologies are used to isolate container runtime environments.
2.2.1 AppArmor / SELinux. AppArmor / SELinux are security frameworks for Linux. As
Mandatory Access Control (MAC) systems they control applications individually. For these, access rights can
be defined in profiles that are finer than the general file rights. Beside the predefined ones, own profiles
can be set up. The purpose of AppArmor / SELinux is to protect security-critical applications, i.e.
primarily applications / processes with network access, but also office applications which could possibly
compromise the system by loading infected documents [
2.2.2 Capabilities. The basic idea behind Linux capabilities is to break up the monolithic root
privilege that Linux systems have had, so that smaller more specific privileges can be provided where
they’re required. This helps reduce the risk that by compromising a single process on a host an
attacker is able to fully compromise it [
2.2.3 Namespaces. An instance of a namespace defines a new environment that virtualizes certain
operational resources such as process users, the file system, or the network in a very lightweight way.
They abstract the respective global system resource in such a way that it looks like an independent
isolated instance for a process within the respective namespace. Lightweight means that no hypervisor
is needed, but the processes simply do not see the other instances of a resource, but they continue to
run in the same kernel [
2.2.4 User Permissions. For the isolation of container runtime environments the Linux own user system is used. Discretionary Access Control (DAC) is restricting access to objects based on the identity of subjects and groups to which a certain subject belongs to [Rusling 1999].
2.2.5 cgroups. Using ”control groups” a user can combine several processes into one group. The
user can then provide these processes and all child processes with parameters for specific subsystems.
For example, a subsystem is a resource controller that manages the available memory. cgroups can be
used to define limits that must not be exceeded by resources, such as a lot of memory. Users can give
priority to some resources over others, such as more frequent processing by the CPU. cgroups can also
measure how many resources have been consumed. With the help of cgroups, entire groups of processes
can also be controlled, e.g. they stop or continue to run [
2.2.6 seccomp-bpf. ”Secure computing mode” is a simple and effective sandboxing tool. It allows
the user to attach a system call filter to a process and all its descendants, thus reducing the attack
surface of the kernel. Seccomp filters are expressed in Berkeley Packet Filter (BPF) format [
Attacks aimed at breaking out of a containerized runtime environment are particularly critical and require a lot of attention. There is always a risk that processes will break out. If this happens, not only the individual container instance is affected, but also all other processes that also run on the host system.
The primary target of such an outbreak is the operating system kernel. If an attacker controls it,
he has the entire system in his hands. Container technologies like
Using a container runtime is therefore not sufficient for isolating applications. To safely isolate applications from each other, one or more additional security layers beyond the boundaries of the container runtime environment are required as shown in F4906384219579641ig. 1 .
This section introduces gVisor, Kata Containers, and Firecracker as approaches to isolating container workloads. It is being discussed whether the approaches meet the requirements of multi-layer security.
On May 2, 2018 Google released their container runtime sandbox
Applications that run in containers on the host system access system resources in the same way
that applications that are not in containers do. System calls are sent directly to the host kernel. Even
without gVisor, the kernel limits the access that a containerized application can make to the system
resources, but such a kernel still offers a large attack surface. gVisor does not use hardware
virtualization to isolate containers, i.e. containers are not packed into virtual machines to provide an additional
security boundary and therefore no virtualized hardware is passed to a guest kernel via a Virtual
Machine Manager. Instead gVisor uses a kernel that is active as a normal, non-privileged process and
supports most Linux system calls. gVisor intercepts application system calls and acts as the guest
kernel, without the need for translation through virtualized hardware. Just like in a normal virtual
machine, an application running in a gVisor sandbox gets its own kernel and a selection of virtualized
devices isolated from the host and other sandboxes. Unlike containers packed in virtual machines,
gVisor is more lightweight, but provides a similar level of isolation and therefore security [
With gVisor, multi-layer security can be implemented. The emulated kernel in the userland represents the first security boundary that an attacker must overcome. The second security boundary is the process isolation mechanisms of the host kernel. Kernel features such as seccomp filters can be used to provide better isolation between host and gVisor kernels. However, they require the user to create a predefined whitelist of system calls. Due to the additional abstraction layer offered by gVisor, filter rules can be used much more universally. Creating system call filter rules for applications that are not known in advance is often a difficult task. The application of filter rules on gVisor as an interface between host and application is easier to configure.
Kata Containers version 1.0 was released on May 22, 2018 [Bertucio 2018]. The technical
foundations of Kata Containers are two other projects that have been in existence for quite some time. The
first are the Clear Containers, which Intel launched in 2015 [
The architecture of Kata Containers consists of six elements [
With Kata Containers multi-layer security can be implemented. The hypervisor provides an additional security boundary and thus increases the security of the sandbox. This makes it more difficult for a harmful process to break out. A sandboxed process must first convince the guest kernel to trigger a malicious virtual machine exit. This also significantly reduces the attack surface at the same time. Afterwards the hypervisor has to be exploited to break out of the sandbox and gain control over the host system. The degree of isolation by the first security boundary is reduced with the number of rights a container receives within the virtual machine. If the user is able to start his own guest systems in the virtual machine, the first security boundary can be omitted.
On November 27, 2018, Amazon Web Services introduced Firecracker, a virtual machine manager
that enables secure, multi-tenant, minimal overhead execution of container and function workloads,
as an open source pro
The main component of Firecracker is its own hypervisor which is directly based on Linux
Kernelbased Virtual Machine (KVM) capabilities. The foundation for this is the Crosvm project [
Firecracker implements multi layer security by defining different nested trust zones. The jailer process sets up system resources that require elevated permissions, drops privileges, and then executes into the Firecracker binary, which then runs as an underprivileged process. In addition, seccomp-bpf filters are used to limit system calls to the guest kernel. This represents the first security boundary. In order to take over the host system, an escape from the virtual machine must be successful and thus represents the second security boundary.
Although their concrete architecture is quite different, all three projects follow a similar approach. They build an additional layer between the application to be isolated and the kernel of the host system. The main difference between the three projects presented lies in the virtualization technology used to form the isolation layer.
Kata Containers chose QEMU as the foundation for its virtualization technology using ”QEMU light”. A sandboxed container runs within a minimal, lightweight virtual machine. Via an internal agent process and an external shim, runtime commands and IO request are intercepted. Firecracker follows a similar approach. Again, virtual machines are used to separate sanboxed applications from host kernels. Unlike Kata Containers, Firecracker does not use QEMU, but implements its own virtual machine manager that uses Linux Kernel-Based Virtual Machine (KVM). It emulates a minimal device to achieve low latencies when starting the VM and low memory footprint on the host system. At the same time, a trusted sandbox environment is provided for an isolated application within the virtual machine. Unlike Kata Containers, Firecracker does not start any containers in the virtual machine. Google’s gVisor, on the other hand, has a completely different strategy than the other two projects presented. Neither microVMs like Firecracker nor a combination of virtual machine and container like Kata Containers are used. Instead, a dedicated buffer layer is created for each individual container to provide an additional security boundary between host system and application. The layer created by gVisor runs as a process in userspace and allows the container to perform system calls without directly accessing the host kernel.
Although there are currently several projects with different ideas for the realization of container isolation, it must be mentioned that they are all more or less experimental at the time of this paper. Even though large cloud providers such as Google or Amazon are behind some of the concepts presented, all approaches are at a very early stage of development. Although Firecracker or gVisor are used in the cloud environments of the providers, it is not certain whether the solutions are also suitable for general production environments outside the provider-specific offerings.
An important step for the productive use of sandbox environments is the compatibility to common
container tools like Docker or orchestration tools like Kubernetes and OpenStack. Both Docker and
Kubernetes use containerd as container runtime. While gVisor and Kata Containers can already be
used as plugins for containerd and are therefore compatible with
This paper presented and compared approaches to isolate container workload. Different use cases and scenarios were presented in which the secure handling of containerized applications is of high priority. Furthermore, state of the art mechanisms for the isolation of containers were investigated and the question why multi-layer security is relevant to provide a secure sandbox environment was answered. Based on this, gVisor, Kata Containers and Firecracker were presented and explained whether the concepts can implement multi-layer security. Finally, the concepts were compared with each other and an outlook on further development was given. In future we seek to develop a comparison criteria catalogue that allows the presented, as well as emerging approaches to be classified and differentiated on the basis of an evaluation framework.
In summary, securing container workload is becoming increasingly important. Not only for providers of big public cloud environments, but for every cloud environment with multi-client capability and the execution of untrusted code on host machines. The development of the approaches presented remains exciting and it will be shown which of the approaches will prevail for general productive use.