This paper explores the security issues found in Container Orchestration Systems (COS) like Kubernetes, Docker Swarm, and Apache Mesos, which are crucial for running modern cloud-based applications. Although COS makes it easier to deploy and manage applications, it also brings a set of security challenges. This paper discusses common security risks in COS, including unsafe configurations, threats to the supply chain, and setup mistakes that could let attackers gain unauthorized access or disrupt the system. The paper examines the responsibilities of different parts of COS. It points out the dangers of giving too many permissions and not having strong network security measures. The discussion covers how containers and the process of managing them should be securely set up to prevent vulnerabilities. Additionally, the paper looks at the security of microservices, a way of designing applications as a set of small services. It talks about how to safely deploy these services, manage their communication, and secure them using various tools and standards. A major focus of the paper is on how COS decides where to run containers, known as scheduling strategies, and how these can affect security. It reviews different strategies and proposes new ones that focus on the importance of each part of an application. By spreading out important parts across different servers, the system can be made more secure. The paper suggests ways to arrange containers in a way to reduces the chance of a widespread attack if one part gets compromised. In summary, this paper dives into the security aspects of COS, presenting a detailed look at the risks involved and offering guidance on how to secure these systems. It emphasizes the need for careful setup, constant monitoring, and smart strategies to place containers, aiming to protect against the ever-changing security threats in the world of cloud computing.
COS plays a crucial role in modern cloud-based software ecosystems [1, 2]. The fundamental concept employed by COS is a container, which is a lightweight package of software bundled with all necessary dependencies required for running the software. Utilizing containers significantly streamlines the provisioning of hardware and ensures reproducible application behavior across various platforms and operating systems. As containers encapsulate all dependencies, they eliminate the risk of conflicts or issues with outdated versions. This contributes to the speed of cloud application development and facilitates the continuous delivery [3] process. Often used examples of COS include Kubernetes, Docker Swarm, and Apache Mesos [4].
The popularity of COS is related to the rise of applications that utilize a microservices architecture [5, 6]. However, the transition to COS and microservices is not without its challenges. Organizations must navigate the complexities of container management, orchestration, and security, which demand a deep understanding of the underlying infrastructure and a careful approach to design and configuration. The shift also necessitates a cultural change, creating an environment of continuous learning, collaboration, and innovation among teams. As businesses increasingly rely on these systems, ensuring the security, reliability, and performance of COS becomes important. This includes not only protecting against external threats but also managing internal risks such as service dependencies, network configurations, and data management practices.
Microservices architecture primarily facilitates faster software delivery [7] by creating bounded contexts or areas of responsibility managed by different teams. Using microservices offers additional advantages. One of these is the containment of application failure scope or “blast radius” [8]. Unlike monolithic applications, those divided into separately deployable pieces are less prone to complete failure due to a single subsystem’s issue, as these are independent applications and processes. Scalability is also an essential aspect [9]. The separation into distinct functions provides better granularity and scalability options. COS can dynamically allocate resources to different functions based on the load, optimizing the system’s scalability.
While microservices architecture with the use of COS feature allows all these benefits, the application architecture is also important, without proper separation of responsibilities and functions proper scalability, resilience, and delivery speed might be hard to achieve. Fig. 1 illustrates the key components utilized by COS: 1. Cluster: This represents a set of nodes, indicating the state of available hardware or virtual resources. 2. Nodes: These are individual physical or virtual units where containers can be deployed. 3. Container Runtime: This is the technology used to run containers, such as Docker [10]. Every node must have a container runtime installed to facilitate container operations. 4. COS Agent: Typically, this is a program that operates within a node and primarily functions as the communication point to manage operations on that node. While the container runtime focuses on managing the containers themselves, the COS agent delegates such requests to the container runtime. 5. Discovery Component: This component is responsible for tracking all the containers and associated applications. It facilitates internal application communication by serving as a central configuration point and may be utilized by a load balancer. 6. Scheduler: Playing a crucial role, the scheduler’s primary function is to identify a suitable node for container deployment. Selecting a deployment strategy involves analyzing the system’s use case, typically balancing between efficient utilization of cluster resources and application resilience or fault tolerance. Common strategies include binpacking and spreading [11]. Binpacking aims to pack as many containers as possible into a single node to maximize resource utilization. Spreading focuses on distributing containers across the emptiest nodes to enhance application fault tolerance [12]. This approach allows for the deployment of multiple instances of the same application component across different nodes, thereby reducing the risk of simultaneous failures due to hardware or other issues.
The adoption of COS does not come without its drawbacks. While COS efficiently organizes the management of running systems, it also adds to the system’s complexity. This complexity is necessary to accommodate the diverse requirements of various applications. For example, Kubernetes alone offers more than ten different configurable abstractions to provide end-users with the needed flexibility. However, this complexity opens the door to potential issues, such as the risk of misconfiguration, which can lead to human errors and security vulnerabilities.
Furthermore, the process of packaging software into container images requires precise attention to ensure that the software is free from vulnerabilities. The architecture of the application significantly influences the overall security of the system. Important considerations include how application secrets are managed, who has access to them, and the level of encryption employed in the communication between container nodes.
The multifaceted nature of COS inherently makes security a more challenging aspect due to the increased number of entry points, exposed APIs, and the amplified potential for errors. This article aims to explore some of the threats associated with COS, with a particular focus on the implications raised from decisions made by the scheduler component, highlighting the critical nature of this aspect in the COS infrastructure.
The OWASP vulnerabilities [13] provide a comprehensive list of the top ten security risks for Kubernetes-based applications, encompassing issues such as insecure configurations, supply chain vulnerabilities, overly permissive Role Based Access Control (RBAC) configurations, lack of centralized policy enforcement, inadequate logging and monitoring, broken authentication mechanisms, missing network segmentation controls, misconfigured cluster components, and outdated and vulnerable components. Martin and Hausenblas delve into these issues, categorizing threats and proposing defensive measures for various scenarios that apply to Kubernetes COS [14]. They also detail the roles and responsibilities of different Kubernetes components and highlight the importance of isolating the container runtime from the host operating system to increase security. Creane and Gupta extend the discussion on security beyond COS itself, addressing potential threats that may arise during the pre-deployment phase, such as when faulty images are built and deployed [15]. They advocate for robust monitoring techniques, alerting rules, machine learning-based systems, proper network configuration, and secure application exposure methods.
Ugale et al. focus on container vulnerabilities in cloud environments, proposing a security framework that conducts vulnerability scans at various levels [16]. Security challenges in systems reliant on containers are explored by Sultan et al. highlighting the need for enhanced vulnerability management [17]. Rice discusses fundamental security concepts crucial for protecting applications in containers, introducing the “blast radius” concept to limit the impact of threats [18]. Lopens et al. propose mitigating security risks by adapting the seccomp profile, which restricts container system calls to minimize the attack surface, emphasizing the development of these profiles in a fast and scalable manner [19]. Belair et al. focus on the security features of the Linux kernel that are employed at the virtualization boundary between the operating kernel and containers [20].
Dias and Siriwardena comprehensively address the mechanisms utilized in microservice security, including deployments, communication, API gateways, JWT tokens, and the OAuth 2 standard [21]. They delve into the deployment strategies using Kubernetes and discuss methods to ensure secure configurations of Docker or alternative container engines. Berardi et al. explore the significance of the human element in the security of applications employing a microservices architecture [22]. The challenges of microservices security, along with contemporary solutions to key issues, are presented by Driss et al. [23], providing a crucial resource for secure software researchers and practitioners by offering a comprehensive catalog of security solutions and mechanisms for applications based on microservices.
One common issue in the configuration of COS, as noted in the OWASP top ten security risks for Kubernetes [13], risk of insecure workload configurations. Running application services as the root user, when not necessary, grants the application excessive permissions, potentially enabling the execution of harmful operations such as initiating malicious processes. The use of a file system that allows write operations can inadvertently permit the installation of unauthorized software, leading to unexpected and possibly malicious container behavior. Moreover, the use of privileged containers grants additional kernel capabilities, which, when coupled with other misconfigurations, might pave the way to access the host (node) itself. Gaining control over the host opens up access to other containers on that node, the container runtime, and the COS agent, all of which could be manipulated for malicious purposes. Furthermore, overly permissive configurations of RBAC permissions can provide unintended access to containers, posing a significant security risk. Addressing these vulnerabilities requires strict adherence to security best practices, including minimizing permissions and ensuring robust access controls within the COS environment. Additionally, compromising the supply chain is another avenue through which containers can be made vulnerable. Exploiting the software architecture deployed within the COS system and introducing vulnerabilities can also provide attackers with potential points of exploitation.
Weak authentication mechanisms and absent network security controls can significantly broaden the scope of an attack, potentially extending it from a single container to an entire node, and from one node to others within the network. A lack of encryption within the cluster can enable an attacker who has gained access to a container to exploit network communications by intercepting traffic between containers. Moreover, permitting direct access between hosts could allow an attacker to exploit more critical APIs available within the cluster (Fig. 2). The outlined approaches reveal that attack vectors typically start with minor threats within the system and progressively expand by accessing additional components. Although adhering to best practices in COS usage can significantly reduce the likelihood of such attacks, it’s important to recognize that the risk can never be eliminated. These best practices involve the use of scanning tools and properly configured cloud environments to ensure that permissions are appropriately restrictive. Nonetheless, the specific requirements of individual applications and the urgency of deployment can sometimes lead to more lenient configurations, increasing the vulnerability to security breaches.
Leveraging the capabilities of scheduling strategies in COS can play a crucial role in mitigating the impact of attacks. The scheduler, responsible for deciding which node will host a new set of containers, inherently influences the distribution of containers across the cluster. This distribution, or locality, can be an important factor in system security.
Recent research indicates that scheduling strategies can be aligned with multiple objectives, such as resilience to failures, efficient resource utilization, ensuring application accessibility, and rapid deployment speeds. Security considerations can also be integrated into these objectives. For instance, the binpacking strategy prioritizes nodes that are already heavily loaded, optimizing resource usage. However, this strategy also concentrates on the components of an application, potentially increasing its vulnerability to localized attacks. Conversely, strategies that spread application components across different nodes can enhance
security. If the components are distributed, gaining unauthorized access to one node does not automatically compromise the entire system, as each node operates independently. This approach can make it considerably more challenging for an attacker to inflict widespread damage, underscoring the importance of container distribution in increasing the overall security posture of the
The following method aims to minimize the impact
of increases the likelihood that containers of similar importance will be placed on different nodes. Strategies like spreading or usage of affinity controls in Kubernetes already help in distributing instances of the same application across different nodes for enhanced fault tolerance. Considering the importance level of services adds an extra layer of security. This approach effectively reduces the risk of an attacker compromising critical components by gaining access to a limited part of the system. The equation to compute the weight of the node, where weight can be used in the node selection process is the following: = ∑
1 one is full or does not meet other conditions of the strategy.
Another risk involves the direct exploitation of vulnerabilities in the application architecture. Consider a scenario where an attacker targets a less important component of the application, possibly because it is less monitored or more prone to oversight. The attacker generates an excessive load on this service. If the container’s resource limits are not properly configured, it could consume an increasing share of the node’s resources, leading to performance degradation of more critical application components on the same node. depicted as container 1 in Fig. 3. Although the historical transactions financial application. While this architecture prioritizes availability over consistency (in terms of CAP theorem [24]), which might be contentious for such applications, the issue persists in other cases where systems favor availability.
This scenario introduces an additional potential objective for scheduling strategies: separating publicly accessible applications from those accessible only within the cluster. This separation aims to prevent situations where an attacker might indirectly overload an internal application by exploiting an externally accessible service. Implementing this measure through scheduling strategies can enhance the system’s security.
The following formula represents the computation of a node weight that along with container ranks also takes into account the public exposure factor: = ( ∑ ) + ( ∑ ) (3) container is publicly available and 0 if it’s not, X is the coefficient used to control the impact of containers rank, Y is the coefficient used to control the impact of containers availability.
System malfunction might be triggered not only by direct the external load but also due to time events, such as salaries coming out at the beginning of the month or high load during the import of bank statements.
Additionally, it’s crucial to configure resource limits properly for each container, ensuring that there’s a defined threshold for resource usage. Alongside setting these limits, robust system monitoring with alerting rules is essential. Prometheus and Grafana are tools that can be used for such systems, providing great flexibility and feature sets, including anomaly detection and alerting [25].
Implementing autoscaling can further improve the system’s resilience, enabling it to handle normal usage growth and providing indicators for potential attack attempts. These measures collectively ensure that the application not only remains operational under typical conditions but also offers a level of protection against such attack scenarios.
The exploration into the impact of scheduler decisions on application security within COS reveals a complex relation between system configurations,
scheduling security outcomes.
This strategies, discussion underscored the multifaceted nature security within COS, particularly focusing on
arise from various sources, including insecure workload configurations, supply chain risks, and the complex architecture of microservices.
The consideration of service importance and the strategic placement of containers to avoid co-locating critical components on the and has of minimizing the attack surface. However, this same
node strategy is are not significant steps toward without challenges.
The dynamic nature of COS and the continuous evolution of threats necessitate a proactive and adaptable approach to security. This involves not only intelligent scheduling but also robust monitoring, precise configuration management, and the implementation of responsive mechanisms such as autoscaling to address any anomalies promptly. Suggested methods for weighting nodes, embedded into existing strategies, present a challenge: deciding which is more important—resource consumption or reduced risk.
The insights from related works in the domains of
COS security,
This paper has extensively discussed the security intricacies and challenges associated with COS like Kubernetes, Docker Swarm, and Apache Mesos, which play key roles in cloudbased software ecosystems. While COS offers substantial benefits in terms of application deployment, scalability, and management, it also introduces a significant amount of security vulnerabilities and risks. The exploration of
COS security highlighted that issues such as insecure workload configurations, supply [5] A. Saboor, et al., Containerized chain vulnerabilities, and inappropriate Microservices Orchestration and permission settings can significantly reduce Provisioning in Cloud Computing: A the integrity and safety of the entire system. Conceptual Framework and Future
Through an in-depth analysis of related Perspectives, Appl. Sci. 12(12) (2022)
works, the paper emphasized the necessity of 5793. doi: 10.3390/app12125793.
using strong security measures, including [6] V. Bushong, et al., On Microservice
vulnerability management, robust configuration Analysis and Architecture Evolution: A
practices, and the implementation of necessary Systematic Mapping Study, Appl. Sci.
security frameworks. The role of COS 11(17) (
Moreover, the paper underscored the Reliability: The Technologies and Tools,
critical influence of scheduling strategies on Cloud Reliability Eng. (
References [11] C. Cérin, et al., A New Docker Swarm Scheduling Strategy, IEEE 7th [1] A. Prajapati, D. Patel, Container International Symposium on Cloud and Scheduling: A Taxonomy, Open Issues Service Computing (2017) 112–117. doi: and Future Directions for Scheduling of 10.1109/SC2.2017.24.
Containerized Microservices, SSRN [12] M. Kleppmann, Designing Data-Intensive
(2024). Applications: The Big Ideas Behind
[2] O. Oleghe, Container Placement and Reliable, Scalable, and Maintainable
Migration in Edge Computing: Concept Systems. O’Reilly Media (2017).
and Scheduling Models, IEEE Access 9 [13] Owasp Kubernetes Top Ten. OWASP.
(
10.1007/s10270-020-00828-z. [16] S. Ugale, A. Potgantwar, Container [4] L. Mercl, J. Pavlik, The Comparison of Security in Cloud Environments: A Container Orchestrators, International Comprehensive Analysis and Future Congress on Information and Directions for DevSecOps, Eng. Proc. Communication Technology (2018) 59(1) (2023) 57. doi: 10.3390/engp 677–685. roc2023059057.
[17] S. Sultan, I. Ahmad, T. Dimitriou,
Container Security: Issues, Challenges,