-This paper presents a novel approach to writing TOSCA templates for application reusability and portability in a modular auto-scaling and orchestration framework (MiCADO). The approach defines cloud resources as well as application containers in a flexible and generic way, and allows for those definitions to be extended with specific properties related to a desired container orchestrator chosen at deployment time. The approach is demonstrated in a proof-of-concept where only a minor change was required to a previously used application template in order to achieve the successful deployment and lifecycle management of the popular web authoring tool Wordpress on a new realization of the MiCADO framework featuring a different container orchestrator.
MiCADO, container orchestration,
Cloud adoption by research, public sector and enterprise organizations continues to grow. Having flexible, on-demand access to computing resources and services can result in significant cost and time savings. Moreover, large, upfront capital investments can be replaced by day-to-day operational costs over a longer period of time. There are, however, definite barriers to entry for the scientific research community and smaller institutions that lack the cloud-specific skills and knowledge necessary for shifting to the cloud. Additionally, organizations may struggle with achieving maximum savings due to a lack of flexibility and scalability at the level of the application.
To support these groups, there is the need for a generic framework which provides support for launching and managing a variety of applications in the cloud. The framework should be tied to no specific cloud service provider and should support a mix of public, private and community clouds. It should also provide flexibility at the application level by providing optimized deployment and runtime orchestration with features such as automated scaling and enhanced security. This flexibility should be provided in the form of a single interface which describes the topology of cloud resources and governs the application with user-defined policies specifying performance, cost, security and other requirements necessary to see the application through its lifecycle.
The European funded COLA [1] (Cloud Orchestration at the Level of Application) project set out to address these issues, and design and develop a generic framework outlined above. Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The proposed framework is called MiCADO [2] (Microservices-based Cloud Application-level Dynamic Orchestrator), a platform for the deployment and dynamic automated scaling of applications in the cloud. MiCADO is entirely open source and implements a microservices architecture in a modular way. The modular design supports varied implementations where components can easily be replaced with a different realization of the same functionality. At the time of writing, the current implementation of MiCADO uses widely applied technologies such as Kubernetes [3] (container orchestrator), Occopus [4] (cloud orchestrator) and Prometheus [5] (monitoring), and some additional custom implemented components. Based on the modular design principles, replacing any of these building blocks is kept as simple as possible.
The user-facing interface for defining the required cloud
topology (containers and virtual machines) and the policies
which regulate the application and its infrastructure is an
Application Description Template [
There are two sections to an ADT – one to describe the
cloud topology of the application and infrastructure, and one to
describe the Quality of Service (QoS) parameters. The policies
which describe these non-functional requirements, such as
placement, data locality, security, scalability, and performance,
were discussed last year in the IWSG 2018 Conference
Proceedings [
In order to support MiCADO as a truly modular framework, the ADT interface has to be supportive of modularity as well. This modularity presented several challenges for designing such a template, especially insofar as describing the cloud resources so that they could be reused across different implementations of the framework. In following with other projects which have implemented TOSCA as a language, the early ADTs of MiCADO featured topology types for cloud resources (virtual machines, containers and volumes) which were all strongly related to their respective end components. This meant a loss of portability, since by changing an end component in the implementation, we now required new TOSCA types to be written and then referenced in new ADTs.
This paper presents a novel approach to writing cloud resource types for TOSCA, which offers reusability of that resource by different orchestration tools. The ADT format is compliant with our understanding and interpretation of TOSCA Simple Profile in YAML v1.0 [8], and follows the core values espoused by the newer Oasis Standards, TOSCA v1.1 and v1.2. The format provides more flexibility and control for those ADT authors who understand the underlying technologies of the respective components they are describing cloud resources for. At the same time, the ADT structure still features variable levels of abstraction, which make it possible for users without component-specific knowledge to author ADTs and deploy applications in MiCADO. We present the steps taken to build reusable types for containerized applications in TOSCA and the way to extend them for specific orchestrators as desired. As a proof-of-concept, we refer to a case study where a MiCADO demonstrator application had requirements exceeding the capabilities of the container orchestrator inside MiCADO at the time. We demonstrate a solution where the modularity of MiCADO is leveraged, and the existing ADT is easily reused to deploy the application under a new MiCADO implementation featuring a different container orchestrator.
The rest of the paper is structured as follows. First, the State of the Art is described, including an introduction to TOSCA, and a look at related work covering the adoption and interpretation of TOSCA in industry and academia. In section III a short introduction to MiCADO and its modular design is described. In section IV, we build the base TOSCA types for containerized applications and then demonstrate how those types can be extended for a specific container orchestrator. We conclude with a proof-of-concept demonstration where a single ADT is used to deploy a microservices application to a different implementation of MiCADO, featuring a different container orchestrator.
Containerization refers to a more lightweight virtualization
approach than that offered by virtual machines, whereby an
application is packaged with its specific dependencies on a
minimal virtualization layer, usually sharing the host kernel.
The most popular container technology of the moment is
Docker [9], but there are others gaining popularity such as:
crio [
The Topology and Orchestration Specification for Cloud
Applications is an OASIS Standard for describing the full
topology and operational behavior of an application running in
the cloud. The topology is defined as building-blocks called
nodes, which represent components such as the software,
virtual machines, storage volumes, and networks which make
up the application. The operational behavior is managed by
defined relationships between the above components and
through lifecycle management interfaces in the form of scripts,
configurations, or API invocations. There are many good
resources for TOSCA, ranging from journal publications [
There is an increasing number of industry products and amount of academic research coming out around the topic of TOSCA and descriptive cloud languages. While most TOSCA adopters claim to solve vendor lock-in while offering highlevels of flexibility and portability, not all of them leverage containers, and few describe cloud resources in a way that is portable across a modular framework.
One of the earliest TOSCA runtimes is OpenTOSCA [
Cloudify [
ARIA [
From an ARIA contributor, came the open-source Puccini
[
The open-source Alien4Cloud [
Another approach is used in the orchestration management
engine TosKer [
As early TOSCA did not inherently support use at run-time,
another solution, CAMEL [
The current wide range of approaches to TOSCA were
found to be either too complexly layered, or not generic enough
to serve the modularity of MiCADO. A previous publication
focusing on MiCADO Application Description Templates in
TOSCA [
MiCADO is an application-level multi-cloud orchestration and auto-scaling framework that is currently being developed in the European H2020 COLA (Cloud Orchestration at the Level of Application) project. The concept of MiCADO is described in detail in [2]. In this section a high-level overview of the framework is provided to explain its architecture, building blocks and implementation.
The generic, technology independent architecture of MiCADO is presented in Figure 1. MiCADO consists of two main logical components: Master Node and Worker Node. Master Node is the head of the cluster performing the collection of information on microservices, the calculation of optimized resource usage, the decision making, and the realization of decisions related to handling resources and to scheduling microservices. Worker Nodes are volatile components representing execution environments for the microservices. These nodes are continuously allocated/released based on the dynamically changing requirements of the running microservices. Once a new Worker Node is allocated and attached to the cluster, the Master Node utilizes its resources by allocating microservices on it. The input to MiCADO is a TOSCA-based ADT that will be the focus of this paper.
The MiCADO Master Node (box with dashed line on the left in Figure 1) includes six components. MiCADO Submitter is the primary service endpoint for creating an infrastructure to run an application, and managing this infrastructure and the application itself. The incoming ADT is interpreted by the MiCADO Submitter and related parts are forwarded to other key components. Creating new MiCADO Worker Nodes and deploying application containers on these Worker Nodes are the responsibility of the Cloud Orchestrator and Container Orchestrator components, respectively. The Cloud Orchestrator is responsible for communication with the Cloud API to allocate and release resources, and build up and shut down new MiCADO Worker Nodes when necessary. The Container Orchestrator allocates new microservices (realized by containers) on the Worker Nodes, keeps track of their execution and destroys them if necessary. The Monitoring System collects metrics on worker node resources and on resource usage of the container services, and makes this information available for the Policy Keeper component on the Master Node. It also provides alerting functionality in relation to the measured attributes to detect values that require reaction; these alerts will also be consumed by the Policy Keeper. Based on the metrics and alerts provided by the Monitoring System, the Policy Keeper applies implemented scaling policies to make scaling decisions and call the components responsible for allocating/releasing cloud resources and scheduling container services among the Worker Nodes. Moreover, this component makes sure that the Cloud and the Container Orchestrator are instructed in a synchronized way during the operation of the entire system. Lastly, the Execution Optimizer is a background microservice performing long-running calculations on demand for finding optimized setup of both cloud resources and container infrastructures.
MiCADO Worker Nodes (boxes with dashed line on the right in Figure 1) contain the Node/container monitor that is responsible for measuring the load of the resources and the resource usage of the container services. The measured attributes are then provided to the Monitoring System running on the Master Node. The Container Executor starts, executes and destroys containers upon requests from the Container Orchestrator. Container components are realizing the user services defined in the (container) infrastructure description submitted through the MiCADO Submitter on the Master Node.
The current implementation of MiCADO utilizes Occopus
[4], an open source multi-cloud orchestration solution as Cloud
Orchestrator that is capable of launching virtual machines on
various private (e.g. OpenStack or OpenNebula-based) or
public (e.g. Amazon Web Services or CloudSigma [
The approach taken to adopting TOSCA into the COLA Project for use with MiCADO is inherently different than the adoption approach by other frameworks and research activities described in the related work. MiCADO orchestrates at the level of the application. This primarily refers to support for container orchestration, where the assumption is that the application or its microservices have already been packed into one or more container images which are all in a ready-state. MiCADO also supports a so-called VM-only deployment, again with the assumption that the virtual machine images to be deployed contain the necessary libraries and the application is ready to accept a command or input.
For TOSCA, this meant that MiCADO could very simply define two broad types of nodes covering the two major cloud resources – one for virtual machines, and one for containers. This gave us a base node type for each, which could be extended to support a variety of cloud service providers or container runtimes.
v tosca.nodes.MiCADO.Compute § tosca.nodes.MiCADO.Compute.EC2 § tosca.nodes.MiCADO.Compute.OpenStack v tosca.nodes.MiCADO.Container.Application § tosca.nodes.MiCADO.Container.Application.Docker § tosca.nodes.MiCADO.Container.Application.rkt
The next step was to define the orchestrator within the MiCADO framework that would act on these resources in order to begin and manage their lifecycle. To this end, we leveraged the interface type defined within TOSCA. In the TOSCA specification, an interface must be defined for each node, to take responsibility for managing the lifecycle of that node. The so-called Standard interface of TOSCA uses deployment and implementation artifacts (typically in the form of shell scripts or Chef or Puppet configurations) to manage that lifecycle through four main stages: create, configure, start, and stop. These script artifacts allow for extra inputs to be fed in at deployment time, defined directly in the interfaces section of the TOSCA template.
In MiCADO, these lifecycle stages are handled by whatever respective orchestrator is responsible for that node. There is no requirement to associate those stages with a script or piece of automation code, but it is still necessary to pass information to said orchestrator, so it can correctly handle each stage of the lifecycle. Deployment artifacts become the required container or virtual machine image. Implementation artifacts are inferred to be the native configuration format belonging to that orchestrator, ready to accept custom inputs just as in the Standard TOSCA interface. To fulfil this requirement, the base interface type is extended for specific MiCADO orchestrators as below.
v tosca.interfaces.MiCADO § tosca.interfaces.MiCADO.Occopus § tosca.interfaces.MiCADO.Swarm § tosca.interfaces.MiCADO.Kubernetes The node and interface types described above are the base for describing cloud resources in all MiCADO ADTs, and by the end of project will be used in over 20 different use case application demonstrators, each with a variety of different requirements and all being able to benefit from a choice of orchestrator. The next section describes the approach taken to define a generic node type for applications in Docker containers, which can easily be reused for defining a generic node type for other resources such as virtual machine instances or other container runtimes.
A. Defining a generic container type
To define the generic set of options which the user could set in the properties section of a Docker container node type, we made a quick review of the options available and inputs required when orchestrating a Docker container with each of Swarm, Kubernetes and Mesos. Any options which were clearly related to orchestration, such as scheduling or update strategies, were dismissed and would become the available inputs for the interfaces section of each orchestrator, to further control lifecycle stages. From the remaining options, though naming and grammar varied slightly, a set of base properties was apparent and became the properties section of the Docker container node type.
To support portability, the generic properties aim to support the naming and grammar of all major orchestration platforms, so moving from one to the other requires no changes to them. The remaining options, tightly related to orchestration can still be set and modified as inputs in the interface section of the definition. To offer full support for the specific settings offered by each orchestrator, the naming and grammar of these options are still orchestrator-specific. Figure 2 provides an example of the portability and extended support offered by this approach. Here, a single simple container is defined in an ADT using Swarm-style naming, then orchestrated by Kubernetes. The generic container properties (in the upper portion of the definitions) are flexible in that they can be expressed using any supported orchestrator’s nomenclature, and then scheduled by any supported orchestrator. When selecting the orchestrator (in the lower portion of the definition), other orchestrator-specific options can be specified under inputs, so long as they match the naming and grammar of that specific orchestrator. Here, an update strategy is defined, not for the container, but for the Kubernetes workload, so orchestrator-specific grammar is required.
One application demonstrator of the MiCADO project
planned for the deployment of WordPress [
When implementation of this demonstrator began, MiCADO featured Docker Swarm as the container orchestrator. Running the NFS server in a Docker container requires elevated privileges not provided in containers by default. The Docker runtime permits elevating these privileges with the --privileged or --cap-add flags, however, as recently as the current stable release at the time of writing (18.09), Docker Swarm orchestration does not offer support for either of these options. As it was implemented, MiCADO was unable to support this particular application demonstrator. However, because of its modular design and supportive TOSCA interface, it was possible to swap out Docker Swarm for another container orchestrator to offer a solution which could support running the NFS share as part of a WordPress deployment. A review of other widely used container orchestration tools showed that both Kubernetes and Mesos Marathon supported elevating privileges in orchestrated containers. Because of its popularity and the range of other features it supports, Kubernetes won the candidature to replace Swarm as the container orchestrator.
On the implementation side, leveraging the modularity of
MiCADO was straightforward. The configuration of Docker
Swarm and its visualizer component were removed from the
Ansible [
Because of the design and modular support built into the TOSCA ADT, only orchestrator specific options had to be changed. The policies and virtual machine definitions remained identical, as their respective components were not changed for this implementation. Figure 4 shows the necessary changes to the container and volume descriptions of the ADT. The core definition of Docker containers, as expressed by their properties, did not change. The definition of the volume providing the link to the NFS share saw a change in the interfaces section, as attaching NFS shares is handled differently by the two orchestrators. The interface section of the container interfaces also saw a change. In the case of the Wordpress container, the update policy was rewritten in the style of a Kubernetes manifest. For the NFS-server container, the assignment of the IP address was rewritten for Kubernetes. The definition of the network is implicit in Kubernetes, so this was removed entirely.
Submitting this now changed ADT to a MiCADO
implementation featuring Kubernetes in place of Swarm sees a
successful deployment of the three aforementioned components
in containers on the public CloudSigma cloud, as seen in the
Kubernetes dashboard capture shown in the top half of Figure
5. The automated scaling functionality of MiCADO is also
preserved, as can be seen in the Grafana [
This paper presented a novel approach to authoring TOSCA templates for the reuse of generic cloud components across different orchestration tools. A proof-of-concept demonstrated that, when requirements of an application exceeded the capabilities of a modular framework, the definition of that application could be reused and successfully deployed on the same framework featuring a different, capable component.
As the development of MiCADO continues, so does the work in testing and supporting its modularity with TOSCA and Application Description Templates. Supporting modular cloud orchestration using TOSCA ADTs is on the horizon, with the aim to provide high levels of portability of a containerized application across a wide variety of cloud service providers.
This work was funded by the COLA Cloud Orchestration at the level of Applications Project No. 731574.
Available: