Facilitating Agile Prototyping of Cloud Applications — A Model-Based Approach Ta’id Holmes Infrastructure Cloud, Deutsche Telekom Technik GmbH Darmstadt, Germany t.holmes@telekom.de Abstract—Modern cloud applications, generally, demand com- blocks (ABBs) with software building blocks (SBBs) (cf. [3]). plex service topologies, e.g., for meeting scalability, maintenance, The service topology, that comprises latter building blocks, may or security requirements. Thus, often, it is desirable to increase be modeled by a team of experts. Some services are available the complexity of service topologies in cloud applications. The re- quired changes, however, may constitute a burden for improving off the shelf, while others need to be implemented by service cloud applications. Changes, overall, are undertaken frequently engineers. Deployment services automate the provisioning of in prototyping or when adopting agile development. Their costs different stages such as for testing or pre-production. Finally, a correlate with the number of people involved. For facilitating ag- continuous integration service executes various tests that have ile prototyping of cloud applications this demonstration presents been implemented by test developers. a model-based approach incorporating different roles. Using, integrating with, and building on top of open-source projects, it In case of iterative development all of these roles repeatably comprises domain-specific language editors and showcases their need to coordinate their activities. If a cloud architect would use and the realized automation fostering iterative development. like a particular service to be decoupled from another because Index Terms—automation, agile, application, cloud, DSL, of scalability issues the alignment of building blocks may need model-based, prototyping, provisioning, service topology adjustment, the model reflecting the service topology needs to be changed, tests may have to be adapted, and service I. I NTRODUCTION implementations have to be modified. For various reasons, modern cloud applications are composed For lowering the burden of improving the cloud application of increasingly complex service topologies (cf. [1]). Factors accordingly the overall turnaround time shall be kept as short that contribute to the complexity of service topologies are the as possible in such cases. With a multitude of people and roles ability to scale out services for meeting varying user loads, involved, agile development of complex cloud applications security requirements that demand the separation of application is challenging: Most of all, coordination needs have to be logic as well as of data, and maintenance considerations that addressed. Next, development of individual services needs foster distribution of self-contained services for decoupling to be facilitated in a sense that integration into the service their lifecycle. topology is eased. Finally, overall automation is required. For expressing and capturing service topologies, the OASIS Tackling these issues, the demonstration proposes a model- Topology and Orchestration Specification for Cloud Appli- based approach and presents a toolset comprising domain- cations (TOSCA) standard 1 provides a metamodel. When specific language (DSL) editors for facilitating agile prototyping describing a cloud application in terms of a metamodel and of cloud applications. Showcasing an overall example, it when following a model-driven engineering (MDE) approach, combines two former contributions: Addressing coordination provisioning can be automated (cf. [2]). Yet, development of needs, a descriptive DSL [4] permits to define roles, artifacts, a cloud application remains difficult as its service topology and services. The alignment of ABBs and SBBs can be needs to be modeled and the various services implemented. In undertaken by an enterprise architect using another DSL which an agile practice, the architecture and implementation of cloud is also used for describing the provisioning of the cloud applications usually experiences early and frequent changes application [5]. Using a running example, this demonstration such as refactorings. For example, the topology may be changed describes the overall process, artifacts, and tools involved. in a way that requires alignment of the model, e.g., when Abstracting from infrastructure as a service (IaaS) providers, services are split into distributed entities. At the same time, the approach automates various tasks and enables developers the respective services need to be adapted as well. to incrementally deploy cloud services facilitating iterative Multiple roles with particular responsibilities are involved in prototyping of cloud applications. the engineering process, such as architects, test developers, and The remainder is structured as follows: Section II introduces service engineers – each one having a different set of expertise. the running example of the demonstration by describing its For realizing a new cloud application an enterprise architect context and by characterizing related problems as a further mo- performs a functional analysis and aligns architectural building tivation. The approach using the running example is presented in Section III. The work is compared to some related work in 1 http://docs.oasis-open.org/tosca/TOSCA/v1.0/TOSCA-v1.0.pdf Section IV and Section V concludes. II. RUNNING E XAMPLE : C ONTEXT AND M OTIVATION Sensor Broker Converter Business Dashboard Functional Intelligence A Architecture Let us now consider an industrial machine-to-machine Mapping (M2M) context in which a new business idea arose. As part of Sensor Messaging Analytics Database Web Web A Abstract SW- Platform Server Server Application Architecture the value proposition it envisages the correlation of sensor data Mapping with other data and the visualization of results in a dashboard. In PoC MQTT PoC PostgreSQL Apache PoC Concrete SW- this scenario M2M devices emit data from sensors to an M2M Part 1 B Broker C Part 2 B C HTTP C Part 3 B Architecture gateway. A backend server receives and processes aggregated Mapping 1 Abstract sensor data from these gateways. For this, the data is normalized Hosting Unit Hosting Unit Hosting Unit Hosting Unit Infrastructure and correlated with user data and data from an external data (Provisioning DSL) Transformation source. Results are stored in a database and visualized in a web application. Figure 1 depicts a rudimentary server landscape 2 Concrete Infrastructure for the M2M scenario. DEV (EC2 IaaS Model) TEST Various roles participate in the engineering: among them are service and web engineers that implement different parts of a minimum viable product (MVP) (see below) as well as Figure 2. From Design to Deployment of the Cloud Application architects that describe the functional architecture. Publish/Subscribe Dashboard MQTT Business via hosting units. Finally the development of the various Intelligence parts of the MVP – as performed by software developers (Role B) and configuration management (CM) experts (Role C) Broker Analytics Database Web Web – requires coordination between the roles. Finally their services Sensors Converters Server Browser have to be deployed (realized with Transformations 1 and 2). As indicated at the bottom, this may occur for multiple stages Figure 1. Server Landscape of a Machine-to-Machine Scenario of the engineering lifecycle (e.g., DEV or TEST). A lack of automation renders the process – consisting of a multitude of steps – time-consuming and costly. As a result, Following the idea of the lean startup philosophy [6] the iterative cycles involving the reprovisioning of (parts of) the value proposition of the business idea is to be tested with service topology may be avoided. Decreasing agility, this may potential customers. For this, prototyping constitutes a suitable proof problematic at an early phase of product development approach for the (rapid) development of a MVP (cf. [7]) and such as when developing a MVP. Besides the loss of flexibility for gaining findings such as customer requirements. (e.g., as required when performing adjustments), the complexity In situations in which products depend on several cloud of the overall process may render effective participation of services a platform as a service (PaaS) may provide a suitable architects problematic. basis for the development and deployment of software as a service (SaaS). It needs to cover the requirements of the product III. D OMAIN -S PECIFIC L ANGUAGES FOR FACILITATING in terms of deployed technologies and available cloud services. THE E NGINEERING OF C LOUD A PPLICATIONS For the development of a MVP the decision to deploy a (certain) PaaS may be an early decision to take. Indeed, one of the For addressing the issues mentioned in the introduction principles of lean management is deferring decisions for being and as further elaborated in the previous section, a model- able to also consider new information. based approach is proposed. That is, separation of concerns As an alternative and as exercised in the demonstration, the is realized between architecture, functionality, and technology entire cloud stack from IaaS to SaaS – realizing the MVP – platforms while automating provisioning and deployment in a needs to be described for cloud provisioning and deployed. role-based development process. After giving an overview, the This is particularly interesting when customized cloud stacks use of the DSLs is illustrated by means of the running example. are preferred over a uniform cloud stack or a PaaS. At the same The demonstrator entirely relies on open source software: The 2 time the specification and deployment of a customized cloud Eclipse Modeling Framework (EMF) was chosen with Xtext stack, e.g., in TOSCA without further tool support, requires for defining the DSLs and for realizing respective editors expert knowledge. This burden may prevent a project to opt and Xtend for implementing the model transformations. At 3 for a tailored cloud setup and hinder lean development. runtime and besides the resulting DSL editors, Git is used For realizing the MVP the functional architecture of the as version control system (VCS), and Puppet 4 for the CM. 5 cloud application needs to become manifested in tangible Finally, OpenStack serves as an IaaS solution. cloud services and resources. The mapping from ABBs to 2 http://eclipse.org/modeling/emf SBBs – performed by architects (indicated with Role A) – is 3 http://git-scm.com shown in the top three levels of Figure 2. Describing the cloud 4 http://puppetlabs.com provisioning, the services are associated with server instances 5 http://openstack.org Provisioning DSL Program Software CM Server W7 urEnvironment project M2M_PoC CM Modules • install  PoC for the costCenter "123456789" Version Control System M2M Scenario platform profile ThreeStage Workspaces • deploy SW hostingUnit sensor stage DEV TEST CM Manifests Coordination Tools Platform service Publisher  Mosquitto hostingUnit broker Cloud-Init  PostgreSQL service Broker Coordination DSL Program Code  Apache HTTP artifact PoC_artifact1 Model Transformation role "SWDeveloper" Infrastructure produces PoC_artifact1 Code IaaS Provider  Members Abstract IaaS Model Generation  Volumes service PoC_part1 Model Transformation dependsOn PoC_artifact1  set up infrastructure  Security Groups  install CM clients  Server EC2 IaaS Model  run CM agents Cloud Application Figure 3. Overview of the Model-Based Approach for the Development and Continuous Deployment of Cloud Applications W7 urEnvironment catalogue A. Approach Overview namespace M2M_PoC serviceDef Publisher realizedBy PoC_part1 In addition to Figure 2, a technical overview of the model- implies services MosquittoClient PyXB based realization is depicted in Figure 3. The cloud provisioning serviceDef Broker implies services Mosquitto is described using a DSL as shown in the upper left part of serviceDef Analytics realizedBy PoC_part2 implies services MosquittoClient PyXB SQLAlchemy the figure. Through a sequence of model transformations an serviceDef Database IaaS consumer is generated that realizes the provisioning of implies services PostgreSQL serviceDef Report realizedBy PoC_part3 cloud infrastructure services. Platform and software services implies services SQLAlchemy ApacheWSGI as referenced in the DSL program are provisioned using a CM system. CM modules are looked up for such services and a Figure 4. Mapping Architectural to Software Building Blocks CM server is instructed accordingly while server instances are provisioned with CM clients. A coordination DSL program declarativly describes roles, this, services are grouped in hostingUnits. Through model artifacts, services, and dependencies between these. Coordi- transformations and automation, the latter are mapped to nation tools are generated that interplay with a VCS and server instances or clusters and appropriate IaaS clients are the workspaces of stakeholders. For developed services, CM generated. Please note, that the services reference the abstract modules are prepared in addition and built for further simpli- serviceDefs from Figure 4. This and further relationships fying the overall engineering process while profiting from the (implies) are exploited for automating the provisioning. functionality and automation as realized by the CM system. The DSL facilitates specification of custom cloud stacks as required for different stages of the engineering lifecycle such B. Business Idea and Functional Analysis as development (DEV), testing (TEST), or production (PROD). Starting from the business idea from Section II, archi- Choosing a profile the provisioning is performed similarly tects conduct a functional analysis. The resulting functional for each of the defined stages. If desired this is done in architecture comprises sensors, a broker, converters, different cloud regions (also defined in the respective profile). a business intelligence (BI), and a dashboard as If not bound to certain stages (e.g., sensor data generators shown in Figure 2. Resulting ABBs are sensors, a for development and test) server instances (e.g., broker) are messaging platform, an analytics engine, a database provisioned for all stages (e.g., in all the related cloud regions). server, a web server, and a web application. Using a DSL editor and profiting from syntax highlighting, code D. Formalizing Dependencies of Services, Artifacts, and Roles completion, and scoping as shown in Figure 4, the ABBs are associated with SBBs (i.e., Mosquitto as a MQ Telemetry A declarative DSL allows to enumerate roles participat- Transport (MQTT) broker, a PostgreSQL server, an Apache ing in the engineering process, artifacts, services, and their HTTP server, and three different parts of a proof of concept dependencies in a generic way. From these, coordination (PoC) implementation). The abstract building blocks are so tools are generated (cf. [4]). Integrated into a VCS these associated with concrete building blocks to be deployed and may send out notifications when artifacts are available and are used (i.e., referenced) for the provisioning as shown later. synchronize workspaces accordingly. In addition to supporting the coordination, automation tools are derived from the DSL C. Domain-Specific Language for Cloud Provisioning programs for preparing the deployment of developed services The DSL program in the upper left part of Figure 3 to the respective hostingUnits and for pushing and applying shows an excerpt for defining the cloud provisioning. For incremental changes through Puppet. Services that are devel- oped such as the PoC parts are referenced by serviceDefs provisioning permits to define rudimentary service topologies. following the realizedBy keyword (see Figure 4). While they can be mapped to TOSCA the DSL aims particularly at giving stakeholders access to the modeling. In fact, a E. Integration with Configuration Management Software provisioning can be described in a couple of lines and can be Higher-level cloud services depending on IaaS are provi- changed easily. Instead of the DSL shown the approach can use sioned through a CM system. For this, Puppet modules are any modeling as long as it does not slow down development looked up for services using name-based matching. Using and iteration cycles. User studies have to be carried out in this the realizedBy keyword Puppet modules are automatically regard in order to answer the question which modeling tools synchronized with required software artifacts using the VCS. for service topologies are best for the development of cloud Yet, module manifests need to be supplied by Puppet experts. applications as required in prototyping. At execution time, the approach integrates with Puppet by V. C ONCLUSION generating manifest files and cloud-init 6 configs (passed as user-data when launching server instances). This way Agile development of cloud applications can be facilitated the different engineers can work independently using their using a model-based approach, e.g., using textual DSLs as workspace focusing on their actual task while the packaging presented. For this a high degree of automation is required and deployment is automated. beyond provisioning. Also, as many stakeholders are involved in the engineering, it is crucial to harmonize their collaboration F. Iterative Development and Continuous Deployment and to provide them with means for participation. This is real- When executing the generated IaaS consumer all the cloud ized by exploiting formalized dependencies and by providing services (see right box of Figure 3) – starting from IaaS (i.e., tailored DSL editors while abstracting from technologies such security groups, volumes, and instances) and PaaS (i.e., Apache as CM and IaaS solutions. HTTP server, Mosquitto, and PostgreSQL) to SaaS (i.e., PoC For wide applicability, the approach directly builds on top Parts 1–3) – are provisioned in an interplay with Puppet and of IaaS and does not rely on a development PaaS. Yet, in thus the cloud application is deployed. If work is performed in case the code generator is adapted, the presented work can the VCS, e.g., when modifying a particular service, changes build on different application programming interfaces (APIs) or can be applied incrementally using Puppet behind the scenes other technologies both for the provisioning and the CM. The facilitating continuous deployment. Changes to the service approach was successfully adopted for the rapid development topology are more critical: While in many cases they are of several demonstrators and a PoC in an industrial context. respected by the demonstrator, a reprovisioning may become R EFERENCES necessary, e.g., when renaming occurs. Please note that while [1] V. Andrikopoulos, T. Binz, F. Leymann, and S. Strauch, “How to Adapt the latter takes longer until all services are provisioned it is still Applications for the Cloud Environment,” in Computing. Springer, 2013, fully automated and may be thus acceptable for prototyping. vol. 95, pp. 493–535. [2] N. Ferry, H. Song, A. Rossini, F. Chauvel, and A. Solberg, “CloudMF: IV. R ELATED W ORK Applying MDE to Tame the Complexity of Managing Multi-cloud Applications,” in IEEE/ACM 7th International Conference on Utility Rapid application development (RAD) (cf. [8]) can help to and Cloud Computing (UCC). IEEE, Dec 2014, pp. 269–277. obtain results in a timely and efficient manner. For instance, [3] M. Iacob, D. Jonkers, H. Quartel, H. Franken, and H. van den Berg, Delivering Enterprise Architecture with TOGAF and ARCHIMATE. various RAD frameworks exist (cf. [9]) that are tailored towards Enschede: BIZZdesign, 2012. web applications and that can be combined with the approach. [4] T. Holmes, “Facilitating Development and Provisioning of Service Topolo- For the development of cloud applications a development PaaS gies through Domain-Specific Languages,” in 18th IEEE International Enterprise Distributed Object Computing Conference Workshops and such as the Google App Engine 7 may provide ready to use Demonstrations, G. Grossmann, S. Hallé, D. Karastoyanova, M. Reichert, (technology) stacks of services while easing the development and S. Rinderle-Ma, Eds. IEEE, Sep. 2014, pp. 422–425. of SaaS. In contrast this work permits to define and work with [5] ——, “Automated Provisioning of Customized Cloud Service Stacks using Domain-Specific Languages,” in 2nd International Workshop on customized cloud stacks. Realizing separation of concerns it Model-Driven Engineering on and for the Cloud, vol. 1242. CEUR- incorporates different roles in the engineering using tailored WS.org, Sep. 2014, pp. 46–55. DSLs and coordinates workspaces. As it directly operates [6] E. Ries, The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, on IaaS deployments in an interplay with CM systems it is 2011. independent of a PaaS and certain (technology) stacks. [7] M. Poppendieck and M. A. Cusumano, “Lean software development: A As mentioned, TOSCA provides a metamodel for service tutorial,” IEEE Software, vol. 29, no. 5, pp. 26–32, 2012. [8] P. Beynon-Davies, C. Carne, H. Mackay, and D. Tudhope, “Rapid topologies. As a standard, it constitutes a desirable basis for application development (RAD): An empirical review,” European Journal an implementation of the approach presented. Similar to the of Information Systems, no. 8, pp. 211–223, 1999. presented work TOSCA can be combined with CM systems [9] I. Vosloo and D. G. Kourie, “Server-centric web frameworks: An overview,” ACM Comput. Surv., vol. 40, no. 2, 2008. (cf. [10]). The DSL building on top of Amazon Elastic Compute [10] J. Wettinger, M. Behrendt, T. Binz, U. Breitenbücher, G. Breiter, Cloud (EC2) 8 concepts and used in this work for describing the F. Leymann, S. Moser, I. Schwertle, and T. Spatzier, “Integrating Configuration Management with Model-Driven Cloud Management based 6 http://launchpad.net/cloud-init on TOSCA,” in 3rd International Conference on Cloud Computing 7 http://cloud.google.com/appengine/ and Services Science, F. Desprez, D. Ferguson, E. Hadar, F. Leymann, 8 http://awsdocs.s3.amazonaws.com/EC2/latest/ec2-api.pdf M. Jarke, and M. Helfert, Eds. SciTePress, 2013, pp. 437–446.