There continues to be a trend towards using the power of cloud computing to tackle inherently large and complicated problem domains. Validating domain-intensive cloud applications presents a significant challenge because of the complexity of both the problem domain and the underlying cloud platform. In this paper, we describe an approach that leverages model-driven engineering to improve testing domain-intensive cloud applications. Our approach combines a set of abstract test commands with various domain and configuration models to define a domain-specific testing language. We have developed a prototype of our approach that provides language editing and platform configuration tools to aid test specification, execution and debugging.
Introduction Validating software-as-a-service applications is difficult due to the large size of the problem domain, coupled with the complexity of the underlying cloud platform. Adequate functional testing is heavily dependent on domain expertise from each product area, and typically requires extensive data setup. Since the software is delivered as a service over the Internet, functional UI testing must be performed using different browsers to ensure a good user experience. In addition, engineers need to be able to set up tests to run on specific configurations of the underlying cloud infrastructure.
Model-driven engineering (MDE) seeks to simplify software development by
raising the level of abstraction through domain modeling, while promoting
communication between groups working on the same system [
In this paper, we describe an approach that leverages MDE to improve the
specification, execution, and debugging of functional tests for domain-intensive
cloud applications. Our approach is the result of investigating new and innovative
ways to test UltiPro, a comprehensive cloud-based human capital management
(HCM) solution [
The major contributions of this research paper are as follows: (1) describes a novel approach that integrates various domain and configuration models into a test case specification language for cloud applications; (2) presents the design of a prototype used to demonstrate the feasibility of the approach; and (3) discusses our experience developing the prototype, focusing on the lessons learned. The rest of this paper is organized as follows: the next section motivates the research problem. Section 3 describes our domain-specific test case specification approach for cloud applications. Section 4 presents a prototype that implements the proposed approach. Section 5 discusses the lessons learned from building the prototype. Section 6 is the related work and Section 7 concludes the paper. 2
Our research has been motivated by the challenges faced when testing UltiPro
[
The large size and complexity of the problem domain makes testing the functionality of UltiPro challenging. Individual product areas (e.g., recruitment, payroll) encompass so many features that each area could be considered as a product itself. Adequately testing UltiPro requires each product area to be validated, which is impossible without domain expertise. Although each product area is large, UltiPro has been designed and developed as a unified solution which seamlessly integrates all aspects of HCM. Validation of the overall product therefore relies heavily on the collaboration of domain experts across all product areas. This ensures that changes to one product area does not have an adverse effect on other product areas.
Testing UltiPro is further complicated because of its development and delivery as a cloud application service. Cloud application services are hosted on complex, distributed infrastructures with multiple servers and architectural layers that extend from the underlying network up to the web-based user interface. To ensure a good user experience, functional UI testing must be performed using different web browsers. Other non-functional factors such as high performance and security requirements also make it difficult to test cloud applications. However, this paper limits the scope of the testing problem for cloud applications to functional UI and platform compatibility testing. 3
Our approach defines a test specification language that can be used to develop automated tests for a particular application domain. As shown in Figure 1, we leverage abstract test commands, domain and platform models, and test automation frameworks for the purpose of creating a domain-specific language (DSL) for testing cloud applications. The DSL allows us to provide test case editing, execution, and debugging tools tailored for domain experts, test engineers, and end users. Abstract tests defined using the DSL are translated into executable scripts that run on the underlying testing tools and frameworks. For the remainder of this section, we describe the various test abstractions and models used in our approach. Transformation of abstract tests into tool-specific testing scripts is discussed as part of the prototype design in Section 4.
!"#$%&'$()*#$( +,--&./#
2+(/-'% 4-99*':" 2""#$(/-'% 4-99*':" )*#$(+&#*(01*'232'&$2,.(4&.56&5*(7/2$,% )*#$(+&#*(01*'232'&$2,.(4&.56&5*
;;%.-9*/'<68#+/)/+%== )*#$(!6$,-&$2,.(),,8#(&./(9%&-*:,%;# !"#$%&'(#$)*+#
0*1#% 23"($*+(/-'"
4-'($-5 23"($*+(/-'"
<,/*8# 4-')/17$*(/-' 05*()-$9 ,*+$-%.#)/'/(/-'"
.*(*%6#(78 >'?/$-'9#'( !"#$%&'(#$*+(/-' At the core of the language is a set of abstract test commands. There are two types of commands: Action Commands and Assertion Commands (left of Figure 1). Action commands apply inputs that exercise the system under test (SUT). This includes stimulating UI controls such as textboxes, dropdowns, and buttons, as well as database-related actions. On the other hand, assertion commands perform UI and database checks to verify the behavior of the SUT. Table 1 describes some of the key test commands defined under our approach. 3.2
Application Domain Models Domain concepts are introduced into our testing language through two types of models: User Interface and Macro Definitions (right of Figure 1). ! Command Description
Blur Loses focus of an element
Clear Empties the value of an element Actions CMloicukse Over PHroevsseerss tahnedmreolueasesepsominoteurseovbeurttaonnewlehmileenhtovering over an element Mouse Out Of Move the mouse pointer from within the hover area of an element Set Assigns a value to an element Is Checks if the value of an element equals a given value Is Like Checks if the value of an element contains a given value as a substring Is Visible Checks if an element is visible Assertions Is Enabled Checks whether an element used for input is enabled Exists Checks if an element is present Has Options Checks if a dropdown provides a specified list of values
Has Number Of Options Checks if the size of a dropdown list is equal to a specified value ! Table 1. Web UI Abstract Test Commands "#$%&'!(')!"**+,$%&'!-&..(')*!&'!/+0!12!-&'$,&3*!456$$&'*7!-8+#90&:+*7!;,&<)&='*7!>,%)*7!?+:$0&:+*@! ! User Interface This model is a generalization of the user interface of the SUT. F!or exampCleo,mimnantdhe casDeesocfripUtilotniPro the UI model consists of abstractions representing itsCAwldedearbRRooplllblabagcakceks aSRneedtmaocrveoestnaotrreersoptoolirsne.tpoToninhttheeosndeathtapebadaasgteaebaasend control objects encapsulate theAcCtioSnSs seRleolclbtaocrks used Rteostoirdeepnretviifoyus wdaetabbaeselevmalueensts in the HTML source document tree. Auto mExaectuetde tests tRhuenns ardeaftearbaesnecsceritphtese abstractions instead of the implemendetaMilastc,hwhich mCahkecekss wtehsetthseretwaosideartasets amreaeqinuatlain as the application changes t a!Atsisoerntions to [8",#9$%&].'!(F'u)!r"t*h*+e,$r%m&'!o-r&e.,.p('a)g*!e&'a!/n+d0!1c2!o-n&'t$r,o&3l*!4o56b$$j&e'c*t7!s-8a+#r9e0&n:+a*m7!;,e&d<)&u=s'in*7g!>,%d)*o7!m?+a:$i0n&-:s+p*@!ecific concepts. For example, a grid control used for entering pay data would be named Pa!yDataEntryGrid. Using such terms allows domain experts and end users to e a!sily identify and specify various aspects of the SUT.
! M! acros A macro in computer science is a pattern that specifies how a sequence of inputs is mapped to a replacement input sequence. Macros are often used to! make programming tasks less repetitive and less error-prone. Our approach le!verages the benefits of macros to improve test specification. Testers can define frequently used test setup, input, or assertion command sequences, and store th!em in a central location. These macros are then named using domain-specific te!rms, and integrated into our testing language. Similar to our abstract test commands, test macros can target user or database interactions. ! ! 3.3 !
Configuration Models S!everal abstractions for configuring the underlying platform and environment of the SUT are integrated into our language. These include abstractions for: S!erver Environment Configuration – Application servers, web servers, database se!rvers, reporting servers that make up the test environment; OS/Platform Configuration – Operating systems on which to test the desktop and mobile versions of the cloud application; Web Browser Configuration – Clients to use during cross-browser compatibility testing, e.g., Chrome, Firefox, Internet Explorer or a combination thereof; and Test Harness Configuration – Modes and settings that allow users to tweak aspects of test execution including timing characteristics, logging, among others. !"##$%&'''''! !!"#$%&'(''')*+,-*.(!/#0!"*1$&++!2(*.#$(! !!3#.4&$'!!!5,&001!6*0.,*7&8!3-*9!:*0-&! !!:&0;,7'!!!<=>?8!"*1$&++@AB! ()*+$%$,-./0!!'' ''C+.,"$&!1+,-2%.' '' !),"3'''''' !!D,E(0!F!6(.#%!*!"*1$&++!G<<<H! !!30-!+*#0I4!1+,-2%.' !!30-!+&7,0!*'!*!"*1$&++!3-9,0,'.$*.&$! !!!!J1!'(..,07!.4(!C'($=*9(?(K.J&K!.&!L*-9,0L8! !!!!!!!!!!!!!!!!!!!"*''M&$-?(K.J&K!.&!L#+.,L! !!!!30-!I+,IN,07!&0!.4(!O&7,0J#..&0! ! 4)0,0' ''6I(0*$,&P!"*1$&++!64&#+-!/#0!.&!:&9%+(.,&0' !!D,E(0!F!=*E,7*.(!.&!.4(!"*1$&++!QE($E,(M!! !!R4(0!F!I+,IN!.4(!6.*$."*1$&++J#..&0! !!?4(0!.4(!"$&I(''6.*.#'O*S(+!,'!L:&9%+(.(-<L! !!30-!.4(!:$(*.(J*.I4('J#..&0!,'!(0*S+(-! ! Fig!. 2. Example Test Case Specification ! ! 3.4
!
Illustrative Example Figure 2 presents an example test case specification defined using our approach. The example test consists of four main blocks: Summary, Declarations, Setup, and Tests. The summary block (Lines 1-4) holds meta information about the test, which includes a purpose, authors, and various configurations. In Line 4, .NET is a configuration that runs tests against the UltiPro desktop web application using three browsers for compatiblity testing. On the same line, Payroll-14 sets up the test to run on a server environment configured for payroll processing.
Applications and data that will be used throughout the test must appear in the declarations block (Lines 6-7). In this block, automatic word completion popups (i.e., intelli-sense) is filtered to a list of available application models and database types. Line 7 of Figure 2 specifies that the UltiPro application model will be referenced whenever the name UltiPro appears in the test.
The setup block (Lines 9-15) contains a set of preconditions for the test, written using a behavioral-driven development (BDD) style syntax. Line 10 illustrates the use of a database macro to Setup a Payroll. Note that elipses are used to mask the actual parameters passed into the macro. Inline, users can declare high-level test steps (Line 12), and define how those steps are implemented as actions or assertions on the application model (Lines 13-15).
Test cases appear within the block named Tests (Lines 17-22). Line 18 provides a name for the single test in the example, while Line 19 demonstrates the usage of a UI navigation macro. The click action command is illustrated in Line 20, while Lines 21 and 22 show the Is and Is Enabled assertion commands. 4
This section presents the setup, design and implementation of the Legend
prototype, which was developed to demonstrate the feasibility of the proposed
approach. Legend primarily consists of tools for authoring, executing and
debugging tests written to validate UltiPro [
Legend has been developed in C# as a Visual Studio (VS) extension. The VS 2010 SDK provides components for extending the VS Editor with a custom language service. The Legend language service supports many of the VS SDK features including syntax coloring, error highlighting, intelli-sense, outlining, tooltips, and debugging. Integration of application and configuration models using domain concepts is a novel feature of Legend that separates it from other test specification languages and tools.
The underlying testing framework used to run Legend tests is an in-house
tool called Echo [
Editing Tools Figure 3 provides a UML package diagram showing the design of Legend DSL Editor. As shown in Figure 3, the editor is comprised of three major packages: EditorExtension, ApplicationModelIntegration and CodeGeneration. Key classes from each package, along with their interdependency relationships are also shown in the diagram.
The EditorExtension subsystem (top of Figure 3) contains the components
that implement token colorization, syntax checking, block outlining, and
intellisense. This subsystem is the main point of interaction between the Visual
Studio editor and the Legend code extensions. The classes with the stereotypes
Providers, Controllers, Taggers, and Sources are derived from the Visual
Studio SDK, and directly interact with the .NET Managed Extensibility
Framework (MEF) [
Integration with the application model is achieved via the ApplicationModelIntegration subsystem (bottom-left of Figure 3). It contains two types of classes: ModelProviders and Models. The ModelProvider classes use reflection to read the page objects, control objects, macros and elements that make up the Models. It is also responsible for filtering intelli-sense on the model, given a specific test context. For example, at the point where a test declares access to a particular web page, the ModelProvider scopes the word completion picklist for elements to include only elements that appear on that page.
Lastly, the CodeGeneration subsystem (bottom-right of Figure 3) provides logic for translating the abstract tests written in Legend into code executable <<Providers>> TokenTaggerProvider <<Taggers>>
TokenTagger <<Providers>> CompletionSourceProvider
<<Sources>>
CompletionSource <<Services>> LanguageService
Legend.EditorExtension
<<Augmentors>> TokenTagAugmentor <<Augmentors>> <<Interface>>
ICompletionAugmentor <<ModelProviders>>
<<Interface>> IApplicationModelProvider +GetApplicationModels() <<Models>>
Element <<ModelProviders>> ApplicationModelProvider <<Models>>
PageObject <<Models>> AutomationContext <<Models>> MacroCategory <<Models>>
Macro <<Models>> ControlObject
<<Augmentors>> LanguageAugmentation <<Services>> <<Interface>>
ILanguageService +GetCurrentCompletions() +GetTags() +UpdateSnapshotState() <<Generators>>
EchoGenerator +GenerateScript() +GenerateAssembly()
<<Generators>> EchoAssemblyBuilder <<Generators>> EchoTestBuilder
<<Parsers>> EchoCommandParser Legend.ApplicationModelIntegration Legend.CodeGeneration by the Echo framework. In order to support debugging at multiple levels of abstraction, the EchoGenerator has two modes of generation: Script – generates test scripts written in the Echo syntax; and Assembly – generates a Common Intermediate Language (CIL) representation of the Echo test. Our prototype maps the CIL to the domain-specific test steps. A test can therefore be executed and debugged at the level of the domain-specific test language, or the script language of the underlying Echo testing framework.
Configuration Tools Abstractions for the test environment and platform configurations are implemented in two distinct XML files called Environments.xml and Parameters.xml. The environments XML contains a list of all the test environments that are connected to a tool that can automatically request the latest UltiPro build. The file is auto-generated and populated with the unique identifiers used by each team to refer to their test environments. Information related to the specific network and database servers, along with any credentials for authentication are also stored in the Environments.xml file.
The Parameters.xml allows users to specify a range of configurations ranging from desktop web settings such as browsers and languages, to mobile web settings such as device screen size and orientation. Although each team can create their own configurations, there are a set of fixed configurations that are available for use across all teams during regression testing. The data in both the environments and parameters XML files are passed directly to the Echo testing framework prior to test execution. 5
A key factor that contributed to the successful development of the prototype was having robust, highly extensible and configurable underlying testing frameworks. Echo and Selenium provided the implementations to support many of the abstractions described in the approach. Building the application models required a collaborative effort among developers, testers and domain experts. Developers would create the page objects and control objects, while testers and domain experts made sure they were named and exposed appropriately for testing. Macro creation was primarily done by testers and domain experts, with occasional assistance from the developers if necessary.
One of the more complex aspects of the prototype implementation was the module for keeping track of the test context in order to filter intelli-sense. This required the creation of a state-based rules engine to allow us to perform different editor actions based on previously specified lines in the test. Although challenging to implement, this feature was necessary to provide meaningful intelli-sense that guides testers during test creation. In other words, testers are only presented with commands, model elements, and other keyword suggestions if they are applicable in the current context.
Based on initial responses to prototype demonstrations, a major benefit of Legend is the ease with which test cases can be specified and reviewed by nontechnical users. The tool can therefore be leveraged by domain experts and end users during acceptance testing. It also allows these stakeholders to assist in debugging issues using a language they understand, and without being concerned with the low-level implementation details of the test automation. Since tests are specified in an english-like syntax, using Legend could reduce or eliminate the need to maintain a separate inventory of test documentation. However, further evidence through a case study or empirical evaluation is needed to be able to fully validate these claims.
A limitation of the current prototype is the lack of an externalized point of extension for the test commands and their syntax. Since domain experts, testers and developers from several teams will be defining new model elements as the application evolves, we need to provide a mechanism that allows new page or control-specific commands to be easily added to the language. Web UI elements with dynamically generated identifiers are also not supported by the prototype, or the underlying Echo testing framework, but are planned for future releases. 6
Although the use of domain modeling to support software engineering is not
new, only a few researchers have leveraged MDE and DSLs to support software
testing [
Hernandez et al. [
Some researchers have proposed model-driven approaches that aid the
development of high-performance and cloud computing systems [
There are several general purpose behavioral-driven development (BDD)
testing tools that help to tie acceptance tests to business requirements [
Conclusion The work in this paper presented an approach that applied model-driven engineering to the development of a domain-specific test case specification language. Our approach is general in the sense that it can be applied to any domain, but in terms of technologies we focus on web-based applications which are deployed on cloud computing platforms. We have implemented a prototype of the proposed approach for a cloud-based human capital management solution. Developing the prototype gave us first-hand experience on some of the benefits and challenges associated with creating a DSL for testing UltiPro. Feedback from interactive prototype demonstrations has been positive. Our next steps are to develop a full implementation of Legend, and perform a case study using data from UltiPro. Acknowledgments. The authors would like to thank Jorge Martinez, Michael Mattera, and members of the Virtual Team at Ultimate Software for their contributions to this work. We also give thanks to the judges and participants of the Summer 2012 Ultimate Software 48 Hours Project for their valuable feedback. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors, and do not necessarily reflect the views of the Ultimate Software Group, Inc.