(INVITED TALK) AGILITY VS. MODEL-BASED TESTING: A FAIR PLAY? Baris Güldali, Michael Mlynarski Software Quality Lab (s-lab), University of Paderborn Warburger Str. 100, 33098 Paderborn/Germany {bguldali,mmlynarski}@s-lab.upb.de ABSTRACT In contrast, agile manifesto [1] proposes a “light- Agile manifesto defines principles for a light-weight weight” development process where (1) individuals software development process aiming at an improved and interactions are favored over processes and tools, customer satisfaction. Automated testing plays an (2) working software is favored over comprehensive important role in fulfilling these principles, because it documentation, (3) customer collaboration is favored enables efficient execution of test scripts for checking over contract negotiation and (4) responding to the quality of delivered software. However, the change is favored over following a plan [1]. However, implementation and the maintenance of the test scripts in the practice, these principles are likely to be misin- can be very tedious and error-prone. In order to deal terpreted such that developers often neglect docu- with that, model-based testing extends the automated menting customer requirements properly. Frequently, test execution by test design and test implementation. this leads to chaos in the development process and to Thus, model-based testing can speed up the test conflicts during the delivery and acceptance. Thus, it automation and improve the maintenance of test is a challenge to follow the principles of agile mani- scripts. Nevertheless, introducing model-based testing festo and thereby not to lose sight of the proper docu- requires some initial and some continual efforts, like mentation and communication of customer needs and creating test models, buying or developing tools, etc. of the efficient and effective development. In this talk, we will discuss how model-based testing We believe that, model-based techniques can help can support agile development without conflicting in dealing with these challenges. In the rest of paper, with the principles of agile manifesto. we will discuss how agility and model-based paradigm fits together. Thereby, we will mainly focus on the Index Terms - Agile manifesto, Automated testing, integration of model-based testing in agile develop- Model-based Testing ment process as an enabling technology for the prin- ciples of the agile manifesto. 1. INTRODUCTION 2. AGILE MANIFESTO As the complexity of software rises, novel software development techniques are required in order to cope In 2001 seventeen software experts, who have with the technical and the organizational challenges in introduced well-known agile methods (e.g. Scrum, the development process. Model-based software Test driven Development (TDD), Extreme development (MBSD) proposes using abstract models Programming (XP) etc.) have defined common for better communication, for maintainable software principles for a lightweight development process. The specification and for efficient code generation. In this new development paradigm should be an alternative to context, model-based testing (MBT) proposes using documentation-driven, heavyweight software models for automating some of the testing activities, development processes. They called these principles e.g. test case generation, evaluation of test results, “agile manifesto”. Agile manifesto includes the which are tedious and error-prone tasks if they are following principles (based on [1]): manually done. In order to profit from model-based 1. Customer satisfaction: The highest priority in techniques in development process, however, some agile development has the customer satisfaction, efforts must be expended, e.g. for introducing tools, which can be achieved by early and continuous for training developers and testers, for creating and delivery of valuable software. This principle has maintaining models, etc. That is why MBSD is said to the highest priority in agile manifesto. All other be a “heavyweight” technique for creating better principles serve to achieve this goal. software. 2. Fast adaptation: In agile development, requirements changes of the customer are welcome, even in the late phases of the sulting software is assumed to be correct with respect development. The flexibility in agile processes to the specification. Test automation plays in impor- enables changes in software for assuring the tant role in agile methods supporting an efficient and customer's competitive advantage. effective development process. Having different fo- 3. Frequent delivery: For customer satisfaction, it is cus, agile methods mostly should be combined in or- important to show that the development process der to fulfill all principles of agile manifesto. makes progress. For showing this to the customer, deliver new versions of software frequently. 3. MODEL-BASED TESTING VS. AGILITY Define together with the customer what “frequent” means. The time slots can range from We believe that model-based techniques can help in a couple of weeks to a couple of months. Try to combining the different tasks in agile development by keep the time slots as short as possible, because using abstract models as primary development frequent delivery leads to frequent feedback. artifacts. Models can support communication between 4. Close collaboration: For achieving fast team members and customers, documentation of adaptation and frequent delivery, it is important customer requirements and design decisions and to understand customer’s business needs and automation of code generation and testing. Thus, consider them during the development model-based techniques can enable an integrated continuously. For that, business people and development throughout the whole project. As next, developers must work together every day we want to focus on how the documentation of throughout the project. customer requirements and their validation can be 5. Motivated members: Identify motivated team supported by model-based testing while following members who can push on the project. Provide principles of agile manifesto. them with the resources they need and support them while getting the job done. 3.1. Model-based Testing 6. Conversation: For achieving fast adaptation and With the emerging popularity of model-based frequent delivery, besides close collaboration software development, the usage of models in with the customer, also the efficient software testing is also desired. There are several communication between team members is definitions of model-based testing (MBT) in the important. The most efficient and effective literature, but the common understanding is that MBT method of exchanging information is face-to-face is “the automation of test design of black-box tests” conversation. [2]. Therefore, MBT uses abstract models (test 7. Working software: Supply the customer with models) of the system under test (SUT) or its working software which is the main measure of environment as the source for test generation. In progress. Delivering working software is addition to models of SUT and the environment, also indispensible for customer satisfaction. the testware itself can be modeled [3]. 8. Sustainable development: Agile processes There are three main advantages of MBT, which promote sustainable development. make this technique interesting: a) enabling high cov- 9. Constant pace: The customers and developers erage, b) need for lower effort and c) enabling early should be able to keep a constant pace for the testing. Because MBT uses sophisticated algorithms whole time of project. and tools for automatic test generation, far more test 10. Good design: Continuous awareness for technical cases than while manual testing can be generated. This quality and good design improves agility. way a very high coverage of the system specification 11. Simplicity: Simplicity is crucial, which means that and/or requirements can be reached. While test cases the amount of work to be done should be kept are not designed and implemented manually anymore, minimal. the effort for this task is significantly low. This works 12. Self-organization: Motivate team members to under the assumption that the modeling effort is lower organize themselves. than the manual test design activity. Last but not least 13. Reflection: Motivate team members to reflect the early creation of test models supports the valida- their experiences at regular intervals. Team tion of requirements even before the system is imple- members should discuss on how to improve the mented. effectiveness and the efficiency in team and should suggest improvements accordingly. 3.2. MBT as a technical enabler for Agility Existing agile methods aim at enabling these prin- Using MBT, the requirements can be captured and ciples. For example, Scrum promotes the close colla- communicated in form of models. Unified Modeling boration of customer or product owner at identifying Language (UML) provides many types of visual software functionalities to be implemented in the next diagrams for describing the desired structure and development cycles [4]. TDD advocates continuing behavior of software. Most of the diagrams have a programming until all predefined test cases are passed quite simple syntax and fairly clear semantics such [1]. Since test cases are seen as specification, the re- that customer and developer can easily learn how to express their requirements more precisely, thus However, MBT is not for free. Introducing MBT enabling the principle close collaboration. The into the agile development process requires some ini- changes in requirements can easily be made on the tial and continual efforts as discussed in [6] and [7]. already created models, thus improving fast These include: adaptation. Models can also support the conversation  Training team members for modeling between team members, where the results of a  Buying or developing modeling tools discussion can be edited into the models immediately.  Buying or developing test drivers and test Also the simplicity principle can be supported by adapters models by using the abstraction, modularization and  Defining modeling notations and test selection decomposition features of modeling. criteria There are different scenarios for creating and using  Creating and maintaining models models in MBT [9]. While some scenarios propose  Eventually extending generated test cases by sharing models (one model for test team and devel- test data opment team), some scenarios require separated mod-  Eventually evaluation of test results els (one models for each test and development team At first sight, these efforts seem to be not propor- respectively). Using shared models can support close tional to the lightweight development purposes of collaboration, face-to-face conversation and simplici- agile manifesto. However, test automation is an indis- ty. However, if same models are used for development pensible part of agility enabling the efficient and ef- and testing, specification errors cannot be found [9]. fective process. Fewster and Graham said in 1999 that Using separate models makes the teams for develop- “automating chaos just gives faster chaos”. MBT is an ment and test more independent and enables finding attempt to make test automation more systematic, specification errors, thus assuring working software. more maintainable. Models having a well-defined syntax and semantics In this paper, we have discussed how agility and can be handled by computers, which obviously bring MBT conceptually fits together. A concrete approach efficiency into the test process. The state-of-the-art for combining agility and MBT can be read in [8]. modeling techniques support creating good design. There, we have described a concrete approach includ- Depending on the context of development, formal or ing tool support for integrating MBT into Scrum. semi-formal notations can be used. The more formal the models are, the better automatable are the test ac- 5. REFERENCES tivities. Especially the automation of the test design task, which is the most costly and time consuming part [1] Beck, K. et al.: Manifesto for Agile Software De- in testing [5], leads to more efficiency. Test automa- velopment. Online resource at agilemanifesto.org tion is the key for assuring working software, frequent (Last visited: 29.07.2010) delivery, sustainable development and constant pace. [2] Utting, M. and Legeard, B.: Practical Model- Within MBT several coverage criteria for selecting Based Testing: A Tools Approach, Morgan Kauf- test cases can be used. One possibility is to cover the mann, 2007 customer requirements, which directly correlates with [3] Baker, P. et al.: Model-Driven Testing: Using the several agile principles. The customer satisfaction and UML Testing Profile, Springer Verlag, 2008 close collaboration principles are supported by refin- [4] Schwaber, K., and Beedle, M.: Agile Software ing and understanding customer requirements while Development with Scrum, Prentice Hall, 2002. modeling them and showing that those requirements [5] Pol, M. and Koomen, T. and Spillner, A.: Mana- were successfully tested. The usage of different selec- gement und Optimierung des Testprozesses, tion criteria and possibly combining them leads to dpunkt.verlag, 2002 higher defect detection rate and therefore facilitates [6] Güldali, B. and Jungmayr, S. and Mlynarski, M. working software. Due to changeable coverage criteria and Neumann, S. and Winter, M.: Starthilfe für mod- and automated test case generation, the test team can ellbasiertes Testen. OBJEKTspektrum, 2010, 3, 63-69 conduct different testing scenarios and gain expe- [7] Güldali, B. and Mlynarski, M. and Sancar, Y.: rience for further development cycles and projects. Effort Comparison of Model-based Testing Scenarios. This flexibility and configurability of MBT enables Proc. of Quombat Workshop at ICST, 2010 reflection in agile development. [8] Löffler, R., Güldali, B., Geisen, S.: Towards Mod- el-based Acceptance Testing for Scrum. Software- 4. A FAIR PLAY? technik-Trends, GI, 2010 (to be published) [9] Pretschner, A., Philips, J.: Methodological Issues As discussed in the last section, MBT can definitely in Model-Based Testing. In M. Broy, et.al. (Eds.), enable many principles of the agile manifesto. The Model-Based Testing of Reactive Systems, LNCS no. main advantage of MBT for the agile world is the 3472, Springer-Verlag, 2005, pp. 281-291. usage of models as primary artifacts and the automa- [10] Beck, K. Test-Driven Development: By Exam- tion of several test activities. This way MBT fits very ple. Addison-Wesley Longman, 2003 well with agility!