Challenges in agile adaptation is inevitable in software development projects and have to be dealt with by software practitioners. The pathway to excellence in agility requires experience of challenges, failure of process scenarios; and the discovery of working solutions by software development teams. The major purpose of this study is to highlight both the challenges organizations faced when implementing agile techniques and the solutions adopted that proved successful. In order to specify these challenges and working solutions, we performed a multiple case study by using the Software Agility Assessment Reference Model (AgilityMod). In this paper, we describe two cases that achieve the highest levels of agility among eight cases and describe their experiences in achieving a good adaptation through the challenges that they faced and the solutions that were found for these challenges. Additionally, we provide two challenges that have not been resolved yet and are subject to further discussions.
Agile software development is one of the most important paradigms that radically
changed how software is developed [
The promises made by the agile manifesto were so tempting that inevitably, agile was considered as a silver bullet. After a while, it was noticed by the software community that agile software development is not a “one size fits all” kind of approach. Every project, whether small or large, distributed or collocated has its own conditions that are specific to those environments. Besides, organizations do not quickly progress from low to high levels of agility. The pathway to excellence in agility requires experience 1Copyright ©2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). of challenges, failure of process scenarios; and the discovery of working solutions by software development teams.
Unresolved challenges may end up with the fossilization of software development
teams, which is a phenomenon described as being stuck with a condition that prevents
someone from improving. Mitigating the impact of such challenges had a significant
role on team performance and breaking the domino effect created by such challenges
[
It is also significant to identify to what degree a software development team
compromises agility while discovering the solutions that work for them. Agile maturity and
agility assessment models are utilized to identify upon which side of the agility line an
organization resides. Previously we assessed the capability of existing models [
We developed the Software Agility Assessment Reference Model (AgilityMod) to
provide a structured model for agility assessment. The Model provides a means for
identifying agility gaps in software development projects [
We performed a multiple case study to identify AgilityMod’s applicability in operational software development projects. The case study included eight industry cases. In this paper, we present the two cases (Case G and Case C), which achieved the best agility results. The main emphasis of this paper is upon the lessons learnt from the most successful implementations of agile and that is why we only focus on the two most successful cases.
The purpose of this paper is to describe both the challenges organizations faced when implementing agile techniques and the solutions adopted that proved successful. We think that it is also very important for the challenges to be described with their own environments where they were observed. From this perspective, the paper will provide a specific insight to readers about what could work, under what kind of conditions. The challenges described here are not at an abstract level, but actual, detailed and specific.
This paper is structured as follows: In Section 2, we discuss the challenges organizations face when embarking upon agile software development and suggested solutions for these challenges, based upon previous documented research. In Section 3, we briefly explain the Software Agility Assessment Reference Model (AgilityMod) that we developed and was subsequently used within the eight case studies. In section 4, we present the case study. In section 5, we both provide the challenges that were faced in agile adaptation and the specific solutions that were successfully implemented in two organizations. In Section 5, we also describe two challenges that still need to be further investigated for effective solutions. Finally, in Section 6, we present our overall opinion on this topic. 2
Agile has had a groundbreaking impact on software development. Debates on what
agile is still continue, as to whether agile is: a development and management
philosophy; a collection of technical practices; a way of life; or all of these [
Nerur et al. specify that agile and traditional approaches diverge in a number of
aspects: approach to control, management style, knowledge management, role of the
customer in development process, role assignment, communication style, development
life-cycle, organizational culture and technology [
Stober and Hansmann compare the characteristics of software development teams to
the characteristics of fractal units in mathematics which are self-similarity,
goal-orientation, self-organization, self-improvement and vitality [
The core set of Agile methods or to use Highsmith’s phrase Agile Software
Development Ecosystems [
For organizations transitioning from traditional approaches to agile approaches,
major challenges are observed in customer involvement, documentation, upfront
requirement analysis, document-driven testing, communication and knowledge sharing [
Sekitoleko et.al [
Kaisti et.al’s study is also valuable as they specified agile challenges in an embedded
systems development domain, a domain that had not benefited much from the agility
so far [
There are a small number of studies on identification of agile adoption challenges
specified through empirical research and focused on real life cases. Identification of
optimum granularity levels for user stories has been studied by Liskin et.al [
Ramesh [
Conboy et. al [
Compared to other research papers described above, the last three emphasize the real-life problems. The others basically focus on the challenges from a high level (abstract) perspective. We do not ignore these attempts however, we are sure that we should be more specific about the challenges, the solutions and the surrounding conditions that created those challenges to provide insight about real life situations to the reader. 3
AgilityMod is a software agility assessment reference model the structure of which was
defined in accordance with the ISO/IEC 15504-Process Assessment Model [
From AgilityMod’s perspective, Agility is the capability of being able to give and obtain feedback rapidly, being adaptive to changing conditions, having confidence on developing solutions to complex problems, being creative and innovative, respecting others, working with humility, learning from mistakes, improving continuously, solving problems/issues with communication and moving away from complex and bureaucratic procedures.
The model mainly consists of two dimensions: Aspect Dimension and Agility Dimension which can be seen on Fig.1.
LEVEL 3: EFFECTIVE se li p rcPnd LEVEL 2: LEAN n i a o iftsaen LEVEL 1: AD-HOC M e li g A LEVEL 0: NOT IMPLEMENTED
Agility Dimension
AGILITY ASSESSMENT REFERENCE MODEL
AgilityMOD Agile Practices and Processes
Aspect
Dimension
The first dimension of the Model is the Aspect Dimension. This dimension is
described with Aspects rather than processes. Because, the formal process layers of
traditional software development are intertwined to each other in agile software
development. Therefore, it is difficult to specify boundaries for agile processes. From this
perspective, the Aspects are a collection of interrelated and interacting activities of agile
processes and practices that are integrated under meaningful and agile compatible
abstract definitions [
The purpose of the Exploration Aspect is to understand the customer/user needs and to transform these needs into artifacts that initiate communication for elaboration on them during construction and manage the changes to these artifacts. The purpose of the Construction Aspect is to develop a high-quality software solution that is ready to be built, including architecture, design, coding and unit testing activities. The Transition Aspect includes practices to establish and maintain reliable and repeatable build, integration and deployment practices to keep the application in a working state throughout development. This makes it possible to obtain feedback in relation to the problems in the process, to make the whole process visible to everyone, and to shorten the response time for changes. Finally, the purpose of the Management Aspect is to perform planning and tracking activities continuously, and estimating collaboratively to achieve efficiency and performing these practices as value adding activities to the project life cycle
At the Agility Dimension, Agility of an aspect is described with four Levels: “Not Implemented (L0)”, “Ad-Hoc (L1)”, “Lean (L2)” and “Effective (L3)”. When an aspect progresses from the bottom level to the top level, its conformance to agile values and principles increases.
At Level 0, the aspect practices are either not achieved or partially achieved. At Level 1, fundamental development processes such as requirements development, design, coding, integration, testing, and deployment are performed consistently. There are transition attempts towards agility by exploring best fitting agile practices or approaches. Aspect practices are implemented and aspect purposes are achieved; however agile values and principles are not fully incorporated into aspect practices. At Level 2, work products are developed iteratively and incrementally, non-value-added activities are eliminated from the aspect practices, balance is achieved between adaptive and predictive works. At Level 3, each aspect is performed to achieve delivering value with high productivity and low defects by employing agile engineering practices and using agile tools to support a continuously improving environment. 4
We performed this study using a qualitative research approach where the researchers
collect data in the natural settings through overviewing documents, observing behavior
or interviewing participants [
We performed a multiple case study that included eight cases. We observed the
applicability of AgilityMod for the identification of agility gaps in software projects and
also to identify strengths and the weaknesses of the Model. We explained the case study
and the results in detail in [
We selected the eight cases randomly in order to observe different levels of agility for each aspect. The rationale behind this approach is that without performing an agility assessment, it is not rational and easy to make judgments about the maturity of the cases, just by knowing the duration of agile adaption efforts.
Below, we both describe the two organizations that the projects were performed in and the two projects that were subjected to this research. Again, we are not describing all eight cases here, just the cases that achieved the hightest agility levels after the assessment with AgilityMod.
Organization G is a government IT organization responsible for developing e-government software for various governance organizations. It is located in Ankara, Turkey.
The project (Case G) that was subjected to the assessment, is an e-government project, providing solutions to 40 foundations which are located in different cities of Turkey and with approximately 25 million Turkish citizens. Case G includes 21 employees divided into four teams which report to a project manager and an assistant project manager. Three of these teams purely work on software modules, the last one is involved in both system infrastructure and software development activities. Each team has a technical team leader. Other members of the team did not have specific roles, each one was involved in analysis, design and development activities. Since the beginning of the project in 2009, 7 million LOC has been developed. The product is developed in iterations, each of which is one month in length. There is a signed contract between the organization and the customer to specify the deadlines and budget.
The functional domain of the assessed project, is classified as a “Controlling Data
System” based on the CHAR group method [
Organization C works within the internet security domain. They develop products with the purpose of securing information on the internet, securing websites and e-commerce applications and personal computers. Org. C is an international company, doing business over 100 countries, having 25 million end users, and over 7000 business partners.
Case C is a digital advertisement sharing platform. It is in use and new versions of the product are being deployed continuously. The purpose of the project is to ensure the security of the advertisements and to deliver harmless and focused advertisements to end users. The project includes 22 employees. There are three different development teams and Scrum Masters for each of the teams. Overall, there are four testers, 13 developers and an architect. Additionally, each team has a program manager. Apart from these members, there is a product owner residing in the US.
Case C is built upon legacy code. Java, PhP and Python languages are being used for different modules of the product. The project includes big data analyzes performed using the tools Cassandra and Hadoop. Scrum is used for project management activities. The product is built iteratively with each iteration lasting three weeks. 4.2
We assessed the Agility of Case G and Case C by both interviewing with project team members and observing of the outputs produced during the project life cycles. Prior to the case study conduct, we developed a list of scripted questions related to the aspect practices and agility practices of the Model. We followed a semi-structured interview which included asking additional questions based on the project context and challenges. Each interview session was recorded and transcribed. The assessor was one of the authors of this paper who had four years of experience on agile software development at that time.
For Case G, the assessment was performed over a three-hour time period with the technical leader of the infrastructure team who had worked formerly as a developer and has knowledge about the project’s processes. For Case C, the assessment was performed in a three-hours with the configuration manager and the quality assurance manager who is also the scrum master of the project.
After the assessments, we developed the agility assessment reports and shared these reports with the case organizations. In addition, we presented the results to the assessment teams. The presentations covered the assessment findings, the aspect levels and the improvement suggestions. After or during each presentation, we discussed the results with attendees. Thus, we ensured the validity of the case study results by discussing our findings and observations with the interviewees and managers in the organizations.
We present the agility assessment results in Fig. 2 and Fig. 3 for Case G and Case C respectively. These figures give the colored schema of the assessment ratings to provide a visual high-level view of the findings. Each column refers to the practices of AgilityMod (APx and GPx). The colors and the numbers in each cell refer to the achieved levels of these practices. We used a-four-level rating scale to express the achievement of the aspect attributes: “not achieved (0-red), partially achieved (1-yellow), largely achieved (2-orange) and fully achieved (3-green) and not applicable (NA)”. For an agility level to be reached, all the practices should be largely or fully achieved.
The Case G achieved Level-3: Effective for all of the aspects. This essentially means that Case G’s aspects are iteratively performed, lean, technically excellent and continuously improving. Fast feedback is obtained and effectively communicated among team members. On the other hand, the customers are located in a different building. The communication between the customer and the team is not as effective as the communication among team members. The ways to communicate with the customer needed to be improved. It was found that retrospective studies to identify improvement areas are not performed regularly. It is suggested to perform regular retrospective studies that would provide much more value to the processes. Another recommendation provided to teams was to establish a generic measurement framework to improve the decision making.
Similarly, Case C also showed very good results. All of the practices of exploration and management aspects of Case C were rated as fully achieved. The Construction aspect of Case C is at Level 3. The weakest aspect of Case C is the “Transition” aspect which is at Level 2: Lean level. The major improvement areas for this project in terms of the Agility are achieving continuous integration, continuous delivery and increasing unit test coverage and automated test ratio, refactoring continuously and managing technical debt better.
In Fig. 4, we provide a comparative radar chart to display the differences between the ideal case (blue line) (the case that the all practices were fully achieved) and the current situation of Case G (orange line) and Case C (grey line). The data to draw the current situation of the cases for each aspect was obtained by adding the rating values in each cell along with the horizontal columns given on Fig. 2 and Fig. 3 (E: Exploration, M: Management, T: Transition, C: Construction). This chart allows us to observe the how far each case’s Aspect is from being fully agile according to AgilityMod.
In this section, we present two types of challenges: First, the challenges that Case C and Case G teams overcame through implementing successful solutions (Resolved). Second, the challenges that need further discussion (Unresolved). A summary of the challenges and working solutions is provided in Table 1. 5.1
Having no on-site customer: As specified in the Agile Manifesto, one of the critical success factors in agile software development is the level of interaction between software development teams with their customers.
In Case G, the Product Owner (PO) lives in the United States, while the rest of the team reside in Turkey. However, the product owner communicates with the program managers regularly (3 to 5 times in a week) over teleconferencing despite the 8 hours difference. The PO does not only communicate with the program managers but also with the scrum masters and the developers when further clarification is required for the backlog items.
This case shows that “distance” is not an excuse for low levels of communication with the customer. The commitment of the customer is a very significant part of overcoming the no on-site customer challenge. In Case C, what we found was that having customers on different continents has given Case C teams significant insight into the importance of communication where they have focused on ways to strengthen their interaction channels.
Granularity level of requirements: In agile software development, user stories are
a widespread way of specifying software requirements. However, the identification of
optimum granularity levels for user stories still remains a challenge. Coarsely
granulated user stories are the source of difficulties and problems in the estimation process
[
In Case C, a well working process has been implemented for this challenge. It was stated that the business needs were transformed into software requirements through the following stages: First, the product owner and/or program managers define the business needs as “epics” in the product backlog in Atlassian’s Jira tool. Second, the program managers and software development teams perform “backlog grooming meetings” once a week where each epic were split-down into user stories.
It is essential to mention here that these grooming meetings were performed independent of the scope of the upcoming sprints. If the teams require further information to clarify some issues, the PO was requested to involve in those meetings.
The teams use two approaches to decide on the optimum granularity level for user stories. The first one is the “story points (sp) estimation”. A user story is not included in a sprint, if its size is above a threshold (for Case C, it was 20 sp.). A user story is discussed, detailed and re-estimated until it reaches the pre-specified sp level. The second one is the “acceptance criterion” which is essential for both of the development and test teams. Definability of the acceptance criteria is an indicator of a well-defined user story which was spilt-down into a sufficient level of detail.
defined in AgilityMod [
In Case C, there had been a stage when this flow had been interrupted. The product backlog had not grown in a constant pace. Upon investigation, it was observed that the issue arose due to communication problems among the PO in the US and the program managers in Turkey. Once they sensed the reason for the problem, they established a communication matrix that had to be updated whenever the PO and the program managers communicated with each other. After a while, the problem became so obvious that the frequency of the communication was once or twice in a month between some of the program managers and the product owner. A communication matrix is not a direct solution for this challenge, but it is a very effective tool to observe communication problems.
Since the effectiveness of communication is considered a critical success factor in agile software development and the effect of the communication matrix observed by the Case C team, they started using it to observe all the interactions within the project.
Consequently, Case C established a correlation between the growth of the product backlog and the numbers in the communication matrix.
Another solution for the continuous product backlog growth in the Case C, is the conduct of regular product backlog grooming (PBG) meetings. The program managers, scrum masters, developers, testers and the product owner are the members of PBG meetings. Teams spend almost two hours a week for elaboration activities. Since it is regular, even a two-hour-time makes a big difference in solving such a significant problem.
Nonfunctional retrospective meetings: Retrospective meetings are one of the ways
to transform good teams to great teams [
When the Case C quality assurance manager observed that none of the improvement suggestions proposed in the retrospective meetings were implemented, he decided two things: The first one was to open action items for each of the issues and assign the items to team members using the Jira tool. The second one was to specify a team quality criterion based upon the percentage of closed retrospective issues in Jira. Both these approaches allowed the Case C teams to perform effective retrospective meetings and observe the results very quickly.
Ineffective review meetings: Conventional review meetings might be a waste of time for software development teams in some cases. Especially when lots of ideas are discussed, without a decision being made. The solution found by the Case G team was to provide a tool support for design and code reviews. They utilized the Confluence tool to review class and sequence diagrams in software design and the Crucible tool for design and code reviews. They specified the rule of everyone on the team can comment on the code parts and all the comments are seen by other reviewers. After this initial review, final remarks are decided with a meeting if necessary.
tional problems in software development teams.
The Case G team members had suffered from high personnel turnover in the testing team and this was not the only problem. They were in a continuous “fire-fighting” reactive state, because of the bugs found in released versions of the product G.
In such a case, it might be very difficult to notice the sources of the problem and easy to blame other people working in the team. Eventually the Case G team noticed that the problem mentioned above was due to a decrease in the motivation level of the testers. The managers in the company were hiring successful, experienced and talented developers to establish a strong testing team. However, they were assigned mostly black-box manual testing roles which do not require good development skills but require domain knowledge.
The managers of the Case G have made a radical decision to quit manual testing and abolished the test team. All of the testers were assigned to different parts of the development team where there is no distinction among team members. Then, they were asked to code the automated unit tests. It was mentioned that this had been one of the breaking points for the Case G team in terms of moving towards agility. They resolved this challenge by collaborative work and adopting shared responsibility.
Ability to manage technical debt: Technical debt is evitable in software development, but it can be managed. In some cases, team needs to develop quick solutions or hot fixes which may cause technical debt. On the other hand, it is very easy to overlook the created technical debt in daily life turmoil, if there is no specific mechanism to control it.
The solution that was found by the Case G team for this challenge by assigning the responsibility of recovery from technical debt to the person who created it and following the progress of such recoveries via a tracking system such as Jira.
Working Solutions Take the commitment of the customer for frequent meetings Decide the frequency of the meetings Involve customer and product owner to team discussions Break down the user stories until the size of each is below a threshold level Define acceptance criteria for each user story Define action items in issue or item tracking system for each of the improvement items and assign that items to a person Review all the previously specified improvement action items before the retrospective meeting Use tool support to review design and code Let everyone in the team comment on the code parts and all the comments seen by other reviewers Decide final remarks with a short meeting Combine test teams and development teams Abandon manual testing except for the exploratory testing Support collaborative work and shared responsibility Give the responsibility to manage the technical debt to the person who created it
Make sure you recorded the technical debt to not to overlook it The following challenges were observed in the projects; however, satisfactory solutions have not been found for them yet.
ment: Knowing the relationship between design elements has a significant impact on identification of changes within an existing software system. On the other hand, the larger the system, the more difficult it is to specify the relations among modules or lower level design parts. In addition, teams mostly overlook and rely on personal experiences for change impact analyses until the system grows to an unmanageable size.
This was what happened in both Cases. The impact of new requirements on modules and lower level module components were evaluated based on personal experiences. To date, they have not experienced significant issues due to not establishing traceability. But, this is a valid concern for them as their systems grow rapidly.
Brown, Nord and Ozkaya emphasize the importance of architectural agility in
achieving success [
The efficiency of the code comments: Code comments are significant especially for the living software systems where a policy of little documentation is applied. Today’s source control systems do not allow developers to check-out code parts without any comments. But the efficiency of the code comments is not evaluated. There is a need to identify and evaluate efficiency of code comments to increase the clarity of the code especially at maintenance phase of a software development life cycle. 6
Agile software development methods are frequently adapted in recent years by the software community as they are seen as well solutions for software development problems. Upon the introduction of this new approach to traditional software development environments, researchers and practitioners started to deal with agile adaptation challenges.
As most of the agile software development models are not highly prescriptive in terms of adoption processes, the experience reports published in this topic remains as an important problem for practitioners in specific contexts.
In this paper, we briefly described the Software Agility Assessment Reference Model (AgilityMod) the purpose of which is to assist software organizations in assessing projects’ agility levels, indicating the gaps that prevent fully obtaining the benefits of agile software development and introducing roadmaps in adopting agile principles/practices. Secondly, based on the multiple case study that we had assessed software projects’ agility with AgilityMod, we selected most successful two cases among the eight cases. We presented the challenges that these cases had faced during agile adaptation and the best solutions that worked very well for them. The major contribution of this study is the insights provided to readers about real life challenges faced and how these challenges were overcame. We also discussed two unresolved challenges that will require further research.
Further studies need to be performed for the discovery of efficient solutions for such problems. We believe that the software industry will benefit from experience reports discussing challenges observed and lessons learnt in different domains.
This study is partially supported by Turkish Scientific and Technological Research Council of Turkey (TÜBİTAK), grant number 113E528. This research is also supported in part by Science Foundation Ireland under a co-funding initiative by the Irish Government and European Regional Development Fund),and by Lero - the Irish Software Research Centre (http://www.lero.ie) grant 10/CE/I1855 & 13/RC/20194