Towards an Economic Foundation for the Decision between Agile and Plan-driven Project Management in a Business Intelligence Context

Abstract

Lacking a formal yet practical decision model, nowadays decision makers mostly follow corporate guidelines or their intuition when it comes to the decision between agile and plan-driven project management in Business Intelligence projects. As one size does not fit all, using management methods hyped by temporary fashion or other management methods not adapted to the situation bears the risk of project failure. Thus, this paper proposes a risk-adjusted net present value-based model to support decision makers in the selection of the appropriate management method for Business Intelligence projects. We focus on two decisive risk parameters - the likelihood of environmental changes and the peril of improper system integration - and a project's estimated cash flows. As a result, the tradeoff between different characteristics of risks and cash flows in a specific project is formalized and made transparent. In summary, this research-in-progress paper sketches the idea of a practical decision model that improves the foundation for the selection of the appropriate management method.

-

Although studies suggest that non-technical factors such as project management dominate a Business Intelligence (BI) project’s success (Adamala & Cidrin, 2011) , the decision on a project management method in BI projects, such as Scrum, Extreme Programming, the waterfall model or Sandboxing, is often not based on rational considerations (Project Management Methodologies. n. d.). Examples include German authorities that require any in-house information technology (IT) project to use t he V-model XT (Höhn & Höppner, 2008 , p. IX). In other cases, the decision on the project management method is the result of nothing more than “this is what we’ve always done” (Project Management Methodologies. n. d.). Thus, many companies or authorities seem to give either mandatory “one-size-fits-all” instructions or apply project management methods inconsiderately. The Project Management Institute defines project management as “the application of knowledge, skills, tools and techniques to project activities to meet the project requirements” (Project Management Institute, 2008, p. 6) . While project management describes the necessity to use knowledge, skills and tools, a project management method(ology) describes more precisely how this knowledge, skills and tools look like: “A methodology is a set of guidelines or principles that can be tailored and applied to a specific situation. In a project environment, these guidelines might be a list of things to do” (Charvat, 2003, p. 3) . As the terms “method” and “methodology” are often used interchangeably in ITrelated literature (Boehm & Turner, 2003; Charvat, 2003; Larman & Basili, 2003; Moss, 2009) , we will consistently use “method” in the following.

There are important differences between BI and other IT or software development projects that need to be addressed. While BI projects are rather data-centric business integration projects, being closely linked to a company’s strategy, software development projects are rather code-centric (Moss, 2009) . This is the reason why non-technical success factors dominate technical success factors in BI projects (Adamala & Cidrin, 2011) . Hence, the selection of an appropriate project management method is critical. Instead of an ex ante determined project management method that might ultimately result in project failure, a situation-based decision prevents the application of a potentially inappropriate one (Charvat, 2003, pp. 18-20) . Therefore, a situation-based decision seems more promising. BI projects – as any kind of projects – tie up capital and accordingly should be seen as investments supposed to increase shareholder value. For the required evaluation of suitable project management methods, a value-based and future-oriented approach is adequate (Coenenberg, Mattner, & Schultze, 2003, p. 3; Mertens, 1999, p. 11) .

Surprisingly, “there are few [researchers] that compare agile method projects with those using traditional approaches, which one would expect when a new range of methods […] claims to be superior” (Conboy, 2009, p. 331) . Furthermore, decisions on project management methods lack a fact-based and comparable approach considering significant parameters and, therefore, bear the risk of incorrect decisions. After an initial literature review (including: Black, Boca, Bowen, Gorman, & Hinchey, 2009; Boehm & Turner, 2003; Charvat, 2003; Conboy, 2009; Fowler & Highsmith, 2001; Kerzner, 2009; Moss & Atre, 2003; Wysocki, 2011) , there seems to be no approach of a quantitative decision model for the selection of the most appropriate project management method in a specific BI project setting. This leads us to following research question: “What would be an appropriate value-based decision model that supports decision makers in their selection of a specific project management method for a given BI project?” This research-in-progress paper first identifies requirements for a practical decision model and provides necessary theoretical background. The subsequent section sketches the idea of a value-based decision model using a simplified example for demonstration. The model enables the determination of the most appropriate project management method in a particular BI project setting. The last section discusses limitations of the current model sketch and describes the further research agenda to enhance both maturity and generalizability of the decision model. 2

Theoretical Background

As indicated above, the decision on an appropriate project management method in a BI context lacks a comprehensible and practical decision model. From a scientific point of view, we expect a decision model to fulfill the following requirements: It has to (1) be applicable to a class of problems, (2) contribute new findings to the body of knowledge, (3) be comprehensible and reproducible, and (4) generate value for a user now or in future (Österle et al., 2010, p. 666). Furthermore, we additionally expect the model to (5) be economically feasible to ensure its applicability, i.e., the effort of application should be justified by its benefits, and (6) adopt a value-based approach using quantitative metrics as indicated in section 1 (Coenenberg et al., 2003, p. 3; Mertens, 1999, p. 11) . As object of research, we limit the considered project management methods in a first step to two extremes: a purely agile project management method (A) and a purely plan-driven project management method (P). Regarding the former, we need to distinguish agile project management methods from iterative and incremental approaches. While iterative development is “a rework scheduling strategy in which time is set aside to revise and improve parts of the system”, incremental development is “a staging and scheduling strategy in which various parts of the system are developed at different times or rates and integrated as they are completed” (Cockburn, 2008, p. 27) . A does not necessarily include iterative or incremental elements, although changes during project runtime might suggest including one or both of the two approaches – corresponding with the principle to “value responding to change over following a plan” (Fowler & Highsmith, 2001, p. 29) . Both extremes, A and P, can be realized differently, for example with Scrum or Extreme Programming in case of A and CMMI (Capability Maturity Model Integration) or other plan-driven project management methods such as the waterfall method in case of P (Boehm & Turner, 2003, p. 14 ff.) . Since we refer to the major characteristics of A and P, the key findings of this paper can be applied to any realized A or P. P promises planning reliability, strong documentation, comparability and repeatability (Boehm & Turner, 2003) . However, the major shortcoming associated with P has been widely criticized: It is inappropriate to handle rapidly changing environments and customer requirements (Abrahamsson, Conboy, & Wang, 2009, p. 281) . Consequently, dynamism is the major risk to be addressed in projects applying P (Maruping, Venkatesh, & Agarwal, 2009, p. 377) . On the other hand, A is characterized as lightweight processes with close customer collaboration and short iterative cycles resulting in independently running modules (Boehm & Turner, 2003, p. 17; Fowler & Highsmith, 2001) . The trend to value working software over documentation, individuals and interactions over processes and tools and responding to change over following a plan bear the risk of delivering perfectly working individual modules which are not thoroughly working as an integrated system (Black et al., 2009, p. 39; Fowler & Highsmith, 2001, p. 29) . Therefore, the risk of improper integration is the major risk to be addressed when applying A. In summary, environmental dynamism and improper system integration are the two major risks to be taken into account in the proposed basic decision model.

Literature shows that there are few researchers who compare projects applying one project management method or the other (Conboy, 2009, p. 331) . Therefore, we seek for support on the decision between A and P in a specific project setting. We are aware that not all relevant project management methods can be classified into one of the two extremes. However, to identify key issues in a first step, we will not take into account those as well as the possibility of merging two project management methods – like already proposed, for example, in Boehm and Turner’s five step risk-based approach (Boehm & Turner, 2003) . While their approach evaluates the project’s characteristics in detail, they do not take into account the project’s expected cash flows, and hence lack the desired quantitative nature.

Since there is no decision model that meets the aforementioned requirements, we suggest a value-based and future-oriented Net Present Value (NPV) model taking into account the project’s planned cash flows and integrating the risk of dynamism and the risk of improper system integration. 3

Proposition of a Decision Model and Demonstration Example The example project has the objective to set up three reports (R1, R2, and R3) based on a data warehouse (DW) that integrates data from three operational databases (ODB1, ODB2, and ODB3) (see Figure 1). The project has to be finished within the next nine months. Data analysis, distribution and presentation Data integration and storage Data transfer and retransfer

R1

R2

R3 The three reports provide different value to the company. Their implementation costs depend on the reports’ complexity. The most important report R1 fulfills new mandatory regulatory requirements and, by preventing sanctions, provides the highest value. It requires the integration of two operational databases (ODB1 and ODB2). R2, a better CRM (Customer Relationship Management) report, has the second priority and also requires ODB1 and ODB2. R3 shows the comparatively smallest value contribution. It offers more detailed analyses for controlling purposes and requires all three source systems to be available.

Costs and returns are supposed to be based on expert’s estimation. For the simplified example we assume following one-time costs that are identical irrespective of the selected project management method: Developing R1 causes costs of €10,000, R2 €30,000, and R3 €40,000. Once the reports are in place, they generate recurring returns of €5,000 (R1), €2,500 (R2), and €1,500 (R3) in each period. A period thereby equals a quarter (three months). Integrating an ODB causes costs of €20,000 each. The development of the DW costs €60,000, which will be – in case of applying A – assigned mostly to the first period t1, taking into account that the data model has to be defined at the beginning. The costs of the remaining modules are assigned to the period when they are implemented and tested (see Figure 2). In case of applying P, we assume that the costs in period t1 are slightly higher than in the following periods due to extensive project planning activities (Black et al., 2009, p. 42) . The risk situation is characterized as follows. In case of a changing environment, we assume additional costs of €30,000 for A and €60,000 for P since the latter one is not able to adjust to changes as flexible. The likelihood for the occurrence is identical for both project management methods. In case of improper system integration, we assume additional costs of €180,000 whereby the likelihood of this peril is three times higher for A than for P.

The project roadmap has the milestones as depicted in Figure 2. Note that we only take into account three periods in the proposed model. We additionally show the final BI system (periods t4 to tn) for the sake of completeness. In case of applying A, in the first iteration the most important steps are taken, namely developing the major parts of the DW, integrating ODB1 and ODB2, and establishing R1. In the second iteration, the DW is completed and R2 is established. The missing ODB3 and R3 are integrated in the last iteration. In case of applying P, the first period is used to design the system architecture, the second period is used for the initial implementation of all system components, and in the last period the system is tested.

Formally, the decision model is based on following assumptions: 

The decision to implement a BI system is final. The company only chooses between two project management methods, agile (A) or plan-driven (P). Both are capable of delivering the same functionality within the same time (Black et al., 2009, p. 42) . Due to its distinct strengths and weaknesses, each project management method has a different cost/benefit structure that is captured in aggregated cash flows per period.

All considered cash flows are discounted to the period the project starts.

The project is affected by two types of risks only. These are the likelihood of environmental changes within a project period δ ϵ [0; 1] and the likelihood of a lack of integration at the end of the project ε ϵ [0; 1]. Both risks are independent. δ is constant for all periods and independent of the chosen project management method. ε is inherent to the chosen project management method and hence differs. δ occurs in each period, ε only occurs in the last one.

If a risk occurs, the company will fix the failure. The costs associated with the correction are considered within the cash flow of the period the risk occurred (again discounted to the period of project start).

The assumptions required for applying a NPV approach are fulfilled. That is, the company reinvests all cash flows in other value-contributing projects and acts within a perfect capital market.

Due to the risk structure of the project, we model the associated risks by a random experiment with three periods. Figure 3 shows an exemplary decision tree for A. In each of the three periods, two outcomes regarding the environmental risk are possible, “no change” with a likelihood of 1 - δ, and “change” with a likelihood of δ. Additionally, in the last period the risk of improper integration occurs with a likelihood of ε while ”good“ integration occurs with a likelihood of 1 - ε. According to A5, we consider both risks by adding their expected costs to the cash flow of the period the risk occurs.

δ . . .

δ

No changes (€ -43,388) Changes (€ -68,128)

"Good" integration (€ -43,388) Improper integration (€ -192,149)

"Good" integration (€ -68,128) Improper integration (€ -216,942) To calculate the NPV, the likelihoods have to be multiplied along all possible paths in the decision tree. The expected cash flow is this likelihood of a path multiplied by the sum of the corresponding cash flows along the path (already discounted based on an internal interest rate of 10%). The total NPV is the sum of these expected cash flows of all possible paths.

Likelihood of environmental changes 0 0.1 According to the common NPV approach, one compares the NPVs of two alternatives and chooses the higher one – given it is positive (Ross, 1995, p. 96) . In our demonstration example, the resulting NPV is expected to be negative since we only take into account the cash flows during project runtime but do not consider returns that are generated after the end of the project. This is possible since A1 ensures that the project is economically feasible in the long term. Furthermore, we extend this NPV approach by a comparison of the NPVs of both project management methods. We simulate all combinations of δ and ε in discrete steps of 0.1 in order to identify the risk parameter combinations that mark the turning point in the decision. Figure 4 visualizes the resulting recommendations for any parameter combination.

We see that the model’s recommendation is not contrary to an intuitive estimation. For highly agile environments with δ 0.8, A is superior regardless of the value of ε. A is also superior in case of low integration risks with ε 0.3. For ε 0.6 and δ 0.5, P is the most appropriate project management method. For all other parameter combinations, the turning point is located in an area of indifference between both project management methods. The shading of Figure 4 indicates the deviation of the resulting NPV in percent if applying the less appropriate project management method instead of the recommended one. While the decision’s turning point is located in a corridor of marginal differences, it reaches up to 40% in the boundary area. This result shows that applying project management methods inconsiderately might result in significant discrepancies in the resulting NPVs. Even though the model does not yet offer surprising findings concerning a recommendation, we appreciate that its results conform to intuitive decisions in this simple demonstration example. This indicates that a more advanced model might give useful recommendations in more complex decision situations when an intuitive decision is no longer reliable.

The proposed decision model requires a sufficient clarity and preciseness of parameter values that might demand great efforts to be provided. To analyze the necessary degree of data accuracy, future research will include sensitivity analyses addressing the parameters of the model. These sensitivity analyses should further strengthen the fact base on which decision makers can derive a recommendation for the appropriate project management method – instead of relying on “one-size-fits-all” instructions or gut feeling. In this respect, this paper already provides a rough model sketch and proves the principle feasibility of an approach based on estimating two risk parameters and a project’s cash flows. 4

Limitations and Outlook

The presented model fulfills or is able to fulfill all six initially stated requirements mentioned in section 2. (1) It is applicable to a class of problems, namely to decisions on the project management method in a BI project. (2) It contributes to the body of knowledge since no formal quantitative model on the decision between agile and plan-driven project management methods could be found in literature. (3) The underlying NPV approach is established and results are easily reproducible. (4) The model claims to add value to the decision process by supporting decision makers in the selection of the most appropriate project management method. (5) Since estimating the project’s cash flows is necessary for project planning anyway, the model is easily applicable. Decision makers have to only estimate two additional risk parameters. (6) Relying on a NPV-based decision model, we propose the way towards a quantitative, value-based and future-oriented approach. Nevertheless, the proposed model is beset with limitations that need to be taken into account when applying it in project settings.

  

The proposed model is only applied to a three period case study.

The proposed model only takes into account two decisive parameters.

The peril of improper integration is assumed to occur only in the last period, while other arguments suggest this risk could be prevented earlier.  

The proposed model only takes into account a “pure-blood” agile and plan-driven approach and neglects other project management methods and hybrids of both extremes.

The proposed model claims to offer a clear recommendation for a decision, yet might underlie errors regarding the estimation of cash flows and risk parameters. Despite the named limitations, the proposed NPV model represents an initial decision model comparing the impact of selected project management methods in a BI context. Based on the presented model sketch, we currently work on a more detailed approach. We will address L1 by the development of a general analytical model that can be applied to any number of project periods. The limitation to two parameters (L2) might be considered a far reaching simplification. Therefore, based on a more detailed analysis it might be an interesting option to add more risk parameters as, for example, the five parameters identified by Boehm and Turner (2003) . Referring to L3, we will consider the possibility that decision makers detect an integration problem not at the end, but earlier during the project. Thus, an extended model might consider both risks in each period. Additionally, further research should address L4 by proposing a general analytical model that can be applied to any project management method. To strengthen the reliability of the decision model and to examine the effects of estimation errors, we plan to address L5 by conducting sensitivity analyses to test the impact of deviations of estimated cash flows and risk parameters. Additionally, it might be interesting to evaluate the model using case studies from different industries in order to identify if – and in which aspects – the model needs to be adapted. Summarizing, since there seems to be a lack of theoretical foundation for a decision on the appropriate project management method in a BI project, we presented a research-in-progress model already giving first recommendations and an outlook on future research.

References

Abrahamsson , P. , Conboy , K. , & Wang , X. ( 2009 ). 'Lots done, more to do': the current state of agile systems development research . European Journal of Information Systems , 18 ( 4 ), 281 - 284 .

Adamala , S. , & Cidrin , L. ( 2011 ). Key Success Factors in Business Intelligence . Journal of Intelligence Studies in Business , 1 ( 1 ), 107 - 127 .

Black , S. , Boca , P. P. , Bowen , J. P. , Gorman , J. , & Hinchey , M. ( 2009 ). Formal versus Agile: Survival of the Fittest . IEEE Computer, 42 ( 9 ), 37 - 45 .

Boehm , B. , & Turner , R. ( 2003 ). Balancing Agility and Discipline: A Guide for the Perplexed Addison-Wesley Longman Publishing Co ., Inc.

Charvat , J. ( 2003 ). Project Management Methodologies: Selecting, Implementing, and Supporting Methodologies and Processes for Projects John Wiley & Sons Inc.

Cockburn , A. ( 2008 ). Using Both Incremental and Iterative Development . STSC CrossTalk , 21 ( 5 )

Coenenberg , A. G. , Mattner , G. R. , & Schultze , W. ( 2003 ). Wertorientierte Steuerung: Anforderungen, Konzepte, Anwendungsprobleme. In A. Rathgeber, H. J. Tebroke & M. Wallmeier (Eds.), Finanzwirtschaft, Kapitalmarkt und Banken, Festschrift für Professor Dr . Manfred Steiner (1st ed., pp. 1 - 24 ). Stuttgart: Schäffer-Poeschel.

Conboy , K. ( 2009 ). Agility from first principles: reconstructing the concept of agility in information systems development . Information Systems Research , 20 ( 3 ), 329 - 354 .

Fowler , M. , & Highsmith , J. ( 2001 ). The Agile Manifesto . Software Development , 9 ( 8 ), 28 - 35 .

Höhn , R. , & Höppner , S. ( 2008 ). Das V-Modell XT : Grundlagen, Methodik und Anwendungen (1st ed.) Springer Publishing Company, Incorporated.

Kerzner , H. ( 2009 ). Project Management: a Systems Approach to Planning, Scheduling, and Controlling (10th ed.) John Wiley & Sons.

Larman , C. , & Basili , V. R. ( 2003 ). Iterative and Incremental Development: A Brief History . IEEE Computer, 36 ( 6 ), 47 - 56 .

Maruping , L. M. , Venkatesh , V. , & Agarwal , R. ( 2009 ). A Control Theory Perspective on Agile Methodology Use and Changing User Requirements . Information Systems Research , 20 ( 3 ), 377 - 399 .

Mertens , P. ( 1999 ). Operiert die Wirtschaftsinformatik mit den falschen Unternehmenszielen? -15 Thesen . In J. Becker, W. König , R. Schütte , O. Wendt & S. Zelewski (Eds.), Wirtschaftsinformatik und Wissenschaftstheorie: Bestandsaufnahme und Perspektiven (pp. 379 - 392 ) Gabler Verlag.

Moss , L. ( 2009 ). Beware of Scrum Fanatics On DW/BI Projects. EIMI , 3 ( 3 )

Moss , L. , & Atre , S. ( 2003 ). Business Intelligence Roadmap: The Complete Project Lifecycle for Decision-Support Applications Addison-Wesley Professional .

Österle , H. , Becker , J. , Frank , U. , Hess , T. , Karagiannis , D. , Krcmar , H. , et al. ( 2010 ). Memorandum zur gestaltungsorientierten Wirtschaftsinformatik . Schmalenbachs Zeitschrift Für Betriebswirtschaftliche Forschung , 62 ( 9 ), 662 - 672 .

Project Management Institute. ( 2008 ). A Guide to the Project Management Body of Knowledge (4th ed.) Project Management Institute .

Project Management Methodologies. (n.d.) . Retrieved July 15 , 2012 , from http://infolific.com/technology/methodologies/

Ross , S. A. ( 1995 ). Uses, Abuses, and Alternatives to the Net-Present-Value Rule . Financial Management , 24 ( 3 ), 96 - 102 .

Wysocki , R. K. ( 2011 ). Effective Project Management: Traditional , Agile, Extreme (6th ed.) John Wiley & Sons.