In this paper, we describe the research proposal for an approach for Evidence-Based Software Portfolio Management; a new way to help software companies in steering their software portfolio's based on cost, duration, defects found on the one hand and stakeholder satisfaction and perceived value on the other. The research approach is based on instruments such as a Cost / Duration Matrix, the identification of success and failure factors for software projects, and the collection of data on finalized software projects from portfolios of different companies in a research repository.
Where many other studies use either a quantitative approach (e.g. analyze core metrics) or a qualitative approach (e.g. perform surveys or interviews) to analyze software projects, we combine both ways and look at a company’s software project portfolio from a holistic point of view. The goal of our research is to combine a quantitative, data-driven approach on analysis of finalized software project portfolios with a qualitative, survey-based approach in order to identify factors related to project success and failure, in Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes. combination with an approach to measure and analyze stakeholder satisfaction and perceived value of software projects.
In this paper, we describe the research proposal for the development of evidence-based software portfolio management as a practical approach on organizing and decision-making with regard to large portfolios of software projects in information-intensive companies. In particular, the main contributions in the current state of the research are: 1. 2. 3. 4. 5.
We propose a Cost / Duration Matrix as an instrument for analysis of good practice and bad practice in large, companywide portfolios of software projects.
We identify success and failure factors for software projects, based on analysis of a large subset of data of finalized software projects from three different companies.
We analyze series of software releases in order to identify additional factors that contributes to projects being best-inclass.
We propose a light-weight value measurement technique based on quantitative analysis and post-project interviews. We provide data of industrial software projects on a standardized set of metrics: project size, cost, duration, and defects. We contrast these core metrics with collected data on stakeholder satisfaction and perceived value, and look for links between them.
The remainder of this paper is organized in the following way: In Section 2 we outline relevant prior work. In Section 3 we describe our research objectives and questions. In Section 4 our research approach is described. Section 5 is about the most important metrics with regard to our research and Section 6 is about the data analysis methods and techniques that we apply. In Section 7 we evaluate validity threats. Finally, Section 8 includes a summary of the current status of our research and planned next steps.
In this section we describe a brief survey of the background of our
research area and related work with regard to our research subject.
A common idea of many research performed in the former
millennium is that success and failure of software projects are
interconnected with process-based activities: in other words, follow
the process and success will come [
Shepperd argues that “the primary goal of more accurate cost
prediction systems remains largely unachieved” [
Varieties of value-based software engineering are examined [
We define as our research objective: “an integrated approach that links existing measurements based on project size, cost, durations, and defects of software projects with a limited set of relatively easy to collect additional measures on stakeholder satisfaction and perceived value, and additional qualitative research on the backgrounds of project success and failure”.
Based on the above we define the following research questions: RQ1: In what way are project size, project cost, project duration, and process quality (measured in number of defects) in a software project portfolio correlated? RQ2: What factors can be found that influence a company’s software project portfolio in a positive or negative way? RQ3: What factors can be found that characterize ‘best-in-class’ software projects? RQ4: Are function points (FPs) compatible with story points (SPs) on agile projects? RQ5: Can a statistical, empirical, evidence-based pricing approach for software engineering, be used as a single instrument (without a connection with expert judgment), in distributed environments to create cost transparency and performance management of software project portfolios? RQ6: In what way are project size, project cost, project duration, and process quality (measured in number of defects) in a software project portfolio correlated with stakeholder satisfaction of finalized software projects?
RQ6.1: Is process quality, measured in number of defects found during a project, an early indicator for stakeholder satisfaction? RQ7: In what way are project size, project cost, project duration, and process quality (measured in number of defects) in a software project portfolio correlated with perceived value of finalized software projects?
RQ7.1: Is project size, measured in function points (FPs), an early indicator for perceived value of finalized software projects?
In this section we describe the approach that we use in order to
answer the research questions as stated above. Because we perform
our research in close cooperation with software companies (to be
read as information-intensive companies, such as banks, telecom
companies, governmental organizations), we set-up the research in
a way that fits with practice. Where appropriate, we perform case
studies [
A precondition that limits our research approach is the fact that we perform research in real, live organizational environments. Therefore the approach must not interfere with the daily operation of the studied software projects. Surveys should impose limited burden on people, and analysis is usually to be useful for improvement purposes in daily operations.
For our research we make use of an existing data set of 352 finalized software projects from three different organizations. This research repository is collected over a period of four years, receding our research. Table 1 gives an overview of the organization of this research repository. During our research we continue the collection of data of finalized software projects; the research repository will mature during the development of the research, both in number of projects and in applied metrics.
Based on the collected metrics as inventoried in Table 1 we calculate three key performance indicators.
1. 2. 3.
For all three indicators we use project size (FPs) as the weighting factor (instead of number of projects). In order to measure stakeholder satisfaction and perceived value we ask all stakeholders of a finalized software project (e.g. project manager, business representative, product owner, business analyst, scrum master, developer, and tester) to rate scores for both metrics on a 5-point scale. Stakeholder satisfaction is measured for both satisfaction with regard to the projects process and the deliverables of the
Year when a project was finalized; the following years Go Live were applicable: 2008 (32), 2009 (59), 2010 (81), 2011 (131), 2012 (41), 2013 (10).
Customers business sector; the following BD were applicable: Finance & Risk (54), Internet & Mobile (54), Payments (50), Client & Account Management (incl. CRM systems) (46), Savings & Loans (40), Organization (incl. HRM) (31), Call Centre Solutions (21), Mortgages (21), Data warehouse & BI (18), Front Office Solutions (17).
Primary used programming language; the following PPL were applicable: JAVA (154), .NET (59), COBOL (55), ORACLE (29), SQL (9), 3GL (8, unknown was what specific languages were applicable here), Visual Basic (6), RPG (6), FOCUS (5), PowerBuilder (5), PRISMA (4), MAESTRO (3). In the analysis 4th Generation (1), PL1 (1), JSP (1), C++ (1), Clipper (1), Document (1), PL/SQL (1), Siebel (1) and Package (1, unknown what specific language was applicable) were referred at as Other.
Classification of the used delivery model; two DM were applicable: Structured (e.g.
Waterfall) (307), and Agile (Scrum) (45). One project reported as DM RUP is included in the analysis of Structured.
Classification of the development; the following DC were applicable: New development (173), Major enhancement (25-75% new) (124), Minor enhancement (5-25% new) (27), Conversion (28).
Characteristics on a specific project (multiple keywords could be mapped on one project, on one project no keyword was mapped); the following keywords were applicable: Singleapplication (270), Business driven (150), Release-based (one application) (144), Once-only project (122), Phased project (part of program) (65), Fixed, experienced team (62), ), Technology driven (58), Steady heartbeat (49), Dependencies with other systems (41), Migration (35), Rules & Regulations driven (33), Multi-application release (21), Many team changes, inexperienced team (17), Package with customization (16), Legacy (15), Security (14), Pilot; Proof of Concept (10), Bad relation with external supplier (9), New technology, framework solution (3), Package off-the-shelf (1).
For every project in the repository the measurements indicated in the table below are inventoried.
Size of a project in Function Points (FPs).
Duration of a project in Months; measured from the start of Project Initiation to (technical) Go Live.
Cost of a project in Euros; measured from the start of Project Initiation to (technical) Go Live.
Effort spent in a project in Person Hours (PHRs); measured from the start of Project Initiation to (technical) Go Live.
The number of errors or faults found in a project from System Integration Test to (technical) Go Live. Not for all projects defects were administrated; for 172 projects defects info was recorded in the repository. *No occurrences are indicated for the 5 Measures, due to the fact that these are different for every measured project. project (the product). Perceived value is measured for four aspects: a company’s customer, financial, internal process, and innovation.
In our research we make use of different data analysis techniques, as described in the following paragraphs.
The most important data analysis instrument that we use is a socalled Cost / Duration Matrix (see Figure 1). This matrix is a model based on power regression of project cost (Euros) versus project size (FPs) and project duration (months) versus project size (FPs). For both regressions the percentage deviation from the mean is calculated for each software project. This percentage deviation is for both cost and duration plotted in a plotter chart; the Cost / Duration Matrix.
As Figure 1 visualizes the matrix shows larger or smaller dots,
depending on the size in FPs of a specific project. A color ranging
from blue to red indicates the process quality (number of defects
per FP) of each project, where blue stand for a good process quality
and red for a bad quality (meaning more than average defects per
FP for a specific project). The Cost / Duration Matrix (see Figure
1) is used as a model to visualize and assess the performance in
terms of cost, duration and quality of software projects based on
four quadrants that describe specific characterizations [
Cost over Time (bottom right): In this quadrant software projects are reported that scored better than the average of the total repository (or a specific subset of the repository) for cost, yet worse than average for duration.
Bad Practice (bottom left): This quadrant holds software projects that scored worse than average of the total repository (or a specific subset of the repository) for bot cost and duration.
Cost over Time (upper left): In this quadrant software projects are plotted that scored better than average of the total repository (or a specific subset of the repository) for duration, and worse than average for project cost.
Keep in mind that the underlying nominator for all software projects in the Cost / Duration Quadrant is functional size (measured in FPs). Due to this we can compare the performance in terms of cost, duration, defects found, satisfaction, and value of projects with different sizes with each other.
We develop a Software Project Benchmark Tool that enables the functionality for practitioners in industry to benchmark a subset of finalized software projects against our research repository. The tool is based on the Cost / Duration Matrix and makes it possible for measurement practitioners in industry to upload a subset of finalized software projects from their own organization and to benchmark the performance in terms of cost, duration, and defects found with that of comparable projects in our research repository.
In order to add new metrics such as Stakeholder Satisfaction and Perceived Value of finalized software projects to our research repository, we build a survey questionnaire that is send to stakeholders once software projects are finalized. We add the metrics resulting from this questionnaire to our research repository and relate the outcomes with software projects in the four quadrants of the Cost / Duration Matrix.
Finally we develop and describe Evidence-Based Software Portfolio Management, as an approach for software companies to prioritize software activities within their software project portfolio, based on quantification of project size, cost, duration, defects found, stakeholder satisfaction, and perceived value of finalized software projects. We test this approach as a whole in a different information-intensive organization to examine whether the outcomes correlate with those companies that are already available in our research repository, and to analyze whether the approach can be used in practice as a valuable addition to tools already in place for a software project portfolio management capability in organizations.
With regard to validity constraints we assess the following threats. We use function point analysis (FPA) as a way to normalize software projects and to make it possible to compare performances of projects with different sizes. We use functional documentation as a source for FPA. A consequence is that low quality documentation can lead to low quality FPAs. However, we thoroughly review all sets of documentation on completeness and correctness and have FPAs performed by experienced, and in many cases certified, experts. FPAs are reviewed by different experts than the ones that performed the count itself to prevent from bias. With regard to data quality we argue that all project data is reviewed by the applicable project managers. All data is discussed with the applicable company management and the financial controller. By normalizing all project data with the functional size in FPs we warrant internal validity, the extent to which a causal conclusion is based on our study. Due to this we can objectively compare performances of all software projects, in order to minimize systematic error. The effect of outliers is limited and the risk on bias is mitigated responsibly based on the diversity of projects and business domains within each subject company, the number of software projects, and the fact that we measure and analyze software project portfolios as a whole in an empirical way.
At this moment the following research results are in place:
We analyzed a dataset containing 352 finalized software projects,
with the goal to discover what factors affect software project
performance, and what actions can be taken to increase project
performance when building a software project portfolio. The
software projects are classified in four quadrants of a cost/duration
matrix: analysis is performed on factors that are strongly related to
two of those quadrants, Good Practices and Bad Practices. A
ranking is performed on the factors based on statistical significance,
resulting in an inventory of ‘what factors should be embraced when
building a project portfolio?’ (Success Factors), and ‘what factors
should be avoided when doing so?’ (Failure Factors). This research
result is documented in a paper that was accepted at ICSE 2014,
SEIP-track [
We aimed to identify distinguishing factors in software releases.
For this purpose we analyzed the metrics of 26 software projects.
These projects were release-based deliveries from two stable,
experienced development teams in a Banking company. During the
measurement period both teams transformed from a plan-driven
delivery model (waterfall) to an agile approach (Scrum). Overall,
we observed that these small release-based projects differ largely
from non-release-based projects. Our research indicates that a
combination of release-based working, a fixed and experienced
development team, and a steady heartbeat contribute to
performances that can be characterized as best practice. This
research result is documented in a paper that was accepted at
IWSM-Mensura 2013 [
A case study that replicates the research above, with additional qualitative research on a series of best-in-class releases from another Telecom company is described in a paper that is to be submitted.
In order to find differences and similarities between two many used
size metrics we replicated a study on the relation between story
points and function points performed in 2011 by a group of
Brazilian researchers. We used data collected in a Banking
organization. Based on a statistical correlation test we conclude that
it appears too early to make generic claims on the relation between
function points and story points; in fact FSM-theory seems to
underpin that such a relationship is a spurious one. The results of
this research were published in a paper that was accepted at
WETSoM 2014 [
We analyzed how a medium-sized west-European telecom
company experienced a worsening trend in performance, indicating
that the organization did not learn from history, in combination with
much time and energy spent on preparation and review of project
proposals. In order to create more transparency in the supplier
proposal process a pilot was started on Functional Size
Measurement pricing (FSM-pricing). In our research we evaluated
the implementation of FSM-pricing in the software engineering
domain of the company, as an instrument useful in the context of
software management and supplier proposal pricing. We found that
a statistical, empirical, evidence-based pricing approach for
software engineering, as a single instrument (without a connection
with expert judgment), can be used in distributed environments to
create cost transparency and performance management of software
project portfolios. A research paper on our research results is
accepted at ESEM 2015 [
The following research topics are to be studied in the remaining part of the research period:
We developed a tool based on the cost/duration matrix as used in
[
As a point on the horizon we will focus on research on the quantification of Stakeholder Satisfaction and Perceived Value of software projects, related to the cost/duration matrix that we defined in earlier research (see Figure 1). We enrich this model by mapping Stakeholder Satisfaction and Perceived Value with regard to a company’s customers, financial, internal process and innovation aspects to cost, duration and quality of finalized software projects. A paper on this subject, including a survey on five finalized projects in a Telecom company is in preparation.
I thank Arie van Deursen and Rini van Solingen for their great support and work as advisors for my PhD activities. Furthermore I thank all companies that support our research for their generosity to allow us to use company data for research purposes. a small software organisation," in IEEE Proceedings of the Australian Software Engineering Conference, 2004.