=Paper=
{{Paper
|id=Vol-1686/WSSSPE4_paper_10
|storemode=property
|title=Sustainable & Productive: Improving incentives for quality software
|pdfUrl=https://ceur-ws.org/Vol-1686/WSSSPE4_paper_10.pdf
|volume=Vol-1686
|authors=Michael Heroux
}}
==Sustainable & Productive: Improving incentives for quality software==
https://ceur-ws.org/Vol-1686/WSSSPE4_paper_10.pdf
Sustainable & Productive: Improving Incentives for
Quality Software
Michael A. Heroux
Sandia National Laboratories, Albuquerque, NM, E-mail:maherou@sandia.gov
Saint John’s University, Collegeville, MN, E-mail:mheroux@csbsju.edu
Abstract—Computational Science and Engineering (CSE) soft- software quality1 , competitive elements must be added that are
ware can benefit substantially from an explicit focus on qual- correlated with increased quality.
ity improvement. This is especially true as we face increased Fortunately, direct evaluation of software artifacts by rank
demands in both modeling and software complexities. At the
same time, just desiring improved quality is not sufficient. We and tenure committees, program managers and publishers is
must work with the entities that provide CSE research teams not required. There are other metrics that, when assessed, will
with publication venues, funding, and professional recognition provide incentive for increasing quality:
in order to increase incentives for improved software quality. In • Funding agencies: For several years, funding agen-
fact, software quality is precisely calibrated to the expectations,
explicit and implicit, set by these entities. We will see broad
cies have requested the submission of data management
improvements in sustainability and productivity only when pub- plans [1], [2]. More recently, a first-of-a-kind software
lishers, funding agencies and employers raise their expectations productivity and sustainability plan was included in a US
for software quality. CSE software community leaders, those who Department of Energy Funding Opportunity Announce-
are in a position to inform and influence these entities, have a ment. This new required element is a simple request for
unique opportunity to broadly and positively impact software
quality by working to establish incentives that will spur creative
information, asking proposal teams to describe how they
and novel approaches to improve developer productivity and produce and sustain their software products.
software sustainability. • Publishers: Reproducibility of scientific results has be-
come a major concern for many publishers. Expectations
for reproducibility are increasing, as are the number
of journals and conference proceedings that offer ar-
tifact and results reviews. Any increased expectations
I. I NTRODUCTION for reproducible computational results have direct impact
on the importance of higher quality software processes,
documentation, source code management and more; all
Funding agencies, publishers and employers play an essen- of which have a strong positive impact on programmer
tial role in determining expected behavior from the compu- productivity and software sustainability when measured
tational science and engineering (CSE) research community. over a sufficiently long time span.
Publishers use a peer review process to ensure that published • Employers: Employers can provide incentives recogniz-
content is accurate, complete and relevant. Funding agencies ing those whose work results in high-quality software.
place expectations on how research is conducted, requiring, The exact metrics need to be adapted depending on
for example, that research teams perform work within certain the situation. Metrics for industry, research labs and
ethical norms. Employers determine promotion, rank and universities will necessarily be distinct, and can either be
tenure and other professional recognition. The value, both observed directly (most feasible for industry) or indirectly
implicit and explicit, that these entities place on software (such as the acceptance of a peer-reviewed paper to a
quality plays a crucial role in the thousands of decisions that software journal) for research labs and universities.
CSE community members make on a daily basis regarding the II. C REATING I NCENTIVES FOR S USTAINABILITY AND
real value of software quality. P RODUCTIVITY
CSE software teams have strong desires to improve devel- Expectations from CSE software clients provide a means to
oper productivity and software sustainability. At the same time, give intrinsic value to virtuous behaviors that are otherwise
competition to publish, receive funding, satisfy project goals
1 Many articles have been written on the characteristics of software quality.
and attain professional promotion and recognition means that
The ISO 9126-1 software quality model definition (see http://www.sqa.net/
the amount of time and resources they can apply to improve iso9126.html for details) lists six main characteristics. For our purposes, we
quality is determined by how much value they can obtain from do not analyze correlations between specific quality characteristics and the
the investment as part of this competition. In order to increase incentives we discuss. Instead we argue that improving incentives will have
a positive impact on all quality characteristics since the incentives represent
This work is licensed under a CC-BY-4.0 license. a holistic approach to improvement.
difficult to reward. CSE software teams certainly want to do Reproducibility of scientific results has become a major
a good job of being productive and producing sustainable concern for many publishers. Numerous highly visible arti-
software tools. But the drive to produce science results on a cles [8], [9], [10], [11] have highlighted the poor result of
short time scale often leads to shortcuts and lack of investment reproducibility efforts. Some community members [12], [13]
up front that is necessary for sustainable software and long- argue that openness and transparency are important ways to
term productivity. Given the demands to provide a computa- provide incentives for scientists to improve reproducibility.
tional answer as soon as possible, especially in a competitive Such a focus also placed increased value on the quality of
environment, other activities such as careful design, efficient software artifacts and processes. Other articles argue that
implementation and thorough software testing environments repeatability as a core value is the fundamental issue in CSE
are often compromised. publications [14], [15].
The connection between these clients and sponsors (publish- The ACM Transactions on Mathematical Software (TOMS)
ers, funding agencies and employers) and science teams who initiated a Replicated Computational Results (RCR) re-
use computation as part of their scientific endeavors can and view process during which a dedicated reviewer is tasked
must be leveraged to provide intrinsic value to other activities with replicating the computational results of a submitted
that can improve sustainability and productivity. manuscript [16], [17], [18]. One of the first author responses
when preparing for the RCR review was a remark that the
A. Funding Agencies
version of software used to generate the computational results
Funding agencies have typically placed very high value on for the paper was tagged so that it could be clearly identified
scientific productivity and impact. In principle this emphasis for the RCR review, as well as any time in the future.
should provide strong incentive to produce high quality soft-
ware. However, often the time cycles for producing science C. Employers
results are short, and the competitive drive to produce results
Industrial employers already have a reward structure in
makes software investments that have longer time cycles out
place that, at least indirectly and evidenced by the large
of scope. This reality was recognized by the US National
volume of industry-focused software literature and highly-
Science Foundation and spurred its development of the SI2
sought speakers, recognizes the value of software quality. This
program [3]. More recently the US Department of Energy
is less true the closer a software business is to the scientific and
(DOE) has developed several funding opportunities that ex-
engineering domains. From the author’s experience working
plicitly focus on software productivity and sustainability:
with commercial, scientific and engineering software compa-
• The 2014 Funding Opportunity Announcement (FOA)
nies, the majority have no explicit software lifecycle model
for Scientific Software Developer Productivity, which and seldom give recognition, except through informal means,
resulted in the establishment of the IDEAS Project [4]. to developers who produce higher quality software products.
While many DOE projects use funds to improve software Research laboratories and university software teams place even
quality, the work is always done as part of achieving less emphasis on employee recognition for software quality. In
another goal. The IDEAS project is a first-of-a-kind fact, there is often a negative opinion about staff members who
effort that enables dedicated research funding for software spend too much time working on software [14], with more
methodologies as a pursuit in its own right. recognition going to staff who use software to do research but
• The DOE Biological and Environment Research (BER)
spend little time producing or reviewing software products.
Office FOA on the Energy-Water-Land Nexus from Development of employee incentives for quality software
February 2016 [5], included a first-of-a-kind requirement can be challenging. Any reward based on direct software met-
for proposals to contain a software productivity and rics can lead to skewed behavior that may actually be counter-
sustainability plan. productive. For example, rewarding a developer for having
• The BER Accelerated Climate Modeling for Energy
few defects can lead to low code generation rates for fear
(ACME) Software Engineering Initiative is a large project of introducing a defect, avoiding challenging programming
whose express goal is to improve the quality of the assignments or similar behaviors that reduce overall productiv-
ACME source code base, train ACME scientists in more ity. Instead, employers can look to recognition from funding
effective software development, and introduce improve agencies and publishers, and assess the overall impact of a
software processes and tools into the ACME project. faculty member’s or research scientist’s software contributions,
B. Publishers with the same regard as for other publication and professional
activities.
At first glance, the role of publishers in improving software
quality may not be apparent, but by raising expectations for in-
III. A T IME OF D ISRUPTIVE C HANGE
dependent reproducibility of computational results, publishers
provide one of the strongest incentives for CSE software teams The CSE community is presently in an era of disruptive
to invest in new and effective software quality methodologies. changes. Similar to the time period when we transitioned
Furthermore, the community incentive at the present time for from sequential and vector computers to distributed memory
improving reproducibility is extremely high [6], [7]. parallel computers, system architecture is changing, driven by
the stagnation of hardware clock speeds, resulting in exponen- informing and shaping funding policy and establishing em-
tial increases in available concurrency. Performance sensitive ployer compensation and recognition systems–have a unique
applications must expose proportional levels of concurrency in opportunity to influence progress. If we want to improve
order to stay on the new commodity curves of threading and software quality, we must change expectations from funding
vectorization scalability. agencies, publishers and employers.
In addition, the tremendous growth in performance potential
ACKNOWLEDGMENT
leads to increased opportunities for multi-scale and multi-
physics coupling. Coupling of previously independent codes Sandia is a multiprogram laboratory operated by Sandia
demands increased collaboration across historical separate Corporation, a wholly owned subsidiary of Lockheed Mar-
efforts. Projects become more inter-disciplinary, distributed tin, for the United States Department of Energy’s National
(since areas of expertise are typically not all in one group Nuclear Security Administration under contract DE-AC04-
or geographic location), and larger. 94AL85000.
All of these disruptions can be better managed and exploited The author thanks the United States Department of Energy,
by having more deliberate efforts to improve software sustain- Office of Science, Advanced Scientific Computing Research
ability and productivity. In fact, now is the best time to invest (ASCR) and Biological and Environmental Research (BER)
in software quality. The stakes are higher than ever. Programs for IDEAS Project funding.
IV. T OWARD S USTAINABLE AND P RODUCTIVE R EFERENCES
[1] US National Science Foundation, “Data Management Guidance for CISE
Numerous studies have shown that activities such as code Proposals and Awards,” http://www.nsf.gov/cise/cise dmp.jsp.
review and test driven development [19] are universally valu- [2] U. S. Department of Energy Office of Science Office of Scientific
able. Furthermore, explicit life cycle models [20], including Computing Research, “Statement on Digital Data Management,” http:
//science.energy.gov/funding-opportunities/digital-data-management/.
models for refactoring legacy codes [21], [22], are also ex- [3] US National Science Foundation, “Software Infrastructure for Sustained
tremely beneficial, especially for larger teams. Productivity Innovation (SI2: SSE & SSI),” http://www.nsf.gov/pubs/2016/nsf16532/
models [23], which attempt to characterize and measure the nsf16532.htm.
[4] “The IDEAS Project,” http://ideas-productivity.org, 2016.
value and cost attributes of a project, can also be extremely [5] U. S. Department of Energy Office of Science Office of Biological
helpful, especially in the face of disruptive changes in com- and Environmental Research, “Regional and Global Climate Mod-
puting platforms and application requirements. eling and Integrated Assessment Research: An Integration Frame-
work for Multi-Model, U.S. Regional Climate Evaluation that In-
The CSE software community can benefit greatly from an corporates Local Human Influences for Research at the Energy-
improved focus on sustainability and productivity, especially at Water-Land Nexus,” http://science.energy.gov/∼/media/grants/pdf/foas/
this time of increased complexity and disruptive change. There 2016/SC FOA 0001531.pdf, Funding Opportunity Number: DE-FOA-
0001531.
are many valuable methodologies, tools and best practices that [6] American Association for the Advancement of Science, “III
the CSE community can explore, adapt, adopt and develop Arnold workshop: Modeling and code,” http://www.aaas.org/event/
to improve developer productivity and software sustainability. iii-arnold-workshop-modeling-and-code.
[7] Supercomputing 2016, “Student Cluster Competition reproducibility
However, for many teams the incentive for investment is initiative winner,” http://sc16.supercomputing.org/studentssc/
not strong enough when compared to other demands; other scc-reproducibility-initiative-winner/.
priorities rank higher. [8] C. G. Begley, G. Robertson, K. Kaitin, K. Asadullah, L. Ellis, P. Sharp,
and S. Begley, “In cancer science, many discoveries don’t hold up
Funding agencies can provide incentives by explicitly re- — Reuters,” 2014. [Online]. Available: http://www.reuters.com/article/
questing software productivity and sustainability plans in pro- 2012/03/28/us-science-cancer-idUSBRE82R12P20120328
posals, and by funding projects that pursue software method- [9] D. H. Bailey, “Fooling the masses: Reproducibility in high-
performance computing,” 2014. [Online]. Available: https://www.xsede.
ologies research and a fundamental activity, not just as part of org/documents/659353/703287/xsede14 bailey.pdf
pursuing another goal. Publishers can play a critical role by [10] J. Arrowsmith, “Trial watch: Phase II failures: 2008–2010,” Nat Rev
increasing expectations for reproducibility, and transparency Drug Discov, vol. 10, no. 5, pp. 328–329, May 2011. [Online].
Available: http://dx.doi.org/10.1038/nrd3439
of software processes used to generate computational results. [11] B. Owens, “Reliability of ’new drug target’ claims called into
Employers, especially research laboratories and universities, question : Nature News Blog,” 2011. [Online]. Available: http:
must start to explicitly value software contributions with the //blogs.nature.com/news/2011/09/reliability of new drug target.html
[12] D. H. Bailey and J. M. Borwein, “Set the Default to
same respect as other research output, and give recognition for “Open”: Reproducible Science in the Computer Age—David H.
faculty and staff who receive publication and funding agency Bailey,” 2014. [Online]. Available: http://www.huffingtonpost.com/
recognition for quality software. david-h-bailey/set-the-default-to-open-r b 2635850.html
[13] V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider,
Funding agencies, publishers and employers can and must and W. Stein, “Setting the Default to Reproducible: Reproducibility
play a key and truly unique role in improving developer in Computational and Experimental Mathematics,” 2013. [Online].
productivity and software sustainability by increasing quality Available: http://icerm.brown.edu/html/programs/topical/tw12 5 rcem/
icerm report.pdf
expectations. In fact, it is not too strong to say that software [14] C. Collberg and T. A. Proebsting, “Repeatability in computer systems
quality will be calibrated precisely to the expectations of these research,” Commun. ACM, vol. 59, no. 3, pp. 62–69, Feb. 2016.
entities. Because of the important roles these entities play [Online]. Available: http://doi.acm.org/10.1145/2812803
[15] S. Krishnamurthi and J. Vitek, “The real software crisis: Repeatability
in providing incentives for improved software quality, CSE as a core value,” Commun. ACM, vol. 58, no. 3, pp. 34–36, Feb. 2015.
software leaders–those who are engaged in editorial work, [Online]. Available: http://doi.acm.org/10.1145/2658987
[16] M. A. Heroux, “Editorial: ACM TOMS replicated computational results
initiative,” ACM Trans. Math. Softw., vol. 41, no. 3, pp. 13:1–13:5, Jun.
2015. [Online]. Available: http://doi.acm.org/10.1145/2743015
[17] J. M. Willenbring, “Replicated computational results (RCR) report for
BLIS: A framework for rapidly instantiating BLASx functionality,” ACM
Trans. Math. Softw., vol. 41, no. 3, Sep. 2015.
[18] F. G. Van Zee and R. A. van de Geijn, “Blis: A framework for rapidly
instantiating blas functionality,” ACM Trans. Math. Softw., vol. 41, no. 3,
Sep. 2015.
[19] K. Beck, Test Driven Development. Addison Wesley, 2003.
[20] R. A. Bartlett, M. A. Heroux, and J. M. Willenbring, “Overview of
the TriBITS lifecycle model: A lean/agile software lifecycle model for
research-based computational science and engineering software,” in E-
Science (e-Science), 2012 IEEE 8th International Conference on, Oct
2012, pp. 1–8.
[21] M. Feathers, Working Effectively with Legacy Code. Addison Wesley,
2005.
[22] M. Fowler, Refactoring (Improving the Design of Existing Code).
Addison Wesley, 1999.
[23] R. van Solingen, V. Basili, G. Caldiera, and H. D. Rombach, Goal
Question Metric (GQM) Approach. John Wiley and Sons, Inc., 2002.