-Computational Science and Engineering (CSE) software can benefit substantially from an explicit focus on quality improvement. This is especially true as we face increased demands in both modeling and software complexities. At the same time, just desiring improved quality is not sufficient. We must work with the entities that provide CSE research teams with publication venues, funding, and professional recognition in order to increase incentives for improved software quality. In fact, software quality is precisely calibrated to the expectations, explicit and implicit, set by these entities. We will see broad improvements in sustainability and productivity only when publishers, funding agencies and employers raise their expectations for software quality. CSE software community leaders, those who are in a position to inform and influence these entities, have a unique opportunity to broadly and positively impact software quality by working to establish incentives that will spur creative and novel approaches to improve developer productivity and software sustainability.
Funding agencies, publishers and employers play an essential role in determining expected behavior from the computational science and engineering (CSE) research community. Publishers use a peer review process to ensure that published content is accurate, complete and relevant. Funding agencies place expectations on how research is conducted, requiring, for example, that research teams perform work within certain ethical norms. Employers determine promotion, rank and tenure and other professional recognition. The value, both implicit and explicit, that these entities place on software quality plays a crucial role in the thousands of decisions that CSE community members make on a daily basis regarding the real value of software quality.
CSE software teams have strong desires to improve developer productivity and software sustainability. At the same time, competition to publish, receive funding, satisfy project goals and attain professional promotion and recognition means that the amount of time and resources they can apply to improve quality is determined by how much value they can obtain from the investment as part of this competition. In order to increase This work is licensed under a CC-BY-4.0 license. software quality1, competitive elements must be added that are correlated with increased quality.
Fortunately, direct evaluation of software artifacts by rank and tenure committees, program managers and publishers is not required. There are other metrics that, when assessed, will provide incentive for increasing quality:
Funding agencies: For several years, funding
agencies have requested the submission of data management
plans [
Publishers: Reproducibility of scientific results has become a major concern for many publishers. Expectations for reproducibility are increasing, as are the number of journals and conference proceedings that offer artifact and results reviews. Any increased expectations for reproducible computational results have direct impact on the importance of higher quality software processes, documentation, source code management and more; all of which have a strong positive impact on programmer productivity and software sustainability when measured over a sufficiently long time span.
Employers: Employers can provide incentives recognizing those whose work results in high-quality software. The exact metrics need to be adapted depending on the situation. Metrics for industry, research labs and universities will necessarily be distinct, and can either be observed directly (most feasible for industry) or indirectly (such as the acceptance of a peer-reviewed paper to a software journal) for research labs and universities. II. CREATING INCENTIVES FOR SUSTAINABILITY AND
PRODUCTIVITY
Expectations from CSE software clients provide a means to give intrinsic value to virtuous behaviors that are otherwise 1Many articles have been written on the characteristics of software quality. The ISO 9126-1 software quality model definition (see http://www.sqa.net/ iso9126.html for details) lists six main characteristics. For our purposes, we do not analyze correlations between specific quality characteristics and the incentives we discuss. Instead we argue that improving incentives will have a positive impact on all quality characteristics since the incentives represent a holistic approach to improvement. difficult to reward. CSE software teams certainly want to do a good job of being productive and producing sustainable software tools. But the drive to produce science results on a short time scale often leads to shortcuts and lack of investment up front that is necessary for sustainable software and longterm productivity. Given the demands to provide a computational answer as soon as possible, especially in a competitive environment, other activities such as careful design, efficient implementation and thorough software testing environments are often compromised.
The connection between these clients and sponsors (publishers, funding agencies and employers) and science teams who use computation as part of their scientific endeavors can and must be leveraged to provide intrinsic value to other activities that can improve sustainability and productivity.
Funding agencies have typically placed very high value on
scientific productivity and impact. In principle this emphasis
should provide strong incentive to produce high quality
software. However, often the time cycles for producing science
results are short, and the competitive drive to produce results
makes software investments that have longer time cycles out
of scope. This reality was recognized by the US National
Science Foundation and spurred its development of the SI2
program [
The DOE Biological and Environment Research (BER)
Office FOA on the Energy-Water-Land Nexus from
February 2016 [
The BER Accelerated Climate Modeling for Energy (ACME) Software Engineering Initiative is a large project whose express goal is to improve the quality of the ACME source code base, train ACME scientists in more effective software development, and introduce improve software processes and tools into the ACME project.
At first glance, the role of publishers in improving software
quality may not be apparent, but by raising expectations for
independent reproducibility of computational results, publishers
provide one of the strongest incentives for CSE software teams
to invest in new and effective software quality methodologies.
Furthermore, the community incentive at the present time for
improving reproducibility is extremely high [
Reproducibility of scientific results has become a major
concern for many publishers. Numerous highly visible
articles [
The ACM Transactions on Mathematical Software (TOMS)
initiated a Replicated Computational Results (RCR)
review process during which a dedicated reviewer is tasked
with replicating the computational results of a submitted
manuscript [
Industrial employers already have a reward structure in
place that, at least indirectly and evidenced by the large
volume of industry-focused software literature and
highlysought speakers, recognizes the value of software quality. This
is less true the closer a software business is to the scientific and
engineering domains. From the author’s experience working
with commercial, scientific and engineering software
companies, the majority have no explicit software lifecycle model
and seldom give recognition, except through informal means,
to developers who produce higher quality software products.
Research laboratories and university software teams place even
less emphasis on employee recognition for software quality. In
fact, there is often a negative opinion about staff members who
spend too much time working on software [
Development of employee incentives for quality software can be challenging. Any reward based on direct software metrics can lead to skewed behavior that may actually be counterproductive. For example, rewarding a developer for having few defects can lead to low code generation rates for fear of introducing a defect, avoiding challenging programming assignments or similar behaviors that reduce overall productivity. Instead, employers can look to recognition from funding agencies and publishers, and assess the overall impact of a faculty member’s or research scientist’s software contributions, with the same regard as for other publication and professional activities.
The CSE community is presently in an era of disruptive changes. Similar to the time period when we transitioned from sequential and vector computers to distributed memory parallel computers, system architecture is changing, driven by the stagnation of hardware clock speeds, resulting in exponential increases in available concurrency. Performance sensitive applications must expose proportional levels of concurrency in order to stay on the new commodity curves of threading and vectorization scalability.
In addition, the tremendous growth in performance potential leads to increased opportunities for multi-scale and multiphysics coupling. Coupling of previously independent codes demands increased collaboration across historical separate efforts. Projects become more inter-disciplinary, distributed (since areas of expertise are typically not all in one group or geographic location), and larger.
All of these disruptions can be better managed and exploited by having more deliberate efforts to improve software sustainability and productivity. In fact, now is the best time to invest in software quality. The stakes are higher than ever.
Numerous studies have shown that activities such as code
review and test driven development [
The CSE software community can benefit greatly from an improved focus on sustainability and productivity, especially at this time of increased complexity and disruptive change. There are many valuable methodologies, tools and best practices that the CSE community can explore, adapt, adopt and develop to improve developer productivity and software sustainability. However, for many teams the incentive for investment is not strong enough when compared to other demands; other priorities rank higher.
Funding agencies can provide incentives by explicitly requesting software productivity and sustainability plans in proposals, and by funding projects that pursue software methodologies research and a fundamental activity, not just as part of pursuing another goal. Publishers can play a critical role by increasing expectations for reproducibility, and transparency of software processes used to generate computational results. Employers, especially research laboratories and universities, must start to explicitly value software contributions with the same respect as other research output, and give recognition for faculty and staff who receive publication and funding agency recognition for quality software.
Funding agencies, publishers and employers can and must play a key and truly unique role in improving developer productivity and software sustainability by increasing quality expectations. In fact, it is not too strong to say that software quality will be calibrated precisely to the expectations of these entities. Because of the important roles these entities play in providing incentives for improved software quality, CSE software leaders–those who are engaged in editorial work, informing and shaping funding policy and establishing employer compensation and recognition systems–have a unique opportunity to influence progress. If we want to improve software quality, we must change expectations from funding agencies, publishers and employers.
Sandia is a multiprogram laboratory operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC0494AL85000.
The author thanks the United States Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) and Biological and Environmental Research (BER) Programs for IDEAS Project funding.