-Applying modern software engineering to scientific software development has many challenges. These include lack of time or incentives to learn software engineering best practices, a lack of understanding or appreciation of the value of modern software engineering, and a shortage of mechanisms to more broadly change the software engineering culture of a community of researchers in a singular concerted effort. The HydroShare project is a large distributed research software development project that has made significant inroads on these challenges. As the HydroShare project enters its fifth year of NSF funding, we discuss how research scientists and research software engineers from the ten collaborating institutions consistently produce high-quality HydroShare code releases every 2-3 weeks that are formally reviewed and tested.
Applying modern software engineering to scientific software
development has many challenges. These include lack of time
or incentives to learn software engineering best practices, a lack
of understanding or appreciation of the value of modern software
engineering, and a shortage of mechanisms to more broadly
change the software engineering culture of a community of
researchers in a singular concerted effort. The HydroShare project
is a large distributed research software development project that
has made significant inroads on these challenges [
This work is licensed under a CC-BY-4.0 license.
At the onset of the HydroShare project, most of the research scientist collaborators on the project were not clear on the difference between software development and software engineering, and they were not familiar with concepts such as iterative software development, test-driven development, code reviews, and continuous integration. Now, as the HydroShare project enters its fifth year of NSF funding, research scientists and research software engineers from the ten collaborating institutions consistently produce high-quality HydroShare code releases every 2-3 weeks that are formally reviewed and tested. The HydroShare team now understands and embraces the value of modern software engineering; indeed, they understand the time savings of producing high-quality sustainable code at the onset as enabling more time spent on their research and not on the timeconsuming alternative of managing potentially poor quality code had they not embraced modern software engineering.
II. SOFTWARE ENGINEERING BEST PRACTICES AND CULTURE CHANGE FOR DEVELOPING SUSTAINABLE COMMUNITY
SOFTWARE
The HydroShare team has achieved a community rhythm in the continual deployment of high-quality community code releases of HydroShare. The community visibility of this rhythm has served as an incentive for community culture change in that team members are pleased with the resulting code, its evolving significant capabilities, and the efficiency by which new features are tested and integrated as contributed by the collaborating team members. This culture change has naturally incented collaborating researchers to take the time to teach modern software engineering best practices to their graduate and postdoctoral students that time has shown have now also embraced these practices. While host universities have provided the requisite training in computer programming to HydroShare researchers, they have not provided the accompanying instruction on software engineering best practices. However, the HydroShare project has provided numerous faculty, graduate students, postdoctoral students, and even undergraduate students the opportunity to learn modern software engineering best practices first-hand in the absence formal classroom instruction of same.
What is important to convey is this propagation and adoption of software engineering best practices across the HydroShare team is happening organically without the need to force it. It is a success story in that what is referred to as a “community rhythm” herein is in effect like an engine that, once started, sustains itself in part by visibly promoting its own success and efficiencies such that there is no questioning of its uptake by new team members. In other words, adoption of these software engineering best practices becomes the new norm – there is no alternative that new HydroShare team members are ever exposed to. This is especially important to those faculty and students who are early in their careers as it has proven a viable mechanism of getting these individuals on a sustainable software path early on.
As it is beyond the scope of a lightning talk summary paper
to fully describe the mechanism of how the success of
HydroShare’s software engineering is achieved, we refer the reader
to a book chapter titled “HydroShare – A case study of the
application of modern software engineering to a large distributed
federally-funded scientific software development project” that
offers a comprehensive discussion of this work [
A concluding note on the software sustainability of Hydroshare: since 2002, CUAHSI [6] – the primary U.S. hydrology consortium with 130 member universities and international organizations – collaborated in the predecessor to HydroShare called Hydrologic Information System, or HIS [7]. HIS is now maintained by the CUAHSI Water Data Center [8] as its community sustainability model. HydroShare is positioned as the successor to HIS, complementing but not replacing it. When the NSF-funded HydroShare award concludes, it will also be hosted by the CUAHSI Water Data Center – its long-term sustainability ensured by modern software engineering that will readily enable the broader community to continually make novel and useful contributions.
This material is based upon work supported by the USA National Science Foundation (NSF) under awards 1148453 and 1148090; any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. [3] NSF collaborative HydroShare award numbers 1148453 and 1148090, http://www.nsf.gov/awardsearch/showAward?AWD_ID=11484 53/ and http://www.nsf.gov/awardsearch/showAward?AWD_ID=11480 90/. [5] HydroShare GitHub https://github.com/hydroshare/hydroshare/graphs/commitactivity. [8] https://www.cuahsi.org/wdc.