=Paper=
{{Paper
|id=Vol-1686/LightningTalkPaper1
|storemode=property
|title=None
|pdfUrl=https://ceur-ws.org/Vol-1686/WSSSPE4_paper_4.pdf
|volume=Vol-1686
}}
==None==
https://ceur-ws.org/Vol-1686/WSSSPE4_paper_4.pdf
Lightning talk: Software Impact Measurement
at the Netherlands eScience Center
Willem van Hage, Jason Maassen, Rob van Nieuwpoort
Netherlands eScience Center
Amsterdam, The Netherlands
http://estep.esciencecenter.nl
{w.vanhage | j.maassen | r.vannieuwpoort}@esciencecenter.nl
Abstract—In this lightning talk we present a new initiative at the topics in eStep, such as software packages, people that were in-
Netherlands eScience Center (NLeSC) that aims to measure soft- volved in the development and use of these software packages,
ware impact, recognizing software as research output. projects in which they were used, and organizations that partici-
pated in these projects. Users can browse, search, and filter the
Index Terms—eScience, software impact, eStep
website. The analytics site that allows visitors to filter the con-
I. INTRODUCTION tents of eStep is illustrated by the screenshot shown in Figure 1.
An example software project page is shown in Figure 2.
The NLeSC is a Dutch funding agency and scientific soft-
The website is automatically generated from metadata rec-
ware development expertise center that supplies specialized
ords that can be submitted added to the website by the develop-
manpower along with research funding for innovative eScience
ers themselves, putting the responsibility at the scientific devel-
projects to all of research. The main goal of the NLeSC is to
opers themselves. Currently, the only contributors to the website
enhance science with information technological innovation. This
are NLeSC employees, but in the near future the website will
is accomplished by alleviating a number of problems that come
become a shared effort by an increasing number of eScience re-
with the adaptation of the scientific process as we know it to in-
search institutes. Increased visibility and software impact meas-
clude new IT. Examples of these issues include missing scien-
urements are incentives for partners to join. We foresee that the
tific software tooling; the lack of skilled scientific software de-
infrastructure will become more federated in the future. We will
velopers; missing specific technological expertise such as big
make the impact measurement technology available to other
data analytics, optimized data handling, and efficient computing;
software catalogs as well.
missing publication fora for scientific software; and the lack ac-
ademic professional assessment by scientific software impact.
Of all these urgent problems, the last might be the most im- III. SOFTWARE IMPACT TESTBED
portant. If academics are assessed by their software, then the ac-
ademic world could eventually reorganize itself so that all the This year, the NLeSC will start with an experimental addi-
other issues are solved. tion to the eStep website to aid scientific developers with the as-
sessment of their software impact. The goal is that each software
package described on the website, and added by the developers
II. ESTEP: THE ESCIENCE TECHNOLOGY PLATFORM themselves, will get an automatically generated dashboard pre-
senting various aspects of the software impact.
To stimulate scientific software development, the NLeSC
There are many different approaches to measure software
has set up the eScience Technology Platform (eStep), which
impact based on different measurable quantities: direct and indi-
consists of three parts: a software catalogue of eScience soft-
rect software (report) citations, software dependency structure,
ware, interfaces, libraries, tools workflows, applications, models
software contribution, forking of the code base, usage in pro-
and data sets; a knowledge base consisting of guides, reports and
jects, endorsements or reviews, and software quality metrics.
recommendations for scientific software development; and doc-
The goal of the software impact measurement website is to
uments describing eScience research such as scientific publica-
tions and demos. serve as a test bed to assess the usefulness and reliability of all
of these different approaches in practice. The NLeSC hopes to
Currently, eStep is used by the NLeSC itself for internal dis-
learn which aspects of software projects are best used to measure
semination of knowledge and for self-evaluation, by NLeSC
software impact, and to use this empirical knowledge to influ-
partners like the funded research institutes to learn about the
ence scientific organizations to start assessing researchers by
portfolio of the NLeSC and new trends in eScience, and by other
these measures. Hopefully this will lead to an increased aware-
users of the eScience software produced by the NLeSC to find
ness at both funding agencies and research institutes that soft-
out how to contribute or whom to contact for more information.
ware drives innovative research, that better software means bet-
To help all of these users to access the information they need
ter research, and that appropriate key performance indicators can
NLeSC launched a new Web site that presents all the interrelated
be used to improve the scientific performance.
This work is licensed under a CC-BY-4.0 license.
Figure 1: The eStep software project analytics webpage (screenshot August 2016).
Figure 2: An example software project description. The screenshot shows the metadata of the MAGMa chemistry tool and
a small part of the description of the tool. (screenshot August 2016).