Researchers expect a clear and well-documented experiment from industry experience reports and experimental research papers. All necessary configuration parameters, the source code, the experiment's machine configuration etc. should be documented in such a way that readers can interpret the results of these publication in detail. If this is the case, an interested reader is capable of redoing the experiments stated and verifying the results of others. This ability to properly interpret experiments and even reproduce them is a cornerstone of good scientific practice.
CEUR ceur-ws.org
“Reproducible Research in all sciences is critical to the advancement of knowledge.
It is what enables a researcher to build upon, or refute, previous research
allowing the field to act as a collective of knowledge rather than as tiny uncommunicated clusters” - [1, Cacho and Taghva p. 519].
This quote illustrates the importance of reproducibility of scientific work and describes an ideal state to gain knowledge. Nevertheless, the reality does not reflect this ideal state. In 2016, a survey with 1,576 respondents was published in Nat2u].r9e0[% of them stated that the scientific community in general is facing a reproducibility crisis. To understand this crisis, this paper tries to shed some light into this complex topic and states the current situation for
However, in research these terms are sometimes used interchang3e]a.bRleypr[oducibility
and repeatability are even used the other way ar4o].unTdhe[re is also a standard called
Reproducibility Badging and Definitions published by the National Information Standards
Organization (NISO)5[]. There, they define further nuances of the three introduced terms above.
Despite their depth, this standard being around for already three years is only referenced by
27 Google Scholar h2itws hen searching for the title of this publication. For this paper, we
will stick to the ACM definitions introduced above due to their clarity and adaptation in the
community.Collberg and others expected computer science to be in a special role compared
to other discipline“sre:producing the work published in a systems conference or journal should be
as simple as going to the authors’ website, downloading their code and data, typing ‘make,’ and
seeing if the results correspond to the published ones” [4, p. 1]. However, in a 30 minutes time
frame they were able to retrieve and build the source code of only 32.3% of the papers analyzed.
Other secondary studies, e.6g,. 7[
All these studies should be a wake-up call for the computer science domain to publish raw data, source code and scripts. For without this background information a correct interpretation of results is not always possible for others which prevents the correct evaluation of the merit of a paper. This can be illustrated with an example from FaaS research: Tw9o, S10L]Rs [ revealed papers reporting about unusual performance results when running a function on a cloud provider. Properly interpreting the original results based on the provider documentation showed that the measurements where due to a memory setting. It unintentionally assigned more than one CPU to the function. Without enough details about the experimental setup, it would not have been possible to falsify the misinterpretations of the original authors. As a consequence, conference chairs and publishers should incentivise researchers to make their experiments open to the public and enable others to reproduce experimental results. 1https://www.acm.org/publications/policies/artifact-review-and-badging-current 2Google Scholar search was performed otnho1f9February 2024.
Since we are interested in the current state of reproducibility studies, the objective of this paper is not to conduct a further reproducibility study but to analyze the already published studies by answering the following three research questions: RQ1 What are publication trends for reproducibility studies in computer science? RQ2 Why do authors try to reproduce the work of others? RQ3 How successful are the reproducibility studies and which challenges do the authors encounter?
RQ1 focuses on the number of studies and visible trends. The reasons for performing such studies are questioned RinQ2. And the last questioRnQ,3, reveals success numbers on how many studies were able to reproduce the original work. One caveat here could be a publication bias, where reproducibility problems could be more likely to be published than1s1u].ccFeosrs [ upcoming reproducibility studies, we also look at challenges the included publications faced to share aggregated learnings.
The agenda of our paper is as follows: In Sec2t,iowne shortly introduce the SLR methodology and present some numbers and an overview of included papers. Results for our three research questions are presented as subsections in Sec3t.iSoenction4 concludes the paper with a short summary and an outlook to future work.
We conducted a SLR to understand the current state of reproducibility studies in the computer
science domain. A SLR is well suited for summarizing and synthesizing the current status on a
topic in a fair way11[]. For the search phase, ACM Digital Library, IEEE Digital Library, DBLP
and Google Scholar were used. While the first three libraries have a computer science focus,
Google Scholar was chosen as a more general search engine with a large corpus. Therefore, we
only used the first 50 entries at Google Scholar, sorted by relevance to get a manageable set of
literature as already done by other S1L2,R1s3[
ACM Digital Library IEEE Digital Library
DBLP Google Scholar
Initial Search (198) 12 5 131 50 (limited)
Filtering (38) 5 3 16 14
Merge and Deduplication
The search term required the phrase “reproducibility study” to be part of the title or abstract.
As the search engines difer in their search options, the exact search string1s5]d. iIfenrt[otal,
198 entries were found in the initial search phase. The inclusion and exclusion criteria are based
on the research questions and were applied to title, abstract and co1n5c]l.uIsniocnase[s of
doubt, the full text was skimmed. The selection criteria were developed upfront and piloted on
some studies as recommended bKyitchenham andCharters [
• In addition to peer-reviewed publications, also preprints.
The following exclusion criteria were used:
• Publications in which the authors do not use or do not mention a specific software
implementation.
• Publications that belong to the field of computational science rather than computer
science, i.e. that are more concerned with the application of computer science in fields
such as medicine.
• Publications in languages other than English (language is often used as an exclusion
criterion1[
All raw data for the SLR are archived at Ze1n8o]d.oIn[clusion and exclusion decisions as well as all other data related to the SLR can be found there. In case of exclusion, a reason is stated. Although requiredKbitychenham andCharters, the quality of the studies was not assessed for every paper11[]. It is assumed that in particular search engines from ACM, IEEE and DBLP have results with high quality. Otherwise, studies of questionable quality would have been excluded during the data extraction phase.
After the merge and deduplication step, 20 results remained. During data extraction, one
paper turned out to be a replicability study and was thus ex1c9l]u.Wdeidth[in the remaining, 14
belonged to the information retrieval do2m0a,i2n1,[
The most obvious fact in our SLR investigation is that the information retrieval domain is a role model for reproducibility studies. 74% of our identified papers (14/19) are from this domain. The implementation of diferent ranking and retrieval algorithms is inherent to this computer science sub-field. For those an interpretation of results is only feasible by checking the raw data
Ref Sub-Field
and experiment details. The same holds true for another little cluster, namely Natural Language Processing (NLP). Here, the research objectives are also highly dependent on input data and its processing.
tsano 456 i lcPub 23 i fo 1 # 0 2016 2017 2018 2021 2022
2023 2019 2020
Publication Year
When we searched for literature, we did not limit the publication year. Nevertheless, the oldest paper in our set of filtered literature was published in 2016.2Fsihgouwres the number of publications by their publication year. The bar for 2023 is colored orange since the search phase was conducted in May 2023, so further reproducibility studies might have been published in the rest of 2023. The distribution of publications over time shows that the number of papers which target reproducibility concerns has been rising over the last years.
Another facet of publication trends are the venues where papers get submitted and presented. The European Conference on Information Retrieval is the top venue for reproducibility studies. A majority of the information retrieval papers were published there (9/14). One reason for this high number of papers is a special reproducibility track with a dedicated call3f.oTrhpiaspers shows that dedicated conference formats can support the reproducibility efort to bring experts together in one place.
To summarize the insights foRrQ1: Information retrieval is the top domain for reproduciblity research due to the domain specific challenges and supported by dedicated conference tracks. Reproducibility studies are a trending topic when looking at the rising publication numbers.
For answering the remaining research questions, we focus on the nine papers 1o.f Table
Five of the nine papers from Tab1l,efocus upon the reproducibility assessme3n5,t2[
As an additional finding it could be shown that some studies mix terms and approaches.
As introduced, we distinguish based on the ACM terms reproducibility, repeatability and
replicability. Three of the publications analyzed in the SLR had a replicability study as an
additional par2t0,[
The success ratios for the nine included papers reveal a mixed picture. Four studies were successful, three were partly successful and two were not successful at all. The two biggest challenges were missing artifacts and lack of documentation.
Studies were considered successful when the measured values were close to the originally
reported ones22[
One challenge of the studies was the availability of artifacts like source code or raw data. For three of the publications considered in this SLR the necessary artifacts were2a2v,2a3i,l3a4b]le [ and the reproducibility for these studies succeeded. Regarding the other six, only incomplete artifacts were available.
Another big challenge was insuficient documentation of experimental sNetuunpzsio. and others, for example, identified unclarities regarding the environment of the primary study. The hardware was not suficiently described in the original publication, especially concerning the CPU and GPU as both could influence the execution time. In the original paper, there were two contradictory statements about the used GPU model. Additionally, the original authors trained one of their models in the cloud because they needed a more powerful environment. In comparison, the authors of the reproducibility study needed the cloud environment for two models which prevented the comparison of the training time for the additional model.
Reproducibility is a key requirement for science and has to be considered when starting a new research project. In reproducibility studies, researchers other than the original authors use a setup as close as possible and reasonable to reproduce and validate the work of others. Starting from this, the objective of this paper was to give an overview about reproducibility studies in computer science.
The publication trends showed that the information retrieval domain is a role model for other domains, supporting its researchers with dedicated conference tracks. Based on the number of publications per year, we saw that the reproducibility topic gains traction. Additionally, we could confirm other meta studies that showed that a lack of raw data and artifacts as well as an incomplete or missing documentation are the most serious challenges for good research.
For future work, we propose a follow-up study on reproducibility which should include additional search engines and incorporate further SLR tasks like sn4o.wWbaelallisnogplan to incorporate the feedback from reviewers to include the term “executable paper” which was a hyped term around 2011 as an additional search term. In addition, we want to broaden our scope by incorporating similar terms like “analysis” and “survey” paired with “reproducibility”, “replicability” and “repeatability” to also include publications which use another term but target “reproducibility studies” in the sense of this paper. 4Snowballing was already performed for this study and the raw results are already available at Zenodo. Due to time and space constraints, these snowballed publications were not assessed but published for follow-up research.