Research on empirical software engineering has increasingly used the data that is made available in online repositories, speci cally Free/Libre/Open Source Software projects (FLOSS). The latest trends for researchers is to gather \as much data as possible" to (i) prevent bias in the representation of a small sample, (ii) work with a sample as close as the population itself, and (iii) showcase the performance of existing or new tools in treating vast amount of data.
The e ects of harvesting enormous amounts of data have been only marginally considered so far: data could be corrupted; repositories could be forked; and developer identities could be duplicated. In this paper we posit that there is a fundamental aw in harvesting large amounts of data, and when generalising the conclusions: the application domain, or context, of the analysed systems must be the primary factor for the cluster sampling of FLOSS projects.
much) FLOSS data has been harvested, and how many times the authors agged an issue in their di erent application domains. Second, we discuss the implications of using `application domain' as the clustering factor in the sampling of FLOSS data, and the generalisations within and outside the clusters.
Index terms| BENEVOL papers 1
The use of open, available data has been a welcomed accelerator in the software engineering research eld. Data on the processes and products available via an Open Source approach has led to an increasingly large number of workshops, conferences, papers and research attempts to describe the phenomenon. Researchers gathered initially in 2001 around the Open Source Software (OSS) workshops, held annually in colocation with the ICSE series of conferences. Before the OSS workshop spawned into the OSS conference in 2005, the BENEVOL community started to group together researchers from the Software Evolution domain. Its initial focus was `(...) to bring researchers to identify and discuss important principles, problems, techniques and results related to software evolution research and practice'1.
While the goal of a few BENEVOL papers has been
to achieve the generality of the results [
The underlying vision of this paper is to open a
proper debate on the importance of context for any
software system, and the uniqueness of its
application domain. This position paper stems from the
work of several prominent researchers who called the
community to `go deeper, not wider' (Michael
Godfrey at MSR 2017) and `minding the mine, mining the
mind' [
The purpose of this paper is to share some
ndings about a selection of papers discussed during the
last few years of BENEVOL workshops. The focus is
speci cally based on BENEVOL papers that have used
FLOSS data. The context of our analysis is the
diversity of FLOSS projects under study, and how that
was re ected by researchers in their ndings. Some
100 papers are analysed in terms of whether FLOSS
projects are used, how many, and whether
considerations of application domains have been used to inform
the sampling of FLOSS projects, or the validity of the
conclusions. We assume that domains are relevant as
a fundamental construct for any empirical software
engineering research [
The vast literature on FLOSS systems of the last 10
years has been possible also due to a series of guidelines
on how to perform quantitative, empirical analysis on
FLOSS processes and products. When SourceForge2
was considered as the de-facto FLOSS forge, a well
received research paper shared more than one insight
on the most common mistakes to avoid when mining
data and results from the projects hosted there [
The acknowledgement of GitHub as the newly
established central focus for FLOSS development
generated a similar requirement, in terms of shared
guidelines to avoid common mining mistakes [
Outside of the FLOSS literature, the diversity and
context of software systems have received some
attention in the past [
Interesting patterns valuable to researchers and practitioners are often identi ed in domain-based analysis of software projects. Results from one domain might not be applicable in another. As such, it is important for results to be representative.
Software categorization or domain clustering has
gained importance over the years. For example, the
knowledge of software trends in a particular domain
can assist developers in the search for domain-speci c
reusable components [
According to Hae iger et al. [
In order to show how FLOSS data has been used and analysed by the BENEVOL community, we report here an investigation of the research papers appeared in the last 5 years of the BENEVOL event. An overall 101 papers have been considered in this study: we share the raw data in the spreadsheet at https://tinyurl. com/y69wkadr.
Each paper was read by one of the co-authors, and summarised along the following points: • Use of FLOSS systems (yes/no): at rst we checked whether FLOSS projects are used in the paper at all. This served as an indicator of the pervasiveness of FLOSS projects in the literature produced by BENEVOL papers. • Number of FLOSS systems used : in second instance, we trawled through the paper, annotating where the authors mentioned how many FLOSS systems were used. In the case of full papers, the abstract, introduction, methodology and conclusion were read for that purpose. • Analysis of application domains: thirdly, we considered the methodology, results and conclusion of each paper, along with the threats to validity, looking for considerations of application domains. We checked if the authors considered this attribute in the sampling of FLOSS projects, whether they limited their results against this axis, or whether it was considered a speci c threat to validity. This attribute was coded as either fyes | nog.
The contributions to the 2016 edition of BENEVOL are not available online, so they had to be excluded from our analysis. The spreadsheet with the categorisation of the papers has been made available for inspection under the following link: https://tinyurl. com/y69wkadr. 3.1
In this section we provide the rst point of our analysis: `how many BENEVOL papers have used FLOSS systems in their analyses? '. As visible in the two plots of Figure 1, researchers (and accepted BENEVOL papers) have steadily used FLOSS systems for their papers. The rst plot shows the absolute numbers of accepted BENEVOL submissions that use one or more FLOSS projects.
The bottom plot of Figure 1 shows the ratio of FLOSS and non-FLOSS papers in the BENEVOL sample of papers. It is getting increasingly more common to use one or more commercial software systems, or a combination of FLOSS and non-FLOSS projects. 3.2
In this section we report on the number of FLOSS systems evaluated by BENEVOL papers. For this purpose, we analysed the methodology description, or the empirical approach, of each paper to determine how many FLOSS systems were reported in the study. Figure 2 displays the cumulative number of FLOSS systems used in BENEVOL papers, per year. The median number of systems has increased from one analysed OSS system in 2012 to 1,127 systems in 2018.
The exponential number of FLOSS systems being used by BENEVOL papers has been accelerated by many factors: (i) availability of open forges (FreshMeat, SourceForge, Savannah, Apache FSF, GitHub and many others); (ii) common, shared toolsets to perform the analyses; (iii) guidelines on how to e ectively use forges.
Below we give a summary of ndings to assess the trends observed in the number of FLOSS systems analysed by the BENEVOL papers.
The trend that we observed throughout the subsequent years of the BENEVOL contributions is, fundamentally, summarised as `the more the better'. Authors have started to include larger and larger FLOSS samples to their papers. We can assume that this pattern has been followed in order to achieve the generality of a paper's ndings. At the last edition of available BENEVOL contributions (BENEVOL 2018), over one million FLOSS systems were considered for investigation, jointly by the accepted papers.
Several BENEVOL papers use ecosystems [
Several BENEVOL papers have used umbrella projects (for example, Gnome). In most cases we considered them as single FLOSS systems: depending on the time of the analysis, these larger projects can contain a variable number of sub-projects. This makes it di cult to de ne the status of the super-project, in terms of number of its sub-projects, as well as their domains. This also makes it di cult to replicate those studies, as well as their results and conclusions.
Throughout the editions of BENEVOL, sampling of a few systems (see for instance [15] or [16]) has given way to whole-forge analyses. Also, there seems to be a general view that `pruning' a sample is a good idea for removing outliers, or for promoting quality. This has an e ect on the sample studied, and the representativeness of the population as a whole. 3.3
The third analysis was based on the application domains of the systems considered in the empirical study. For all the papers (not only for those using FLOSS projects), we tried to establish whether the authors considered the results, ndings or discussion as constrained by the type of system (e.g., its domain). This included checking how the threats to external validity (if any) addressed limited the conclusion to the domain(s) under investigation.
We grouped the papers into two categories (and plotted them accordingly per year): 1. papers that directly considered application domains as drivers in the variability of the results (stack "YES" in Figure 3); 2. papers that didn't considered application domains as drivers (stack "NO" in Figure 3).
The results of this analysis are shown in Figure 3: a ratio (%) is used to separate the papers in the two categories. It is clear from the visualisation that BENEVOL papers do not generally acknowledge the variability of results as driven by the domains of the systems involved. Earlier papers (especially from the 2012 batch) have a good cover of domains in the evaluation of the results, but this is not re ected in the later editions of BENEVOL.
As visible in the Figure, the majority of ndings on FLOSS, as reported by BENEVOL papers, do not mention application domains. In some cases, researchers have acknowledged the variability of the results [17, 18], and hinted that other factors could play a role in such variability. We considered as a \limited" acknowledgment of the relevance of the application domain when authors mentioned the diversity of the systems under study. 4
The birdseye view on the type of BENEVOL contributions (Sections 3.1, 3.2 and 3.3 above) reveals some interesting trends when dealing with FLOSS projects. Below we discuss in more detail whether FLOSS papers were analysed ("YES" or "NO"), and whether domains were considered in the analysis ("YES" or "NO"). So far in the BENEVOL series, few papers explicitly addressed the importance of domains when analysing systems, or when discussing ndings. An interesting perspective is given in [19], since it considers a very speci c type of systems, the `cross-system packages'. These systems are likely to show similar characteristics since they are supposed to act as vectors to an from the overarching system.
By drawing on the importance of the application domains in this paper [20], the authors signify the importance of domain analysis when creating a theoretical and practical framework that supports the development and the evolution of adaptive data-intensive software systems for ubiquitous environments in their study. Thus, they focus on data and in particular on the problem of nding the most suitable portion of data that have to be provided by the application in the of context of `self-adaptive system'.
Likewise in the 11th edition of BENEVOL (2012), [21] examined the impact and role of social media on software development. The authors argued that \social media is poised to bring about a paradigm shift in software engineering research" particularly in OSS community.
In the 2014 edition, only one BENEVOL study
focusing on OSS projects implicitly highlighted the
need to investigate projects from various domains [
We concluded that application domains are not well represented or studied in the papers that use FLOSS data. 4.2
The vast majority of BENEVOL contributions, based
on FLOSS systems, do not consider domains as one
of the factors to take in consideration. An interesting
example of this approach is given in [
More in general, the approach of researchers is to
focus on speci c languages or source code models (see
for instance the paper in [
We concluded that most of the papers studied from the BENEVOL series do not consider the application domains as an important factor for software analysis or evolution. 4.3
A few of the papers that we analysed are not based on FLOSS systems, but more in general on commercial, or in-house software. In a few cases, we observed that the authors actually considered the limitations of their case studies to the one domain that was investigated.
As a few of such examples, we noted a paper
based on a banking system [
In general, the BENEVOL papers using non-OSS software as their case studies do not use the domains to aggregate results. Nonetheless a few BENEVOL contributions have shown a clear pathway into not generalising the ndings to all domains. 5
This paper analysed how open source software has been used by the BENEVOL contributions between 2012 and 2018. We showed the increasing number of BENEVOL contributions that used FOSS projects for their analyses.
Although the majority of contributions do not acnowledge the importance of domains when discussing the ndings, there is an increasing number of papers that limit the results, or the data sampling, to speci c domains. We believe that one of the major challenges for empirical software engineering is to better understand the role of domains, especially in the evolution of software systems. We propose for papers that empirically analyse software systems to acknowledge such challenge in a `threat to domain validity'. [12] Maelick Claes. Applying biological evolution to software ecosystems a case study with gnome. [14] Jose Javier Merchante and Gregorio Robles. From python to pythonic: Searching for python idioms in github. [15] Ward Muylaert and Coen De Roover. Untangling source code changes using program slicing. In BENEVOL, pages 36{38, 2017. [16] Jie Tan, Mircea Lungu, and Paris Avgeriou. Towards studying the evolution of technical debt in the python projects from the apache software ecosystem. In BENEVOL, pages 43{45, 2018. [17] Zeeger Lubsen, Andy Zaidman, and Martin Pinzger. Using association rules to study the coevolution of production & test code. In Mining Software Repositories, 2009. MSR'09. 6th IEEE International Working Conference on, pages 151{ 154. IEEE, 2009. [18] Christian Rodr guez-Bustos and Jairo Aponte.
How distributed version control systems impact open source software projects. In Mining Software Repositories (MSR), 2012 9th IEEE Working Conference on, pages 36{39. IEEE, 2012. [19] Eleni Constantinou, Alexandre Decan, and Tom Mens. Breaking the borders: an investigation of cross-ecosystem software packages. arXiv preprint arXiv:1812.04868, 2018. [20] Marco Mori and Anthony Cleve. A framework to support the development and evolution of selfadaptive data-intensive systems. In 11th edition of the BElgian-NEtherlands software eVOLution symposium (BENEVOL 2012), 01 2012. [21] Maelick Claes. Applying biological evolution to software ecosystems a case study with gnome. In 11th edition of the BElgian-NEtherlands software eVOLution symposium (BENEVOL 2012), 01 2012.