Background: Since Alitheia Core was proposed and subsequently retired, tools that support empirical studies of software continue to be proposed, such as Codeface, Codeface4Smells, GrimoireLab and SmartSHARK, but they all make diferent design choices with overlapping functionality. Aims: We seek to understand the design decisions adopted on these toolsgood and bad-and their consequences to understand why their authors reinvented functionality already present in other tools, and to help inform the design of future tools. Method: We used action research to evaluate the tools, and determine principles and anti-patterns to motivate a new tool design. Results: We identified 7 major design choices among the tools: 1) Abstraction Debt, 2) the use of Project Configuration Files, 3) the choice of Batch or Interactive Mode, 4) Minimal Paths to Data, 5) Familiar Software Abstractions, 6) Licensing and 7) the Perils of Code Reuse. Building on the observed good and bad design decisions, we created our own architecture and implemented it as an R package. Conclusions: Tools should not require onerous setup for users to obtain data. Authors should consider the conventions and abstractions used by their chosen language and build upon these instead of redefining them. Tools should encourage best practices in experiment reproducibility by leveraging self-contained and readable schemas that are used for tool automation, and reuse must be done with care to avoid depending on dead code.
the other more recent tools using an action research [2] approach.
Research into software architecture and quality requires Our contributions in this paper are twofold: first, we the analysis of large quantities of data. For researchers present a set of key design decisions derived from the this often means mining data from multiple open source aforementioned tools which either facilitated or hindered software projects. Pre-processing data, calculating met- reusability, reproducibility, interoperability and extenrics and flaws, and synthesizing composite results from sion of functionality. Second, we present our tool, Kaa large corpus of project artefacts is a tedious and error iaulu2., which builds upon the design decisions made prone task lacking immediate scientific value [ 1]—it is from these tools, and which we believe fills a gap in the seen merely as a means to an end. This was the motiva- existing mining software repositories ecosystem. tion for the Alitheia Core [1], which was made available in 2009 for the software engineering community. It provided features for data collection, integration and anal- 2. Studied Tools and Lessons ysis services and emphasized an easy to use extension Learned mechanism. Yet, as of today, Alitheia Core is a dormant (read-only) project in GitHub1 and several other tools The tools that we studied are Codeface [3], Codereplicate at least some of its functionality. face4Smells [4], GrimoireLab [5, 6] SmartSHARK, [7, 8]
What went wrong? Why have many tools re- and PyDriller [9]. We now present our observations reimplemented the same “tedious and error prone” tasks? garding the strengths and weaknesses of these tools in And do the current tools live up to the promise of Alitheia terms of their design choices and note, throughout the Core? In this work, we revisit lessons learned by the work, lessons learned by the authors of Alitheia Core [1] Alitheia Core authors and the design choices made by presented in [10]. Many of these lessons are applicable and worthy of consideration in new tools with similar intents. We employed an action research methodology in studying these tools, but do not describe the details of that research here, due to space limitations. 2The documentation for the R package can be found at https: //github.com/sailuh/kaiaulu 2.1. Abstraction Debt specific abstractions per data type, whether via CLI or API as issueShark and PyDriller do, facilitates building additional functionality specific to a particular data type, or audience. Moreover, CLIs can be built on top of a welldefined API, providing the benefit of both interfaces, as we do in Kaiaulu.
We have observed diferent levels of abstraction employed in the surveyed tools, ranging from applications that are built as monoliths to those built from smaller components. This is consistent with what has been noted in machine learning systems as abstraction debt [11], i.e. a lack of key abstractions to support the functions and growth of MSR tools. 2.2. Tool Configuration Files vs Project
Codeface was created as a monolithic application, in Configuration Files which an entire project’s Git log or mailing list is analyzed. It abstracts a complete end-to-end pipeline, imple- In [8], the authors of SmartSHARK noted that one of their mented by a command line interface (CLI), and outputs goals was to support replication through the storage of a database dump of a project. It is therefore dificult for data in a single harmonized schema. Replication, it is other applications to build on some of its unique fea- argued, is supported by a common dataset. However, we tures, for example, using its Git log parser that parses at have observed that replication is also being done within function (rather than file) granularity. configuration files in Codeface.
Both GrimoireLab and SmartSHARK define several Codeface uses a concept we named project configucomponents, each with its own CLI, but the component ration files. These files provide a single compact source abstractions they employ are not the same. To provide a where parameters associated with the acquisition and point of comparison, Grimoire’s Lab Perceval provides a manipulation of a dataset can be stored. Project configuCLI to obtain data from many data sources (e.g. GitHub, ration file parameters are required for tool execution, and Git, Bugzilla, Jira, mailing lists, etc), serving as a single they are a pragmatic, lightweight and human-readable interface for data collection. In contrast, SmartSHARK way to specify reproducible results. Project configuration defines its abstraction per data source type and, in the ifles also save time when a project is re-analyzed in other case of data acquisition, at a more fine-grained level than studies, as some project-specific information may not be Perceval. For example, consider issueShark and vcsShark, obvious from the dataset alone. two components of SmartShark. IssueShark defines ab- Of all the tools we have reviewed, only Codeface prostractions for diferent types of issues tracker sources, vides users with a means to specify project configuration and vcsShark for diferent types of version control sys- ifles. This led to a large collection of project configuratems. SmartSHARK’s abstractions facilitate defining ad- tions that have been versioned in Codeface over time ditional features specific to a data source type, such as 4. This information, which supports repeatability, may separating static vs. dynamic data in issue trackers (e.g. otherwise not have been possible (or at least easy) to creation time of the issue vs. comments), regardless of reconstruct if all that was shared was the data. its underlying implementation (e.g. Jira or Bugzilla)3. We note that externalizing parameter choices in data
Pydriller is a single component and is smaller in scope acquisition and manipulation tasks has been more promias it only abstracts Git repositories. However it is dif- nent in machine learning frameworks, for example to ferent from the other tools in that it provides an API define experiments in configuration files, 5 which include instead of a CLI. Its motivation is also diferent: it wraps machine learning model selection and choice of model around PythonGit, which in itself provides a Pythonic hyper-parameters [12].
API to nearly all features of Git, to provide an API catered From the above, we derive the following lessons: intetowards mining software repositories only. In providing grating configuration files that are human-readable and just a subset of Git functionality, it exposes functionality leveraged by the tool can enable reproducibility, without catering specifically to the needs of mining repositories. the hurdles of sharing large quantities of primary data.
The decision between choosing a CLI or API has tradeofs. An issue with command line only interfaces occurs 2.3. Batch Mode, Interactive Mode, and when an end-user may be interested in a diferent ab- Literate Programming straction of the data not preconceived by the authors.
However an API requires the user to be familiar with As we noted before, with the exception of PyDriller, every the programming language the tool was built on top of, tool defines a CLI, but not an API. This means the only whereas a CLI does not.
From the above we derive the following lessons learned: End-to-end pipelines such as Codeface’s limit the ability of other researchers to build on top of them. Defining more 3https://github.com/smartshark/issueSHARK#introduction 4See https://github.com/siemens/codeface/tree/master/conf and https://github.com/maelstromdat/codeface4smells_TR/tree/ master/Configurations for Codeface and Codeface4Smells respectively
5https://xnmt.readthedocs.io/en/latest/experiment_config_ ifles.html way to interact with these tools is batch mode. Mean- age both APIs and Notebooks, which is a common practice while, PyDriller does not ofer a CLI, only an API, which in R packages, therefore avoiding abstraction debt. confers its users the ability to leverage Python’s interactive mode to explore the data. However, it does not 2.4. Minimal Paths to Data include a CLI for batch mode processing, for out-of-thebox data acquisition, processing or data analysis. What According to [10, p.233], the efort required to learn how we observe then is that existing tools decide on either infrastructure code works has to be proportional to the CLI or API, but not both. We believe, however, that the gains and account for deprecation. We agree with this mining of software repositories requires a tool capable of observation. Let us look at how existing tools manage both, supporting an iterative process of data exploration, this concern. and when concluded, a way to enact batch processing to When using GrimoireLab components (in particular scale up. Perceval) the minimal path to data is surprisingly short.
To illustrate our claim—as no existing tool provides Provided with a Git repository URL, or a local copy, it both capabilities—we provide a few examples: in a recent will output a JSON file to stdout. Likewise, provided with socio-technical study, we needed to do identity matching, a URL to a website mbox or local file, it will also provide applying heuristics that have been published by other au- a JSON file to stdout. A developer can easily integrate thors (e.g. [13, 14]) to assign identities to developers who wrappers to its CLI, and users can easily obtain data for use diferent names and e-mails in version control sys- a project of interest. In this ecosystem, a database is tems and mailing lists. Consider the case where we chose available, but it is optional: users need not to concern the simplest method, where developers whose name or themselves with learning GrimoireLab’s Elastic Search e-mail match are assigned the same id. At first glance, database to obtain data. this seems like a reasonable assumption. However, it This is in contrast to Codeface and SmartSHARK, both was due to experimenting interactively with the identity of which require user familiarity with MySQL and Monmatching API that we discovered that all core develop- goDB respectively, along with their data model schemas ers, due to the use of an issue tracking system, ended up to obtain the equivalent version control system and mailsharing the same e-mail address. We noted this case as ing list data. The minimal path to data in these cases is a unit test until a better heuristic could be found, and much longer, including the setup overhead and integrathen examined the data for other cases until we were tion with other tools. satisfied with the results. We then saved the observed When data integration is sought in the database, parameters in a project configuration file, and used it to GrimoireLab retains its approach of keeping the data deploy a batch process to collect various computationally closest to source, and not harmonizing it in a schema intensive architectural metrics. that facilitates integration [8]. Codeface’s MySQL and
We have had similar experience in determining and SmartSHARK’s MongoDB provide a harmonized schema, testing heuristics to filter files in a repository, or deter- which makes it easier for users to store the various types mining the method that developers adopt to annotate of data. issue numbers in commit messages. Because each project In the case of PyDriller, which provides an API, the may apply its own conventions, tools that ofer an experi- minimal path to data requires familiarity with the Python mentation capability, and then defer mass data processing programming language. This ofers the convenience of to batch more eficiently support the full workflow of a reshaping the data to the user’s final need, but adds an researcher in mining software repositories. overhead to the user for familiarization with the API,
The described interactive data explorations could cer- instead of just the raw data schema from the source of tainly have been done in a Python or R session, but it interest (which the user is likely already familiar with is better to leverage literate programming using, for ex- for their research purposes). One researcher [15, p.39] ample, Python or R Notebooks, so that the rationale of who extended Codeface4Smells identified a problem of the design experiment is not lost. However, care must be Pipeline Jungles [11], due to heavy reliance on a folder taken to not extensively rely on notebooks without fur- hierarchy and file name conventions. ther refactoring functionality into the code base, leading Our lessons learned here were: databases need not be to dead experimental code paths [11]. a requirement to provide users with various data sources.
Our learned lessons here were: existing tools choose ei- This also simplifies component reuse by other tools and dether APIs or CLIs (supporting batch or interactive modes). creases the likelihood of reinventing the wheel. Providing However, making both interfaces available will better sup- a minimal path does not exclude providing a database for port users in their various research eforts in mining soft- researchers, as Perceval shows. However providing a harware repositories. The use of Notebooks to illustrate and ex- monized schema can save researchers from having to replain the API complements the API, provided functionality implement code to integrate the same kinds of infrastrucis not entirely written in Notebooks. In Kaiaulu, we lever- ture over and over. Lastly, providing an API gives some lfexibility to users to reshape the data with the tool. But through a chain of dependencies which exemplifies the user familiarity with the programming language and API concern posed here. mecoSHARK is a component that is a kind of overhead and this does not seem ideal, as the serves as a wrapper for OpenStaticAnalyzer10, with a data could be provided directly via a CLI leaving a task for last commit date of July 13, 2018. In turn, OpenStatthe researcher to adapt it in their own programming lan- icAnalyzer also wraps several other dependencies, inguage. As such, we believe having available a CLI that out- cluding FindBugs 11, last released in March 15, 2015. puts the data as Perceval does, and a harmonized schema In its bug tracker12, FindBugs requests for bugs to no as in Codeface and SmartSHARK, provides the best combi- longer be reported, noting that SpotBugs13, FindBugs’ nation. successor, should be used instead. This confirms that the mecoSHARK wrapper, which provides OpenStatic2.5. Other Design Decisions Analyzer functionality to SmartSHARK,is now dependent on dead code, further increasing the burden of We briefly mention here other (more minor) design deci- the SmartSHARK ecosystem maintainers. Nonetheless, sions that we believe may cause dificulties in adoption. SmartSHARK’s approach to wrap black-box packages
Familiar Software Abstractions. Both Perceval and into common APIs is considered good practice [11]. PyDriller leverage a common interface for end-users. As a means to mitigate this risk, relying on and conThey are both Python libraries, and provide the expected tributing work to open source communities that more interactions for CLI and API respectively. In Perceval’s carefully assess the health of projects and try to maintain CLI, provided with a list of parameters and flags, data is them, such as the Apache Software Foundation, ROpenoutput to stdout. PyDriller exposes an API, an extension Sci14, and CHAOSS15 may be an important consideration. to a programmer’s familiar programming paradigm. This For example, ROpenSci accepts R packages via a streamis in contrast to ecosystems that define a diferent abstrac- lined peer review process and, for accepted packages, tion, such as SmartSHARK, where detailed instructions provides community support, package promotion, and must be followed to extend its functionality 6. Exten- fast-track publication to journals16. sion instructions are also not available for Perceval or Codeface.
Licensing. Another important consideration in 3. The Kaiaulu R Package reusing a code component is how permissive its license is.
For example, stringr, an R package to manipulate strings Based on the above observations and lessons learned, we used by XGBoost, a popular machine learning algorithm, now describe the design decisions behind the Kaiaulu was replaced by stringi, another R package to manipu- R package that we created. We created this in R, as we late strings, solely based on the diference in licenses. 7 believe the R ecosystem provides a reasonable approach Similar reasoning also led an R package that represents to implementing the lessons learned from existing tools data tables eficiently to adopt a diferent license because a single architecture. the existing license “could be interpreted as preventing Batch Mode, Interactive Mode, and Literate Proclosed-source products from using data.table”8. Lack of gramming in Kaiaulu. We chose to use the R lanclarity on interactions of open source licenses has been guage17, in contrast to existing packages that use Python reported by [16]. Among the tools we studied, we have (with the exception of Codeface, which includes both observed the following licenses: Codeface adopts GPL Python and R, but not as a package) due to the familiarity 2.0, PyDriller Apache 2.0, SmartSHARK Apache 2.0, and of the authors with the language and a preference for its Grimoire’s Lab GPL 3.0 and LGPL 3.0. package architecture.
Perils of Code Reuse. With the availability of pack- Minimally, the structure of an R package consists of the age managers such as CRAN and PyPi which greatly facil- package metadata and its API. In addition, the R ecosysitate code reuse, you can declare dependencies on others’ tem promotes best practices to include vignettes, which code instead of copying it into your own project, taking leads R users to expect an API and R Notebooks when advantage of their functionality without assuming the installing packages from CRAN (The Comprehensive R burden of maintenance. However code interdependence also poses risks [17], such as dependencies going extinct [18]. Hence, care has to be taken to avoid dependencies to non-maintained third-party code.
An interesting example occurs in mecoSHARK9 Archive Network).18 CRAN also treats R Notebooks as programming languages encourage variable definitions ifrst class citizens in an R package 19 showing on each at the beginning of a program ; 3) Do create a dependency package’s website any R Notebooks available. Because of between the CLI and the project configuration files, to R package structure, complying with familiar software facilitate batch processing and reproducibility. abstractions (see Section 2.5) automatically brings the Our intent is that users will first observe the R Notebenefits of literate programming (see Section 2.3). books to get a better understanding of the API for a
Abstraction Debt in Kaiaulu. R natively supports particular task of interest, and in doing so will familiartables and vectors as data types, which is a familiar ab- ize themselves with both the relevant portion of the API straction for data analysts. To capitalize on this, Ka- and the project configuration file. If the interest is only, iaulu’s parse_ functions map most data sources (Git logs, for example, to understand how to parse Git logs, using mailing lists, file dependencies, software vulnerability for example the Git log R notebook, then users should feeds, metrics, etc.) as tables with standardized col- not be concerned with specifying the mailing list. When umn naming, which allows for quick identification of comfortable, users can then use their newfound underwhat data can be combined. Kaiaulu also ofers various standing to scale the analysis to the entire project using transform_to_network_ functions to represent and inter- the configuration file for the CLI, build their own analyactively visualize these networks20 which in turn enable ses as vignettes, or define new CLI interfaces. This design more complex socio-technical analyses at diferent gran- is consistent with a mining software repositories workularities: functions, files, classes, etc. lfow, in which a researcher should first explore the data
Tool Configuration Files vs Project Configura- qualitatively to assess threats to validity, before scaling tion Files in Kaiaulu. Following the design choice of up data processing in batch mode without clarity of what Codeface (see Section 2.2), and building on best prac- assumptions the tool is making using default parameters tices for machine learning configuration files [ 11] we or arbitrary thresholds. implemented project configuration files using YAML. Be- Kaiaulu also further decreases the minimal path to cause we externalize all parameters in project configu- data in terms of how it handles third party dependencies. ration files, an important concern is that the file does Users need only concern themselves with installing denot grow overly complex, requiring documentation of its pendencies for their task of interest. For example, if the own. That is, we do not wish the minimal path to data to interest is only to parse Git logs, they need only set up increase as new features are added, as we discuss next. Perceval, and provide its binary path as a parameter to
Minimal Path to Data in Kaiaulu. As discussed in Kaiaulu’s parse_gitlog to obtain the parsed data. More Section 2.4, it is important that the path to data remains generally, the parse_ API minimizes efort to researchers as small and easy as possible. How should users be in- by transforming various tool-specific data formats, if the troduced to project configuration files? Should a manual researcher so desires, into tables, and performing minipage be dedicated to explain its various parameters? We mal processing on potentially inconsistent fields, such as again build upon familiar concepts, specifically with the ifle paths, to make them internally consistent. intent of applying the rule of least surprise [19, Ch.11]21 i.e. ‘do the least surprising thing’. In an R package, it 3.1. Kaiaulu’s Features is expected that R Notebooks provide examples of how to leverage the API to accomplish a task by combining multiple functions, while individual functions provide self-contained examples, which can be obtained in the R environment at any time by preceding a function name with a question mark, e.g. ‘?parse_gitlog’.
To build upon this we: 1) Do not create any dependency between configuration files and the API: functions take as input parameters which are familiar to any programmer, not a filepath to the project configuration file; 2) Use of project configuration files only in the first code block in R Notebooks to load the variables required to use the functions of the API, similar to how best practices in static In [10], Gousios notes a lesson he learned in the transition from Alitheia Core to GHTorrent: ‘Open now trumps open when it is done’. Kaiaulu is a new tool, but it already includes substantial usable functionality: parsers for Git, mailing list mbox archives (leveraging Perceval), and identity matching which enables socio-technical studies. And it is easy to add new capabilities, building upon the infrastructure we have created. Of note, unique to Kaiaulu is an interface to visualize networks interactively22. Kaiaulu has, as a core abstraction, a network model. We use this to, for example, relate and analyze authors, committers, commits, files, CVEs and CWEs rather than merely reporting project-level metrics. 18https://cran.r-project.org/web/packages/ 19See for example under Vignettes: https://cran.r-project.org/ web/packages/ggplot2/index.html 20https://github.com/sailuh/kaiaulu/blob/master/R/network.R 21Also publicly available at: http://www.catb.org/~esr/writings/ taoup/html/ch11s01.html 22For examples, see the various R Notebooks available in Kaiaulu documentation.
The authors wish to thank Damian Tamburri, Corne Broere, and Massimo Manca for identifying bugs, providing use cases and suggesting features for Kaiaulu. This work was supported in part by the US National Science Foundation under grant CCF-1817267. In this paper, through an action research approach, we have determined a set of key design decisions observed in existing tools, and iteratively developed Kaiaulu, an R package for mining software repositories building on our observations and lessons learned.
In Kaiaulu we have implemented, and are currently in the process of openly releasing, a comprehensive set of capabilities to mine, analyze, and visualize software repositories, including social smells [4], architecture smells and metrics [20], and bug timelines based on prior work by other authors. Kaiaulu is licensed under MPL 2.0. studies through a smart data platform, Empirical
Software Engineering 23 (2018). [8] A. Trautsch, F. Trautsch, S. Herbold, B. Ledel,
J. Grabowski, The smartshark ecosystem for software repository mining, arXiv preprint arXiv:2001.01606 (2020). [9] D. Spadini, M. Aniche, A. Bacchelli, PyDriller:
Python framework for mining software repositories, in: Proc. 26th ACM Joint Proceedings of
ESEC/FSE, ACM Press, 2018, pp. 908–911. [10] T. Menzies, L. Williams, T. Zimmermann, Perspectives on Data Science for Software Engineering, 1st ed., Morgan Kaufmann Publishers Inc., 2016. [11] D. Sculley, et al., Hidden technical debt in machine learning systems, in: Proc. 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, MIT Press, Cambridge, MA,
USA, 2015, p. 2503–2511. [12] G. Neubig, et al., XNMT: The extensible neural machine translation toolkit, in: Conference of the Association for Machine Translation in the Americas Open Source Software Showcase, 2018. [13] C. Bird, A. Gourley, P. Devanbu, M. Gertz, A. Swaminathan, Mining email social networks, in: Proc.
International Workshop on Mining Software Repos[1] G. Gousios, D. Spinellis, Alitheia core: An extensi- itories, ACM, 2006, p. 137–143. ble software quality monitoring platform, in: 2009 [14] J. Zhu, J. Wei, An empirical study of multiple names IEEE 31st International Conference on Software and email addresses in oss version control repositoEngineering, 2009, pp. 579–582. ries, in: IEEE/ACM 16th International Conference [2] S. Easterbrook, J. Singer, M.-A. Storey, D. Damian, on Mining Software Repositories (MSR), 2019, pp.
Selecting Empirical Methods for Software Engineer- 409–420.
ing Research, Springer, 2008. [15] F. Giarola, Detecting code and community smells [3] M. Joblin, S. Apel, C. Hunsen, W. Mauerer, Clas- in open-source: an automated approach, Master’s sifying developers into core and peripheral: An thesis, Politecnico di Milano, 2016. empirical study on count and network metrics, in: [16] D. A. Almeida, G. C. Murphy, G. Wilson, M. Hoye, IEEE/ACM 39th International Conference on Soft- Do software developers understand open source ware Engineering, 2017, pp. 164–174. licenses?, in: IEEE/ACM 25th International Confer[4] D. Tamburri, F. Palomba, R. Kazman, Exploring ence on Program Comprehension, 2017, pp. 1–11. community smells in open-source: An automated [17] M. Valiev, B. Vasilescu, J. Herbsleb, Ecosystem-level approach, IEEE Transactions on Software Engineer- determinants of sustained activity in open-source ing (2019) 1–1. projects: A case study of the pypi ecosystem, in: [5] D. Moreno, S. Dueñas, V. Cosentino, M. A. Fer- Proc. 26th ACM Joint Proceedings of ESEC/FSE, nandez, A. Zerouali, G. Robles, J. M. Gonzalez- ACM, 2018, p. 644–655.
Barahona, Sortinghat: Wizardry on software [18] J. Coelho, M. Valente, Why modern open source project members, in: IEEE/ACM 41st International projects fail, in: Proc. 11th Joint Meeting on FounConference on Software Engineering: Companion dations of Software Engineering, ACM, 2017, p.
Proceedings, 2019, pp. 51–54. 186–196. [6] S. Dueñas, V. Cosentino, G. Robles, J. M. Gonzalez- [19] E. S. Raymond, The Art of UNIX Programming, Barahona, Perceval: software project data at your Pearson Education, 2003. will, in: Proc. 40th International Conference on [20] R. Mo, Y. Cai, R. Kazman, L. Xiao, Q. Feng, ArSoftware Engineering: Companion Proceeedings, chitecture anti-patterns: Automatically detectable ACM, 2018, pp. 1–4. violations of design principles, IEEE Transactions [7] F. Trautsch, S. Herbold, P. Makedonski, on Software Engineering (2019).
J. Grabowski, Addressing problems with replicability and validity of repository mining