Towards an interdisciplinary, socio-technical analysis of software ecosystem health Tom Mens Bram Adams Josianne Marsan Software Engineering Lab MCIS Lab Faculty of Business Administration University of Mons Polytechnique Montreal Laval University Mons, Belgium Montréal, Canada Québec, Canada tom.mens@umons.ac.be bram.adams@polymtl.ca josianne.marsan@sio.ulaval.ca Abstract—This extended abstract presents the research goals have a centralised management for overseeing the ecosystem’s and preliminary research results of the interdisciplinary research health and survival. Instead, maintainers of SECO components project SECOHealth, an ongoing collaboration between research need to understand and make decisions about the socio- teams of Polytechnique Montreal (Canada), the University of Mons (Belgium) and Laval University (Canada). SECOHealth technical impact of important events affecting SECO health aims to contribute to research and practice in software en- and recommend corrective actions (e.g., improving SECO gineering by delivering a validated interdisciplinary scientific quality and its attractiveness to key actors). Unfortunately, methodology and a catalog of guidelines and recommendation there is only little support or best practices to enable SECO tools for improving software ecosystem health. maintainers to perform these tasks. Index Terms—software, ecosystem, evolution, health, recom- mendation, prediction, survival, sustainability, resilience, socio- technical II. A BOUT SOFTWARE ECOSYSTEM HEALTH I. I NTRODUCTION From a biological point of view, health can be defined The two-year interuniversity SECOHealth project1 started as “the extent to which an organism’s vital systems are on October 1, 2017. The three authors of this extended abstract performing normally at any given time” [1]. This definition are its Principal Investigators. SECOHealth aims to contribute can be transposed to SECO health [2] by considering a SECO to research and practice in software engineering by delivering as a living organism, whose constituent software projects are an interdisciplinary scientific methodology and a catalog of the vital technical systems that need to perform “normally” in guidelines and recommendation tools for improving software order to have a healthy ecosystem, and whose community is ecosystem (SECO) health. Those will enable key ecosystem healthy if all community members are performing normally. actors to better monitor and control the SECO health and equip them with corrective and preventive measures to ensure SECO health problems can be very diverse in nature, and their SECO’s survival and sustainability. The interdisciplinary can have many different causes. For example, in March 2016, methodology used in our project will also guide other re- the npm ecosystem experienced the problem of a package get- searchers in interdisciplinary projects involving open source ting unpublished, causing several thousands of transitively de- communities or SECOs. pendent packages to break. The underlying cause was a typical SECOs are large collections of interacting and interdepen- case of rage quitting, where the owner of the package decided dent software projects that share a common technological to remove all of its packages2 . Another documented example platform and that are maintained by large online communities of rage quitting relates to “toxic” communication styles of of contributors. They pervade every aspect of human life open source communities, such as the one of the Linux Kernel including entertainment, health, economy, industry, politics, community, causing a prominent developer to quit3 ; or the case education and science. Commercial SECOs such as mobile where a central contributor to the bug handling community of app stores or the Internet-of-Things have taken over our daily Gentoo Linux unexpectedly left, causing a major disruption lives by storm, to the extent that the functioning of our modern in the community’s activity [3]. From a more technical point digitally-enabled society would be severely impacted if SECOs of view, typical examples of health problems are packages degrade in stability or even cease to exist. containing bugs or security vulnerabilities, causing potential Yet, despite the strategic importance of ensuring the overall problems in packages depending on it. The impact of the well-being of SECOs, their health is still ill-apprehended, as problem grows as the number of transitive dependencies on SECOs are subject to constant evolution, due to an increasing a problematic package grows. pace of events (e.g., technological or environmental changes). What makes this especially challenging, is that SECOs do not 2 blog.npmjs.org/post/141577284765/kik-left-pad-and-npm 3 https://slashdot.org/story/15/10/05/2031247/ 1 www.secohealth.org linux-kernel-dev-sarah-sharp-quits-citing-brutal-communications-style 7 III. P ROJECT G OALS contributors (e.g., mean time to respond to a question, mean time to fix a bug), social network structure and its evolution SECOHealth aims at providing a scientific methodology (e.g. turnover rate), contributor activity and productivity, and and disciplined set of techniques to understand and control the quality of interaction between all human stakeholders. the health of software ecosystems. We adopt a socio-technical Phenomenological health characteristics include the amount perspective since the technical and social layers of SECOs are of company involvement (i.e., paid contributors), market share, strongly interwoven [4]. Our project aims to: presence of competing products, and so on. • define a conceptual model of SECO health; With respect to the social health problem of developer • explore analogies from other scientific disciplines such turnover, we conducted an empirical study on the npm and as ecology and toxicology; RubyGems ecosystems. Using the statistical technique of • determine indicators capable of measuring the different survival analysis we identified which social or technical factors aspects of SECO health; in a SECO coincide with a higher or lower probability of • determine events that affect the health of a SECO and its developer abandonment [10]. constituent projects; Concerning technical health, we carried out a quantitative • empirically validate these health indicators and events, empirical analysis of the evolution of package dependency both qualitatively and quantitatively; networks for seven package distributions of varying size • build and evaluate models to predict the impact of a given and age [11]. We proposed metrics to capture the growth, event on SECO health; changeability, reusability and fragility of these dependency • build and evaluate a socio-technical dependency model networks. We observed that the dependency networks tend to to understand how health problems propagate throughout grow over time, while a minority of packages are responsible a SECO; for most of the package updates. The majority of packages • propose a catalog of guidelines and recommendations for depend on other packages, but only a small proportion of supporting SECO health. packages accounts for most of the reverse dependencies. We Joining our complementary strengths in theory-driven and observed a high proportion of “fragile” packages due to a high data-driven investigation, we will follow a mixed-methods and increasing number of transitive dependencies. approach [5], combining bottom-up data mining and top- down interview/survey-based research, as well as combining IV. I NTERDISCIPLINARY R ESEARCH state-of-the-art quantitative and qualitative analysis techniques SECOHealth will view SECOs as ecological ecosystems emanating from different scientific disciplines. comprised of a population of living organisms (interdependent Under the approval of Research Ethics Committees from the software projects and their interacting communities of con- participating universities, we conducted face-to-face interviews tributors), and will produce health indicators and prediction at the European Open Source Summit of the Linux Foundation models by drawing inspiration from well-known principles (Prague, October 2017) with 17 SECO practitioners. The and theories from other disciplines, such as the notion of interviews followed the guidelines of Patton [6], with the goal biodiversity in ecology [12], or the notion of toxicity in of understanding what SECO health means for practitioners, toxicology [13], [14]. what indicators they use themselves or could be used given SECO health needs to be studied at different levels of the right data, and which events have impacted SECO health granularity since the health of the SECO as a whole depends in the past. on the health of its social and technical components, and vice We will operationalise the SECO health indicators into versa. At a micro-level of analysis (i.e., within and between concrete metrics, and perform SECO data mining to measure individual projects of a SECO), we will explore the impact and evaluate the identified health indicators. We will build of toxicity, arguing that certain behaviour and interactions in and empirically validate prediction models of how SECOs the SECO community can be toxic to not only the individual will react to events, by relying on historical data from version software projects, but even to the SECO as a whole, and control systems, code review and bug repositories, mailing lists hence can jeopardize its health and sustainability. Examples and developer fora. of possible toxic social behaviour may consist of deviant or Based on the recent research on SECO and community aggressive behaviours, for example in the form of flame wars health [2], [7]–[9], we will consider three high-level character- as a reaction to bad quality code contributions [13]. One istics of health: technical (i.e., concerning technical software promising way to assess such toxicity is by measuring social artefacts), social (i.e., concerning contributor communities and debt [15], i.e., social interactions between SECO members that the relations between their members) and phenomenological have been strained due to time pressure or lack of attention, (i.e., concerning external/internal events and their manifesta- and at some point might blow up and cause friction within the tion). Technical health characteristics include traditional soft- community of developers involved in a software project. ware quality metrics, software dependency structure, software At a macro-level, we will study how health problems of growth rate, size and frequency of software updates, bug fixes, SECO components evolve and propagate to others. Among security vulnerabilities, obsolete or deprecated components, others, we will test the principle of biodiversity by analysing and so on. Social characteristics include responsiveness of to which extent the SECO’s resilience decreases when its 8 diversity decreases. By resilience we refer to the ecosystem’s [6] M. Q. Patton, Qualitative research and evaluation methods : integrating capacity of resisting to disturbances, or recovering from a theory and practice. SAGE Publications, 2015. [7] F. Fotrousi, S. A. Fricker, M. Fiedler, and F. Le-Gall, “KPIs for software perturbation quickly. Diversity will be analysed according to ecosystems: A systematic mapping study,” in Int’l Conf. Software a variety of factors (e.g., geographical, activity-related, time- Business (ICSOB). Springer, 2014, pp. 194–211. related, gender-related, artefact-related). Some of these factors [8] J. Y. Monteith, J. D. McGregor, and J. E. Ingram, “Proposed metrics on ecosystem health,” in Int’l Workshop on Software-defined Ecosystems have been shown to have positive effects on software health. (BigSystem). ACM, 2014, pp. 33–36. For example, gender diversity has been shown to have a [9] K. Manikas and D. Kontogiorgos, “Characterizing software activity: positive effect on the productivity of GitHub teams [16]. The influence of software to ecosystem health,” in Int’l Workshop on Software Ecosystems (IWSECO), European Conf. Software Architecture Workshops, 2015. V. R ELATED W ORK AND R ELATED P ROJECTS [10] E. Constantinou and T. Mens, “An empirical comparison of developer retention in the RubyGems and npm software ecosystems,” Innovations SECOHealth can be considered as a successor project of the in Systems and Software Engineering, vol. 13, no. 2-3, pp. 101–115, ECOS project (Ecological Studies of Open Source Software 2017. Ecosystems) [17] that was carried out from 2012 till 2017. [11] A. Decan, T. Mens, and P. Grosjean, “An empirical comparison of dependency network evolution in seven software packaging ecosystems,” As part of that project, we carried out multiple empirical Empirical Software Engineering, 2018. studies of the evolution of open source software ecosystems [12] T. Mens, M. Claes, P. Grosjean, and A. Serebrenik, Studying Evolving (e.g., Gnome [18], CRAN [19]–[21], Debian [22], npm and Software Ecosystems based on Ecological Models. Springer, 2014, pp. 297–326. RubyGems [23]–[25]). [13] K. Carillo and J. Marsan, ““The dose makes the poison” - exploring the The SECOHealth members are actively involved in the toxicity phenomenon in online communities,” in Int’l Conf. Information CHAOSS (Community Health Analytics of Open Source Soft- Systems (ICIS), 2016. [14] K. Carillo, J. Marsan, and B. Negoita, “Exploring the biosphere – ware) initiative of the Linux Foundation. Metrics Committee. towards a conceptualization of peer production communities as living Its goal is to define, implement and assess metrics for open organisms,” in SIGOPEN Pre-ICIS 2017 Workshop on Open Phenomena, source community health and sustainability. While CHAOSS’ Association for Information Systems (AIS), Seoul, South Korea, 2017. [15] D. A. Tamburri, P. Kruchten, P. Lago, and H. v. Vliet, “Social debt initial focus is on metrics at the level of individual software in software engineering: insights from industry,” Journal of Internet projects, SECOHealth will focus on ecosystem-wide health Services and Applications, vol. 6, no. 1, May 2015. metrics. To operationalise our metrics, we aim to use Bitergia’s [16] B. Vasilescu, D. Posnett, B. Ray, M. G. J. van den Brand, A. Serebrenik, P. T. Devanbu, and V. Filkov, “Gender and tenure diversity in GitHub open source GrimoireLab tool chain for software development teams,” in Int’l Conf. Human Factors in Computing Systems (CHI). analytics4 , which is one of the tool suites actively supported ACM, 2015, pp. 3789–3798. by CHAOSS. [17] T. Mens, M. Claes, and P. Grosjean, “ECOS: Ecological studies of open source software ecosystems,” in IEEE Int’l Conf. Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE), 2014, pp. ACKNOWLEDGMENT 403–406. This research is carried out in the context of FRQ-FNRS [18] B. Vasilescu, A. Serebrenik, M. Goeminne, and T. Mens, “On the variation and specialisation of workload: A case study of the Gnome collaborative research project R.60.04.18.F “SECOHealth” ecosystem community,” Empirical Software Engineering vol. 19, pp. and FNRS Research Credit J.0023.16 “Analysis of Software 955–1008, 2014. Project Survival”. We thank our external project collaborators [19] D. M. Germán, B. Adams, and A. E. Hassan, “The evolution of the R software ecosystem,” 2013, pp. 243–252. Kevin Carillo (Toulouse University, France), Damian Tamburri [20] M. Claes, T. Mens, and P. Grosjean, “On the maintainability of CRAN (Politecnico di Milano, Italy), Gregorio Robles (Universidad packages,” in IEEE Int’l Conf. Software Maintenance, Reengineering, Rey Juan Carlos, Spain) and Bogdan Negoita (HEC Montréal, and Reverse Engineering (CSMR-WCRE), 2014, pp. 308–312. [21] A. Decan, T. Mens, M. Claes, and P. Grosjean, “When GitHub meets Canada) for actively participating to this project. CRAN: An analysis of inter-repository package dependency problems,” in Int’l Conf. Software Analysis, Evolution, and Reengineering (SANER), R EFERENCES 2016. [22] M. Claes, T. Mens, R. D. Cosmo, and J. Vouillon, “A historical analysis [1] X. Wang and S. Lantzy, “A systematic examination of member turnover of Debian package incompatibilities,” in Working Conf. Mining Software and online community health,” in Int’l Conf. Information Systems (ICIS), Repositories (MSR), 2015, pp. 212–223. 2011. [23] E. Constantinou and T. Mens, “Social and technical evolution of [2] S. Jansen, “Measuring the health of open source software ecosystems: software ecosystems: A case study of Rails,” in Int’l Workshop on Beyond the scope of project health,” Information and Software Technol- Software Ecosystems (IWSECO), European Conf. Software Architecture ogy, vol. 56, no. 11, pp. 1508 – 1519, 2014, special issue on Software Workshops, 2016. Ecosystems. [24] A. Decan, T. Mens, and M. Claes, “An empirical comparison of de- [3] M. S. Zanetti, I. Scholtes, C. J. Tessone, and F. Schweitzer, “The rise pendency issues in OSS packaging ecosystems,” in Int’l Conf. Software and fall of a central contributor: Dynamics of social organization and Analysis, Evolution and Reengineering (SANER), 2017, pp. 2–12. performance in the Gentoo community,” in International Workshop on [25] E. Constantinou and T. Mens, “Socio-technical evolution of the Ruby Cooperative and Human Aspects of Software Engineering (CHASE), ecosystem in GitHub,” in Int’l Conf. Software Analysis, Evolution and 2013, pp. 49–56. Reengineering (SANER), 2017, pp. 34–44. [4] T. Mens, “An ecosystemic and socio-technical view on software mainte- nance and evolution,” in Int’l Conf. Software Maintenance and Evolution (ICSME), 2016, pp. 1–8. [5] R. B. Johnson and A. J. Onwuegbuzie, “Mixed methods research: A research paradigm whose time has come,” Educational Researcher, vol. 33, no. 7, pp. 14–26, 2004. 4 http://grimoirelab.github.io 9